ØMQ transport layer plugin for Elasticsearch
Java
Pull request Compare This branch is 13 commits ahead, 1 commit behind tlrx:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
src/main
.gitignore
README.textile
jzmq-1.0.0.jar
pom.xml

README.textile

ØMQ transport layer plugin for Elasticsearch

This plugin is a fork from https://github.com/tlrx/transport-zeromq. It expose the index REST interface through a ZeroMQ push pull socket, without sending response. It’s designed for mass indexing into ElasticSearch, for example with node-logstash.

Versions

ØMQ Transport Plugin ElasticSearch ØMQ
master (0.0.5) master (0.20.1) 2.2
master (0.0.4.1-SNAPSHOT) master (0.19.10) 2.2

Installation

Requirements

Before installing and using this plugin with Elasticsearch (0.19.10), you need to install ZeroMQ Java Binding library (2.2) on your system. This binding uses native library to work.

The ZeroMQ website is a great place to start installing the libraries, specially the Java binding page

If you want to develop or modify this plugin, I encourage you to read the ØMQ – The Guide which is very well documented.

Installation

Type the command in your favorite shell :

$ bin/plugin -url http://warp-repo.s3-eu-west-1.amazonaws.com/transport-zeromq-0.0.5.zip -install transport-zeromq

Elasticsearch automatically install the plugin:

-> Installing transport-zeromq...
Trying http://warp-repo.s3-eu-west-1.amazonaws.com/transport-zeromq-0.0.5.zip...
Downloading ............................................DONE
Installed transport-zeromq

Then, you must replace the file plugins/transport-zeromq/jzmq-1.0.0.jar with the Java binding compiled for your system.

Finally, edit Elasticsearch configuration file config/elasticsearch.yml and add the properties:

# ZeroMQ Transport config
zeromq.bind: tcp://0.0.0.0:9700

Restart Elasticsearch.

If you have the following error

Initialization Failed …
1) UnsatisfiedLinkError[no jzmq in java.library.path]2) NoClassDefFoundError[Could not initialize class org.zeromq.ZMQ]

You can try to update your LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

You can also read the documentation of the ØMQ Java binding.

Behind the scene

The plugin exposes the REST interfaces of Elasticsearch over ØMQ sockets. The implementation uses a push pull pattern, ElasticSearch is the pull socket.

For example, a ØMQ client can send the following message:

POST|/twitter/tweet/2|{"user" : “kimchy”, “post_date” : “2009-11-15T14:12:12”, “message” : “You know, for Search”}

It will receive no response.

The transport layer converts ØMQ messages in a given format into REST request objects that can be handled by ES.