ØMQ transport layer plugin for Elasticsearch

This plugin exposes the REST interfaces of Elasticsearch over ZeroMQ messaging library.

Versions

ØMQ Transport Plugin	ElasticSearch	ØMQ
master (0.0.3)	master (0.19.2)	2.2
0.0.2	0.18.2	2.1

Installation

Requirements

Before installing and using this plugin with Elasticsearch (0.19.2), you need to install ZeroMQ Java Binding library (2.2) on your system. This binding uses native library to work.

The ZeroMQ website is a great place to start installing the libraries, specially the Java binding page

If you want to develop or modify this plugin, I encourage you to read the ØMQ – The Guide which is very well documented.

Installation

Type the command in your favorite shell :

$ bin\plugin -install tlrx/transport-zeromq/0.0.3

Elasticsearch automatically install the plugin:

-> Installing tlrx/transport-zeromq/0.0.3...
Trying https://github.com/downloads/tlrx/transport-zeromq/transport-zeromq-0.0.3.zip...
Downloading ..........DONE
Installed transport-zeromq

Then, you must replace the file plugins/transport-zeromq/jzmq-1.0.0.jar with the Java binding compiled for your system.

Finally, edit Elasticsearch configuration file config/elasticsearch.yml and add the properties:

# ZeroMQ Transport config
zeromq.router.bind: tcp://*:9700
zeromq.workers.threads: 2
zeromq.workers.bind: inproc://es_zeromq_workers

Restart Elasticsearch.

If you have the following error

Initialization Failed …

1) UnsatisfiedLinkError[no jzmq in java.library.path]2) NoClassDefFoundError[Could not initialize class org.zeromq.ZMQ]

You can try to update your LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

You can also read the documentation of the ØMQ Java binding.

Behind the scene

The plugin exposes the REST interfaces of Elasticsearch over ØMQ sockets. The implementation uses a router-dealer pattern, where multiple ROUTER sockets (2 by default, see zeromq.workers.threads) are listening to incoming messages (each in a dedicated thread) send by DEALER sockets on the zeromq.router.bind address. This way, it is possible to send REST-like messages with ØMQ clients and get the replies back.

For example, a ØMQ client can send the following message:

POST|/twitter/tweet/2|{"user" : “kimchy”, “post_date” : “2009-11-15T14:12:12”, “message” : “You know, for Search”}

It will receive the following response back:

201|CREATED|{"ok" : true, “_index” : “twitter”, “_type” : “tweet”, “_id” : “2”, “_version” : 1}

The transport layer converts ØMQ messages in a given format into REST request objects that can be handled by ES.

The expected format for incoming messages is:

 <Method PUT,DELETE, POST...>|<URI, including parameters>|<JSON content> 

The format for outcoming message is:

 <Status code>|<Status name>|<JSON reply content> 

But any other message format can be easely implemented if needed.

Simple ØMQ client to test the plugin

The SimpleClient Java class in test package shows how to create a simple ØMQ client and send messages. In your test, take care to add the native library to classpath (-Djava.library.path=/usr/local/lib).

Add new document

Message:

java org.elasticsearch.zeromq.test.SimpleClient tcp://localhost:9700 PUT /twitter/tweet/2 “{\”user\“: \”kimchy\“,\”post_date\“: \”2009-11-15T14:12:12\“,\”message\“: \”You know, for Search\“}”

Reply:

201|CREATED|{"ok" : true, “_index” : “twitter”, “_type” : “tweet”, “_id” : “2”, “_version” : 1}

Search for document

Message:

java org.elasticsearch.zeromq.test.SimpleClient tcp://localhost:9700 GET /twitter/tweet/_search?q=user:kimchy

Reply:

200|OK|{"took" : 29, “timed_out” : false, “_shards” : {"total" : 5, “successful” : 5, “failed” : 0}, “hits” : {"total" : 1, “max_score” : 0.30685282, “hits” : [{"_index" : “twitter”, “_type” : “tweet”, “_id” : “2”, “_score” : 0.30685282, “_source” : {

    “user” : “kimchy”,

    “post_date” : “2009-11-15T14:12:12”,

    “message” : “You know, for Search”

}}]}}

Delete a document

Message:

java org.elasticsearch.zeromq.test.SimpleClient tcp://localhost:9700 DELETE /twitter/tweet/2

Reply:

200|OK|{"ok" : true, “found” : true, “_index” : “twitter”, “_type” : “tweet”, “_id” : “2”, “_version” : 2}

Other examples

The ZMQTransportPluginTest Java class in test package has other examples.

---

This software is licensed under the Apache License, version 2 (“ALv2”).

Thanks to David Pilato for the Maven pom and README files.