WebSockets for ElasticSearch
Java
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
src
.gitignore
README.md
pom.xml

README.md

Elasticsearch WebSocket transport plugin

Follow @xbib on Twitter

NOTE: I plan to move this plugin over to a Ratpack-based plugin: https://github.com/jprante/elasticsearch-plugin-ratpack If you have questions, concerns, comments, feel free to open an issue.

This is an implementation of WebSockets for Elasticsearch.

WebSockets are implemented as an Elasticsearch transport plugin using the latest implementation of WebSockets in the Netty project.

The WebSocket protocol specification defines an API that enables web clients to use the WebSockets protocol for two-way communication with a remote host. It defines a full-duplex communication channel that operates through a single socket over the Web. WebSockets provide an enormous reduction in unnecessary network traffic and latency compared to the unscalable polling and long-polling solutions that were used to simulate a full-duplex connection by maintaining two connections. WebSocket-based applications place less burden on servers, allowing existing machines to support more concurrent connections.

Elasticsearch offers a HTTP REST API for nearly all the features available, so using it via curl or via script languages is comparable to a HTTP client connecting to a HTTP server. Some limitations apply when using the REST API and WebSockets come to the rescue.

Motivations for implementing an Elasticsearch WebSocket transport layer are

  • to supersede the HTTP request/response model by a full-duplex communication channel

  • to implement scalable and responsive real-time apps, like distributed publish/subscribe services

  • to attach thousands of clients to an Elasticsearch node without service degradation

  • to allow sequences of bulk index and delete operations on a single connection

  • to implement new types of streaming applications like subscribing to change streams from ElasticSearch indexes

Versions

Elasticsearch version Plugin Release date
1.4.0 1.4.0.0 Jan 8, 2015
1.3.1 1.3.1.0 Jul 30, 2014
1.2.1 1.2.1.1 Jun 9, 2014

Installation

./bin/plugin -install websocket -url http://xbib.org/repository/org/xbib/elasticsearch/plugin/elasticsearch-transport-websocket/1.4.0.0/elasticsearch-transport-websocket-1.4.0.0-plugin.zip

Do not forget to restart the node after installing.

Project docs

The Maven project site is available at Github

Issues

All feedback is welcome! If you find issues, please post them at Github

Overview

Modules

The transport plugin uses Netty WebSockets for server and clients. WebSocket clients can connect to an Elasticsearch Node with the transport plugin installed. Between nodes, WebSockets connections may establish when needed for forwarding messages. The more nodes are installed with websocket transport, the more clients can get connected.

WebSocket Module

Websocket

The WebSocket module includes a module that allows to expose the Elasticsearch API over HTTP. It is superseding the standard HTTP module on port 9200-9299.

The http mechanism is completely asynchronous in nature, meaning that there is no blocking thread waiting for a response. The benefit of using asynchronous communication for HTTP is solving the C10k problem.

When possible, consider using HTTP keep alive
when connecting for better performance and try to get your favorite client not to do HTTP chunking.

Setting Description
websocket.port A bind port range. Defaults to 9400-9499.
websocket.max_content_length The max content of an HTTP request. Defaults to 100mb
websocket.compression Support for compression when possible (with Accept-Encoding). Defaults to false.
websocket.compression_level Defines the compression level to use. Defaults to 6.

Node level network settings allows to set common settings that will be shared among all network based modules (unless explicitly overridden in each module).

The network.bind_host setting allows to control the host different network components will bind on. By default, the bind host will be anyLocalAddress (typically 0.0.0.0 or ::0).

The network.publish_host setting allows to control the host the node will publish itself within the cluster so other nodes will be able to connect to it. Of course, this can't be the anyLocalAddress, and by default, it will be the first non loopback address (if possible), or the local address.

The network.host setting is a simple setting to automatically set both network.bind_host and network.publish_host to the same host value.

Both settings allows to be configured with either explicit host address or host name. The settings also accept logical setting values explained in the following table:

Logical Host Setting Value Description
local Will be resolved to the local ip address.
non_loopback The first non loopback address.
non_loopback:ipv4 The first non loopback IPv4 address.
non_loopback:ipv6 The first non loopback IPv6 address.
[networkInterface] Resolves to the ip address of the provided network interface. For example en0.
[networkInterface]:ipv4 Resolves to the ipv4 address of the provided network interface. For example en0:ipv4.
[networkInterface]:ipv6 Resolves to the ipv6 address of the provided network interface. For example en0:ipv6.

TCP Settings

Setting Description
network.tcp.no_delay Enable or disable tcp no delay setting. Defaults to true.
network.tcp.keep_alive Enable or disable tcp keep alive. By default not explicitly set.
network.tcp.reuse_address Should an address be reused or not. Defaults to true on none windows machines.
network.tcp.send_buffer_size The size of the tcp send buffer size (in size setting format). By default not explicitly set.
network.tcp.receive_buffer_size The size of the tcp receive buffer size (in size setting format). By default not explicitly set.

Disable WebSocket

The websocket module can be completely disabled and not started by setting websocket.enabled to false.

License

Elasticsearch Websocket Transport Plugin

Copyright (C) 2012 Jörg Prante

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations