New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bulk UDP #2201
Comments
A Bulk UDP service is a service listening over UDP for bulk format requests. The idea is to provide a low latency UDP service that allows to easily index data that is not of critical nature. The Bulk UDP service is disabled by default, but can be enabled by setting `bulk.udp.enabled` to `true`. The bulk UDP service performs intenral bulk aggregation of the data and then flushes it based on several parametres: * `bulk.udp.bulk_actions`: The number of actions to flush a bulk after, defaults to `1000`. * `bulk.udp.bulk_size`: The size of the current bulk request to flush the request once exceeded, defaults to `5mb`. * `bulk.udp.flush_interval`: An interval after which the current request is flushed, regarldess of the above limits. Defaults to `5s`. * `bulk.udp.concurrent_requests`: The number on max in flight bulk requests allowed. Defaults to `4`. The network settings allowed are: * `bulk.udp.host`: The host to bind to, defualts to `network.host` which defaults to any. * `bulk.udp.port`: The port to use, defaults to `9700-9800`. Here is an example of how it can be used: > cat bulk.txt { "index" : { "_index" : "test", "_type" : "type1" } } { "field1" : "value1" } { "index" : { "_index" : "test", "_type" : "type1" } } { "field1" : "value1" } > cat bulk.txt | nc -w 0 -u localhost 9700
A Bulk UDP service is a service listening over UDP for bulk format requests. The idea is to provide a low latency UDP service that allows to easily index data that is not of critical nature. The Bulk UDP service is disabled by default, but can be enabled by setting `bulk.udp.enabled` to `true`. The bulk UDP service performs intenral bulk aggregation of the data and then flushes it based on several parametres: * `bulk.udp.bulk_actions`: The number of actions to flush a bulk after, defaults to `1000`. * `bulk.udp.bulk_size`: The size of the current bulk request to flush the request once exceeded, defaults to `5mb`. * `bulk.udp.flush_interval`: An interval after which the current request is flushed, regarldess of the above limits. Defaults to `5s`. * `bulk.udp.concurrent_requests`: The number on max in flight bulk requests allowed. Defaults to `4`. The network settings allowed are: * `bulk.udp.host`: The host to bind to, defualts to `network.host` which defaults to any. * `bulk.udp.port`: The port to use, defaults to `9700-9800`. Here is an example of how it can be used: > cat bulk.txt { "index" : { "_index" : "test", "_type" : "type1" } } { "field1" : "value1" } { "index" : { "_index" : "test", "_type" : "type1" } } { "field1" : "value1" } > cat bulk.txt | nc -w 0 -u localhost 9700
as the udp is not reliable,so does that mean the data maybe losed for some reason? is that right? |
Yes, UDP comes with its downsides, but sometimes its acceptable. |
Hello! |
@vbichkovsky it should work, can you provide an example? |
The file: http://pastie.org/4909910 Commands I typed: I'm using a virtual box (debian squeeze) set up via Vagrant. Ports 9200 and 9700 are forwarded to 9292 and 9797. Elasticsearch was installed using .deb package downloaded from the site. |
I can see that the message sent is broken down into two, one in the size of |
You were right, it is netcat. |
@vbichkovsky luck guess on my end regarding netcat, thanks for checking it out!, Questions on ES are best asked on the ES google group. We lurk there quite a bit, or on IRC (but sometimes, you might need to ping us when we are online...). |
Hi @kimchy, I'm currently using 0.90.10. To enable this udp bulk api, I only need to add bulk.udp.enabled: true in the elasticsearch.yml file? Just wanna double check, since I'm not seeing bulk.udp.enabled in the elasticsearch.yml file shipped with ES installation. |
A Bulk UDP service is a service listening over UDP for bulk format requests. The idea is to provide a low latency UDP service that allows to easily index data that is not of critical nature. The Bulk UDP service is disabled by default, but can be enabled by setting `bulk.udp.enabled` to `true`. The bulk UDP service performs intenral bulk aggregation of the data and then flushes it based on several parametres: * `bulk.udp.bulk_actions`: The number of actions to flush a bulk after, defaults to `1000`. * `bulk.udp.bulk_size`: The size of the current bulk request to flush the request once exceeded, defaults to `5mb`. * `bulk.udp.flush_interval`: An interval after which the current request is flushed, regarldess of the above limits. Defaults to `5s`. * `bulk.udp.concurrent_requests`: The number on max in flight bulk requests allowed. Defaults to `4`. The network settings allowed are: * `bulk.udp.host`: The host to bind to, defualts to `network.host` which defaults to any. * `bulk.udp.port`: The port to use, defaults to `9700-9800`. Here is an example of how it can be used: > cat bulk.txt { "index" : { "_index" : "test", "_type" : "type1" } } { "field1" : "value1" } { "index" : { "_index" : "test", "_type" : "type1" } } { "field1" : "value1" } > cat bulk.txt | nc -w 0 -u localhost 9700
A Bulk UDP service is a service listening over UDP for bulk format requests. The idea is to provide a low latency UDP service that allows to easily index data that is not of critical nature.
The Bulk UDP service is disabled by default, but can be enabled by setting
bulk.udp.enabled
totrue
.The bulk UDP service performs intenral bulk aggregation of the data and then flushes it based on several parametres:
bulk.udp.bulk_actions
: The number of actions to flush a bulk after, defaults to1000
.bulk.udp.bulk_size
: The size of the current bulk request to flush the request once exceeded, defaults to5mb
.bulk.udp.flush_interval
: An interval after which the current request is flushed, regarldess of the above limits. Defaults to5s
.bulk.udp.concurrent_requests
: The number on max in flight bulk requests allowed. Defaults to4
.The network settings allowed are:
bulk.udp.host
: The host to bind to, defualts tonetwork.host
which defaults to any.bulk.udp.port
: The port to use, defaults to9700-9800
.bulk.udp.receive_buffer_size
: The receive buffer size, defaults to10mb
.Here is an example of how it can be used:
The text was updated successfully, but these errors were encountered: