Create a logstash input and output

[deontologician]

Elasticsearch rivers are on their way out, and the recommended replacement is logstash. Logstash handles realtime ETL between systems, so creating an input and output would allow RethinkDB to plug into RabbitMQ, Elasticsearch, and other systems without needing a custom adapter for each system.

(Thanks to @barkerja for the suggestion)

http://logstash.net/docs/1.4.2/extending/
examples: https://github.com/elasticsearch/logstash/tree/master/lib/logstash

[thelinuxlich]

very cool!

[barkerja]

👍

[sharipov-ru]

👍

[wojons]

@deontologician

I have had to fix some existing logstash plugins before if you would like i can help work on the code for getting a rethinkdb input and output going.

[deontologician]

Yeah, if you're interested, please do. It'll probably be a bit before I get to this one

[FlagSystemes]

👍

[tlvenn]

+1 that would be really great to have this. @deontologician I read in another issue that you have started working on this, hope it goes well.

[deontologician]

Hey guys, I just put the initial (very very beta) version of the logstash input up at https://github.com/rethinkdb/logstash-input-rethinkdb/tree/master .

Currently, it needs a logstash filter to be able to pipe directly into ElasticSearch, so this is one piece of the puzzle, not the entire thing. That being said, I'd really appreciate any early feedback you can provide if you're interested in playing around with it.

[frankleng]

@deontologician any ETA on the 2.1 release that will make this more useful?

[danielmewes]

@frankleng We actually restructured the roadmap for the two upcoming releases slightly. The features this needs are now going to be in 2.2 (2.1 will instead have auto failover and improved cluster management). According to the current plans, 2.2 should be out relatively quickly after 2.1. My guess is about 8 weeks from now.
Sorry for the delay.

[frankleng]

@danielmewes Thanks Daniel! That sounds good. Rivers won't be removed until late versions of ES 2.x, so we'll keep using it for now.