Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more outputs than Elasticsearch and Logstash #943

Closed
monicasarbu opened this issue Feb 8, 2016 · 5 comments

Comments

Projects
None yet
5 participants
@monicasarbu
Copy link
Contributor

commented Feb 8, 2016

Currently you can configure in the Beats the following as outputs (in chronological order):

  • Elasticsearch
  • Redis
  • Logstash

Once we added support for Logstash, we deprecated Redis as you can use the redis output plugin to export data to Redis.
For someone that wants to integrate the Elastic Stack into its own system and make use of a queuing system before inserting data to Elasticsearch, the deployment looks like this:

Beats -> Logstash -> Redis/Kafka -> Logstash -> Elasticsearch

We received many requests from the community to add support for more outputs, besides Elasticsearch and Logstash to avoid having an additional Logstash instance to translate the data to the desired queuing system. Here are the links to the GitHub issues:

After internal discussions, we decided to follow the community feedback, so we will un-deprecate the Redis output and add support for Kafka output.

With these improvements, the deployments are simplified:

Beats -> Redis/Kafka -> Logstash -> Elasticsearch

TODOs:

  • Undeprecate Redis output
  • Update Redis output to support the same option as the Logstash and Elasticsearch outputs #1353
  • Implement Kafka output: #942

We will continue to be very selective in terms of what outputs we support. For the moment, we are only considering Redis and Kafka based on their popularity.

@adiworkoholic

This comment has been minimized.

Copy link

commented Mar 1, 2016

So... is the support for a rabbitMQ output off the table :(

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Mar 1, 2016

@adiworkoholic Our current focus/priority is on implementing Redis and Kafka. Best is to subscribe to the issue elastic/libbeat#313 as we will post an update there in case we add further outputs.

@brandbintley

This comment has been minimized.

Copy link

commented Mar 31, 2016

Kafka support is awesome. How do we see handling security? If I got it right, libbeat supports TLS while Kafka supports SSL and SASL.

@froztbyte

This comment has been minimized.

Copy link

commented Apr 6, 2016

I'm also in favour of a RabbitMQ output.

My previous model for this was beaver[0](with a sincedb configured) connecting to rabbitmq, logstash reading off of logstash then outputting to elasticsearch and others. The benefits of this design is that it gives quite a strong ability to deal with various failures or operational necessities. On a future model of Filebeats and friends, I would ideally like to be able to reuse this same design, or a design with its strengths

  • [0] The one downside with beaver was that it was synchronous by default. That wasn't great.
@monicasarbu

This comment has been minimized.

Copy link
Contributor Author

commented Apr 21, 2016

Closing the issue as all the tasks were finished.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.