Suggestion to add Elasticsearch output

[raghu999]

Graylog should have an elasticsearch output like gelf and splunk and be able to send Data to different ElasticSearch Clusters Now that it has ability to Ingest data via Http and ElasticSearch supports Cross Clustering. This allows graylog to play the role of aggregator like fluentd.

Expected Behavior

Graylog should be able to Write Different streams to multiple elasticsearch clusters via ouputs option for teams who are managing more than 10 Elasticsearch clusters

Current Behavior

Currently Graylog supports writing different streams after processing to different graylog cluster with gelf output and also supports writing to splunk and others. This suggestion allows garylog users to use the capability of Federation.

Possible Solution

We have Different streams and our cluster is processing Almost 250000Msg/Sec at peak as we know Elasticsearch has limits on how much data can be ingested . We want Graylog be able to send the streams to different elasticsearch clusters as it currently allows to send to splunk or to a different outputs natively. This gives ability for one graylog cluster to write to multiple elasticsearch clusters like fluentd and With Tribe nodes pointed to a different Graylog cluster we can have the federated search capability. So with 2 Graylog Clusters one for Write and one for Read we can manage 100's of Elasticsearch clusters.

Context

We have more than 15 Elastic search clusters currently we have 15 graylog and mongo clusters aswell. The idea gives graylog ability to manage 100's of elastic search clusters with just two Graylog clusters which can be scaled based on read/writes.

Your Environment

• Graylog Version: 2.3.1
• Elasticsearch Version: 5.x
• MongoDB Version: 3
• Operating System: Centos7.2

[raghu999]

Also Relates to
#1004