Logstash part of ECK

[a1exus]

Hello World!

I see deploying Elasticsearch, Kibana and even APM, but NOT Logstash.

So there is my feature request)

[idanmo]

/cc @agup006

[dbwest]

What's the status on this?

[wescran]

Would also like to know the status on this. Would be great to have.

[dmitryzykov]

+1 for Logstash as part of ECK

[wutkemtt]

+1 Would be really helpful having LogStash integrated in ECK.

[emanuelflp]

+1

[wutkemtt]

Is there a plan to release this feature in the next weeks ?

[agup006]

We do not have any updates on Logstash as part of ECK at this time. Today, ECK can manage ingest node as part of a deployment that can be used alongside beats

[bitgandtter]

Any insights on why logstash is not part of ECK? the ELK stack misses that important piece. There must be a really important reason to not put a priority on this. Can maintainers expose the reason for it?

[tumluliu]

Is there any update to this issue? The missing of logstash is the final blocker for me to deploy ECK

[iFreeon]

+1 on what is the product team reasoning for not including Logstash. Is there a better way to do import logs or some integration with Kube where we don't need it any more?

[kksudo]

+1

[xlanor]

+. Even a year later, logstash support is not a priority?

The closest I could find was this medium article, will be attempting it shortly
https://medium.com/@raphaeldelio/deploy-logstash-and-filebeat-on-kubernetes-with-eck-ssl-and-filebeat-d9f616737390

[wutkemtt]

To all who are looking for the logstash feature set in ECK:
Probably Ingest Pipelines is the missing feature. I realized this during the update to the latest version 7.8. Definition of pipelines is easy and can contain nearly all processors known from logstash. Just take a look in Kibana under Stack Management.

[xlanor]

@wutkemtt , I took a look at ingest pipelines, but I don't see rabbitmq support in there like logstash does. Did I miss it?

Either way, using that medium article works, but I had to modify ES url to http instead of https.

The templates provided in that article require some quick modification to get it deployed as a deployment instead of a pod.

[wutkemtt]

I just needed the processors, so I don't know about the RabbitMQ support, sorry.

[framsouza]

this guide may help you to setup it https://github.com/framsouza/logstash-eck-ingress-controller

[brsolomon-deloitte]

Probably Ingest Pipelines is the missing feature. I realized this during the update to the latest version 7.8. Definition of pipelines is easy and can contain nearly all processors known from logstash. Just take a look in Kibana under Stack Management.

Ingest pipelines are only able to receive data rather than consume it. This is a key difference from logstash which can, for instance, act as a consumer to pull events from a beats -> kafka -> logstash -> elasticsearch pipeline.

Some rationale from maintainers as to why logstash is not included in ECK CRDs/portfolio would be appreciated.

[KetulRadadiya]

+1

[kplanz]

There is an official recipe for using Logstash with ECK: https://github.com/elastic/cloud-on-k8s/tree/main/config/recipes/logstash

There is nothing yet in the docs about this and it's obviously not a CRD for Logstash, but still may be helpful for some people.

[duclm2609]

I still don't get it why Logstash is not include. It has been 3 years already.

[Asafkbalink]

Vouching for this. 3 years. How is this is not included but APM is?

[jeansalama]

+1 to add Logstash with ECK or helm charts

[FranAguiar]

What's the point of deprecate the logstash chart if is not included in ECK?

[flexitrev]

I'm happy to announce that we are adding Logstash to ECK elastic/logstash#15030

[a1exus]

I'm happy to announce that we are adding Logstash to ECK elastic/logstash#15030

well, that's new... ;-)

thanks! after 3 long years))

[danielloader]

Hey looks good!

How would you do this in the new operator model?
https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html#installing-plugins

I would need to install the kafka plugin when using this.

[flexitrev]

This is support for a limited set up plugins built in, including kafka. We plan on adding more. You can see the use-facing documentation here - https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-logstash-technical-preview-limitations.html

[Kushmaro]

This is now available, with ECK 2.8. docs here enjoy folks, thanks for the likes and comments here!

Closing this issue.