Feature Request: Prometheus Exporter for Monitoring Filebeat

[anandsinghkunwar]

I see filebeat already has X-Pack monitoring that needs an ES backend, I'd like to make a feature request for a prometheus exporter for filebeat to monitor its metrics via prometheus.

Here is the link to the discussion on elastic forum

[ruflin]

If you run your beat with the config http.enabled: true it will expose an http endpoint under localhost:5066 with the same stats that are reported to X-Pack Monitoring. It's not in prometheus format but I wonder if that is enough? We are also discussing potentially having a Metricbeat module for it, so you monitor Filebeat with Metricbeat if you want.

[anandsinghkunwar]

This will work on filebeat, right? Couldn't find any documentation regarding this in filebeat monitoring.

[ruflin]

It will work for all Beats. The config option can be found here: https://github.com/elastic/beats/blob/master/libbeat/_meta/config.reference.yml#L1013 It's not really documented yet. We plan to expand the API endpoint to expose more stats and metrics in the future.

If you start using it, curious to hear more and your input as this API is still in it's early phase.

[PSUdaemon]

Would be great if there was an option to export the Prometheus format via this endpoint. Would you accept a patch to do that? Maybe add something like http.format and http.path options to the YAML.

[exekias]

Hi there,

I'm wondering if an external exporter would make more sense here. We should add better documentation for the metrics endpoint, that should probably be enough for any vendor to gather the metrics their need?

[anandsinghkunwar]

Both seem viable options, I have seen projects with the format=prometheus inbuilt for their exposed metrics. An external exporter isn't a bad choice either.

[frittentheke]

@exekias @anandsinghkunwar Why run another peace of software just to change the formatting of the exposed metrics?

Especially when running filebeat inside Kubernetes having the metrics scraped by Prometheus is so simple, provided they are in the right format ;-)

[anandsinghkunwar]

"the right format" seems to be the missing piece here. What do you suggest @frittentheke ?

[exekias]

I've opened #7798 to document the HTTP endpoint, it should expose enough metrics to write an exporter.

I would prefer to see an exporter first, as a way to learn about what are the metrics wanted by users and how they deal with them. We can revisit this decision in the future.

[paddie]

@exekias We're also very keen on gathering the performance data of filebeat using prometheus. As @PSUdaemon asked, would you guys be interested in us doing the work of incorporating something like what he proposed?

A http.format and http.path or http.metric configuration setting seems like a non-breaking change that would simplify matters greatly (and allow for additional formats for formats).

Running a service alongside this service is something we'd really rather not do, so we might just branch this service to make this happen for the short term. This of course introduces the obvious downside of our branch becoming out of sync with the main branch and future bug-fixes would be annoying to keep up with 😞

So, can we gauge the interest of having one of our Go guys write a POC of this and have someone look it over?

[ruflin]

The endpoint will not only be used for metrics in the future but potentially also to interact with the Beat. If we a config option like proposed above, it would apply to all end points.

As @exekias mentioned I think it's a decision we should delay and also get the API endpoints more stable on our end first

@paddie If you fork the project to make this happen, please share a link to the code here if possible.

[frittentheke]

@anandsinghkunwar I absolutely would go for Prometheus format of metrics, which seems to be the de-facto standard everybody is currently using ... see https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md

There is extensive documentation on best-practives regarding Prometheus exporters:
https://prometheus.io/docs/instrumenting/writing_exporters/#writing-exporters

Also all required go libraries to write Prometheus exporters are readily available: https://github.com/prometheus/client_golang
having such a metrics endpoint could and should be completely independent from any admin http endpoint / rest-api to interact with beat. @paddie if you implement this as an independent endpoint would likely allow this to be reused.

[yubobo]

Is this under development？or any conclusions? @exekias @PSUdaemon @ruflin
thanks~

[paddie]

@yubobo we went with the separate exporter approach since we really just needed to move on this and couldn't quite be bothered with being blocked for the duration of a branch we would have to maintain.

ping @shivas for the exporter that we are going to release in the not too far-away future.

[shivas]

@yubobo yes, we gonna release exporter for filebeat (with rest *beat coming too), i already have working filebeat exporter. I even already reserved port for exporter on prometheus list of ports

[anandsinghkunwar]

@shivas Can you give an ETA on the filebeat exporter, as I am thinking of writing my own for our usecase.

[paddie]

@anandsinghkunwar I'm code reviewing it now, we might put it out in a beta today to have more people play with it.

[paddie]

@anandsinghkunwar @yubobo @shivas just released it:

https://github.com/trustpilot/beat-exporter

[shivas]

@paddie don't jump yet :) just got approval to OSS it, so ☝️ repo will be public in few minutes

[shivas]

https://github.com/trustpilot/beat-exporter - it's live, please review, give feedback, maybe some namings are misleading or whatever... we releasing it even before rolling it out for our own needs

[ruflin]

@shivas One thing I was wondering on what is the recommended setup if Metricbeat and Filebeat are running on the same host machine?

[shivas]

@ruflin well, it's scraping exporting just one endpoint, so most likely setup would be to run second beat-exporter on different port. One for filebeat, one for metricbeat

[paddie]

Would it make sense to maybe enable the option of exciting multiple beats with one? As long as the metrics don't overlap in naming (which I don't think they do with the current beat type prefixing approach). Could be worth exploring..

…

On Fri, Sep 7, 2018, 17:18 Audrius Karabanovas ***@***.***> wrote: @ruflin <https://github.com/ruflin> well, it's scraping exporting just one endpoint, so most likely setup would be to run second beat-exporter on different port. One for filebeat, one for metricbeat — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7432 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAaEQgdBLg-iCxru9ouu3fVBXrNKb9yiks5uYo5IgaJpZM4U5Zyq> .

[paddie]

Exporting* On Sat, Sep 8, 2018, 11:49 Patrick-Ranjit D. Madsen <pamad05@gmail.com> wrote:

…

Would it make sense to maybe enable the option of exciting multiple beats with one? As long as the metrics don't overlap in naming (which I don't think they do with the current beat type prefixing approach). Could be worth exploring.. On Fri, Sep 7, 2018, 17:18 Audrius Karabanovas ***@***.***> wrote: > @ruflin <https://github.com/ruflin> well, it's scraping exporting just > one endpoint, so most likely setup would be to run second beat-exporter on > different port. One for filebeat, one for metricbeat > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#7432 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAaEQgdBLg-iCxru9ouu3fVBXrNKb9yiks5uYo5IgaJpZM4U5Zyq> > . >

[anandsinghkunwar]

As there is an exporter now, should this issue be closed?

[exekias]

Thanks everyone! I'm closing this issue as we have at least one exporter supporting beats out there

[frittentheke]

@anandsinghkunwar @exekias while it's really really awesome to have an exporter now with @shivas work, I personally still quite like to see filebeat or beats in general to expose their metrics in Prometheus format.

[paddie]

@frittentheke yeah, we'd like that too. I think the maintainers wanted some proof that people actually was using the metrics (in its current state) before committing time to actually pull in changes specific to prometheus.

We hope us doing some work on it will validate that, and also light a little fire under the people blocking it :)

[nikolay]

External exporters are workarounds. I found exporters as plugins for Kibana and Elasticsearch - if can at least the HTTP endpoint support plugins a la Kibana & Elasticsearch so this can be incorporated to simplify the deployment?

[towolf]

If filebeat is running as a DaemonSet you want to run yet another container just to monitor it?

Is this a case of NIH?

[paddie]

I think opening a new issue that explicitly states the option of the prometheus metrics endpoint is better than us still commenting on this issue @towolf @nikolay

[SleepyBrett]

I'm going to plus one this. Having a prometheus/openmetrics endpoint would be a huge win for us. My current log->mtail->prometheus architecture is brittle and dumb.

[hodgesds]

I don't see why this issue was closed, having a separate tool to do this doesn't fulfill the original feature request. There already has been work done on the ingestion side of Prometheus so I think this work can now be prioritized.

[nikolay]

I don't get why Elastic co. keeps ignoring the existence of Prometheus! I have to maintain custom Docker images so that I can install Elasticsearch and Kibana plugins, and also a custom sidecar for Filebeat! I don't get why their health endpoints don't have a simple ?format=json|prometheus parameter and just have two serializers for the two major types!

[towolf]

I don't get why

Vendor lock-in? Or, NIH syndrome?

[sepich]

One thing I was wondering on what is the recommended setup if Metricbeat and Filebeat are running on the same host machine?

Would it make sense to maybe enable the option of exciting multiple beats
with one?
As long as the metrics don't overlap in naming (which I don't think they do
with the current beat type prefixing approach).

Alternative exporter implementation, with multiple Beats scraping support and metric filtering:
https://github.com/sepich/beats-exporter
Example k8s DaemonSet for Filebeat+Journalbeat with single exporter provided.

[SuperQ]

Yes, sadly, the json provided by the "X-Pack Monitoring" is pretty bad. There's no help data, the stats are limited and not very descriptive. There's no type information (counter|gauge). There's an "uptime", but no process start time. It includes useless system load average metrics that are unrelated to the process itself. It's missing several useful Go metrics like go_goroutines.

[frittentheke]

There are even pushes to make the Prometheus style metrics a standard ... https://openmetrics.io/

[dudubis]

@anandsinghkunwar @yubobo @shivas just released it:

https://github.com/trustpilot/beat-exporter

Can you please share the matching grafana dashboard if you have one? Thanks!