Prometheus: Adding Elasticsearch and Fluentbit metrics

[ricsanfre]

Enhancement Request

Adding Elasticsearch and Fluentbit metrics
.

Implementation details

Fluentbit exports prometheus metrics when enabling its built-in HTTP server. Metrics are available at /api/v1/metrics/prometheus endpoint. See fluentbit documentation

For exporting elasticsearch metrics, a Exporter (elasticsearch-exporter) maintained by Prometheus' community can be used. (https://github.com/prometheus-community/elasticsearch_exporter)

[ricsanfre]

About enabling fluentd Prometheus metrics

Since fluentd is also being included as log aggregator it needs to be configured to export Prometheus metrics. Details about exporting Prometheus metrics in [fluentd monitoring documentation] (https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus)

[ricsanfre]

About Monitoring Fluent-bit storage metrics.

Interesting presentation at FluentCon about monitoring fluentbit in a production environment: https://www.youtube.com/watch?v=OhlyY6glf0A

When enabling file buffering (production usual configuration), Fluentbit storage metrics should be monitored as well. These metrics are not exposed by Fluentbit in prometheus format (metrics endpoint). They are exposed in JSON format at /storage endpoint.

There is a open issue in Fluentbit to export storage metrics with prometheus format (fluent/fluent-bit#5334).

The presentation shows a way to export storage metrics to prometheus deploying a prometheus json exporter as sidecar in fluentbit pod.