Challenges for Integrations using Prometheus Input

[ishleenk17]

While working on multiple service integrations, we came to a point of using prometheus as input for many integrations.
As a result we started collating some of the challenges we are facing with Prometheus input and probable solutions for the same,
This document list the challenges and solutions

cc: @ruflin
Pinging @elastic/obs-service-integrations

[andrewkroh]

Can you please share the document publicly? The user community might be interested in this discussion.

[ishleenk17]

Listing down the conclusions from the document in the ticket as well:

Conclusions:

• For curated integrations, we should have a compact set of metrics and individual mapping should be followed here.
• Ways of handling metrics explosion:

1. Flattened type, this will index the entire object as a single field:
   https://www.elastic.co/guide/en/elasticsearch/reference/current/flattened.html
2. Using metrics_include/exclude: Here only the included metrics will be mapped.
   https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-prometheus-collector.html
3. If required for some scenarios, dynamic_mapping: false setting :
   https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html#dynamic
4. Mapping limit settings (Might have data rejections issues):
   https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html.
   A probable solution to handling data rejections: Make total fields limit less of a nuisance to users elasticsearch#89911

• Have as many options as possible exposed to the user. Eg: Bearer_token, ssl_authenticaion_authorities should be exposed
• Discussion around the generic way of handling integrations using Prometheus as endpoint for exporting metrics: This will be handled once we have the Prometheus input which will be providing custom mapping and ingest pipeline

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!