Add metadata to prometheus metrics

[jonny-wg2]

Hi, I am curious if there is a way to add metadata from an es query to a prometheus metric?

[braedon]

Hi @jonny-wg2, what kind of metadata are you after, and how would you need it exposed?

[jonny-wg2]

Sorry for the late reply.

Essentially, I am wondering if it is possible to add prometheus labels to a prometheus metric that is created by the exporter.

For example, adding the following to exporter.cfg

[query_es_query_custom_tag]
QueryIndices =
QueryJson = {
              "query": {
                "bool": {
                  "must": [
                    { "match" : { "tags": "custom-tag-here" } }
                  ],
                  "filter": {
                    "bool": {
                      "must": [
                        { "range": { "@timestamp": { "gte": "now-15m", "lt": "now" } } }
                      ]
                    }
                  }
                }
              }
            }
           

Would create a metric with
es_query_custom_tag_hits with the metric only having the total hit count - such as

es_query_custom_tag_hits{endpoint="metrics",instance="ipaddress:port",job="es-exporter",namespace="monitoring",pod="es-exporter-86545d94cd-vfrml",service="es-exporter"} 3

Would there be a way to also include prometheus labels with metadata from the metric? Such as the tag, or other metadata?

[braedon]

Creating result buckets with an aggregation (e.g. terms or filter) is usually the best way to create labels from data like tags.

Your example seems to only match a single tag, however. In cases like that, the tag data would usually be part of the metric name, not a label? Is there a particular reason you need it as a (single value) label?

There have been a few requests before for the ability to add extra labels, but so far they've all been solved by changes to the query (e.g. aggregations).

[braedon]

Closing for now - feel free to re-open if you'd like to discus your use case further.