Implement filters for Prometheus legend labels

[FauxFaux]

Fix #3545.

This allows replacements in labels in legends for the Prometheus data source. e.g.

{{ foo | replace("bar", "baz") }}

I also implemented the toPercent() function as an example. Other functions should be easy to add. I omitted the hostname parsing functions as they are pretty complex, but would be great to add as another PR. Personally, I care most about the replace() functionality.

There are various things I consider bugs in the old parser, which I have maintained, e.g. {{ this could never work }} expands to this could never work.

I stuck with a regex-based parser as:

• the existing code was broken specifically so it could use a simple regex based solution
• Angular's $parse can't be used, as it's not safe for arbitrary inputs (like the previous code).
• I couldn't find any reasonable JS libraries for this.

[torkelo]

looks interesting. Will have to see if this is something others are interested in :)

A little hesitant to add so much code and complexity on a whim so will have to wait to see what demand there is. Would prefer if this kind of logic was in the Prometheus query language instead.

[FauxFaux]

I count at least 28 unhappy people on the bug report (excluding me!). The bug report also suggests that the Prometheus people think it is a dashboarding concern, not a query language concern (in that they implemented it in dashboarding, before deprecating their dashboarding because Grafana was doing it).

We're actually still running 3.0, and using {{ foo.replace(//,'') }}, because this is considered such a serious problem.

[bergquist]

I think this sounds like a good idea. I wonder if we could make this more generic for other datasources as well.

Btw. Please be patient with this pull request. We are currently super swapped with work. We will get back to it.

[RichiH]

Sounds useful to me.

[coryfklein]

This would be extremely useful for us. Our legend labels are extremely verbose and often take up way too much space on our dashboards. Being able to display only the uniquely identifying info from the label would have a significant impact for us.

[torkelo]

this sounds very useful, but think it either needs to be a feature of the prometheus query language or a general feature built into core Grafana templating

[raiviskrumins]

+1
In our use case "substring" function would help a lot:

const functions = {
  ...
  substring: function(val, start, end) {
  	return val.substring(start, end)
  }
};

[FauxFaux]

For @raiviskrumins' substring, it might be useful to allow numerals; currently foo | substring("5", "7") would be required. As such, I don't plan to update the pull request.

[rdtr]

super useful! looking forward to seeing this merged! 🙏

[phantium]

Would love to see this merged.

[fredsig]

+1

[adamlamers]

+1, definitely missing this feature after upgrading.

[jstsch]

+1, labels in combination with Prometheus and many instances really need some sort of transformation.

[torkelo]

The template functions and syntax needs to implemented in the backend code as well, otherwise Prometheus queries that use this new legend format filters will not work when used in Alerting conditions.

https://github.com/grafana/grafana/blob/master/pkg/tsdb/prometheus/prometheus.go

[FauxFaux]

Does the pr_action: changes needed label imply that you would like me to do the work? I am happy to do so, but I don't understand what changes I could make that are likely to be accepted.

I can work out how to raise the code up to a level where it applies to all legend formatting, but that sounds like a very abusive change; presumably other data sources already have some syntax in their legend formatting which we would probably like to be bug-compatible with, maybe this is impossible? Is it worth investigating this?

I still believe there is significant value in merging this change as-is. Is there something you object to in the change itself that I could fix? I realise the code is.. hairy, and you might not want to support it long-term, but it's very, very isolated, and I have attempted to document its behaviour with tests. It's string processing in JS so doesn't feel risky from a security point of view.

[torkelo]

Sorry for not be clear. The changes needed is to the backend prometheus code, as these new filter will not work if you use it for a query that is used in alerting.

[torkelo]

Would like to merge this but cannot in it's current state as the legend format filter syntax & interpolation needs to be implemented in backend code as well.

https://github.com/grafana/grafana/blob/master/pkg/tsdb/prometheus/prometheus.go#L115

If not done soon we will have to close this PR.

[gdhgdhgdh]

FYI it's possible to work around this by making the PromQL queries more ugly, e.g. based on the default buckets where 5 is the largest quantified bucket, and +Inf above it, I'm successfully doing this:

label_replace(probe_mqtt_duration_seconds_bucket, "le", "5+", "le", ".Inf")

Now I have +Inf replaced by 5+ and my pseudo-histogram in mode: series is displaying correctly:

[eloo]

Hi,
is there any progress for this PR?
Would really like to use this feature :)

[tboerger]

So who is gonna do the backend integration? Just found this PR because of @bergquist at PromCon. This is really a feature I need to keep the labels more simple like dropping the hostname and port from scraped hosts.

[mmoleksf]

Any news on this? It would be really handy to have RegEx in this field.

[pkcinna01]

Can it just be documented that this may break alerts and update the alert panel to show a warning if it sees replacement filters in legend of prometheus queries? It would be great to have this functionality back and it sounds like waiting for the solution that addresses back end concerns might take a long time (if ever).

[bergquist]

@pkcinna01 having two different implementations is not an option. That would create a bad experience for those who use Grafana alerting. If anyone is up for contributing the backend part of this it would be very appreciated.

[Astro03]

@FauxFaux what happened to this PR? is it no longer going in?

[nicktimko]

1040 clicks to this issue from this post: https://community.grafana.com/t/regexp-in-legend-format/995/4

Is the solution to use 3.0.x?

[Armadill0]

I would really appreciate this feature!

[vvp83]

Hi! We need this feature! May be merge? From 6 Feb 2017 PR. This is JOKE???

[tboerger]

@vvp83 provide this and it will be merged I guess => #6627 (comment)

[pengisgood]

any progress on this extremely useful feature?

[sherif84]

+1

[chrisandrews7]

+1

[ntavares]

+1

[bagofmice]

Absolutely would love to do some string manipulation on legends. Would help clear screen of redundant data, like the shared part of a FQDN.

[davkal]

Would love this feature too, but closing it for inactivity. This does not mean we won't consider this again at some point. But it was highlighted that a solution needs to be implemented in both the backend and the frontend. I'm happy to guide someone through the process: david@grafana.com

[lrstanley]

I'm.. very confused. @torkelo mentions that this needs backend changes. Why would it need backend changes? It's only for visual effect (from my understanding?) and has nothing to do with the backend (prometheus)?

EDIT: to clarify, visual effect probably isn't the best terminology. I guess I don't understand why this isn't just a parser/injection item, and why backends have to do anything special. When using {{ xxx }}, this doesn't get passed to the backend (prometheus in this case), so not sure why it would with this change.

I.e. do the replacement/substring/etc at injection time. The variable is already known, and it gets injected before the backend even knows about it.

Also don't see this having anything to do with prometheus specifically.. this should be something that's possible on any backend because it has no ties to the backend type?

[SanderMertens]

I second @lrstanley's comment. I'm not sure what would be preventing support for a regex in the "legend format" field similar to the "Regex" field in the variable editor.

[tboerger]

With backend it doesn't mean the resource like prometheus, this simply means the grafana backend/api.

[benitogf]

The label_replace function worked for me on this one, removing a prefix on a metric name with:

label_replace({__name__="prefix_metric_x"}, "__name__", "$2", "__name__", "(prefix_)(.*)")

[m-kratochvil]

@benitogf the label_replace function is not supported on range-vector queries, so at best it's a partial workaround.. but better than nothing for now I gues..

[jdx-john]

Simply being able to treat the Legend text as a regex seems like it would be really useful and rather cosmetic. e.g. we have label values which are taken automatically from an enumeration so all values are like MY_ENUM__FOO, MY_ENUM_BAR, etc. Being able to trivially replace these as FOO, BAR would be great as I don't want to manually alias each possible value.

[ishangupta-ds]

We also have a similar requirement where we want to replace a label's value on rendering it in the legend table and annotation by a regex match. This feature would be helpful if implemented. eg. - {{ chaos_injection_time | replace("", "N/A") }}