Add possibility to change text case

[grobie]

We have unfortunately two systems which expose labels in both upper case and lower case. While we're about to consolidate these, this takes time. It'd be helpful if prometheus would allow to transform such labels, so that the issue can be mitigated by creating an evaluation rule.

I'm not sure if lowercase() and uppercase() make sense, or if we could come up with a more generic variant to replace any kind of characters (like tr). Or is there some trick to achieve that with label_replace already?

[fabxc]

To be clear, you are taking about PromQL here, not relabel configs?

[grobie]

Yes. Maybe in the form of replace(<vector>, <label>, <regex>, <replacement>), example replace(foo, service, "[A-Z]", "[a-z]").

The problem with "use configuration mangement" is that a static list of all label values would need to be known upfront.

[grobie]

Any other comments or concerns? I've been thinking about the function name for a bit now, maybe label_translate(<vector>, <label>, <regex>, <replacement>) in the style of the tr command? Other options value_replace or also value_translate.

[brian-brazil]

I think offering a tr variant would get too complicated, as we can't sanely do all the options in one function. Keeping it simple with upper/lowercase would be best.

value_replace and value_translate would operate on values not labels, and the value_ would be redundant as all functions operate on values unless they contain label.

[juliusv]

I'll add lower(str) and upper(str) functions if people are ok with that.

[grobie]

I'm not sure how that would work, can you give an example? How can I apply a function with such signature to a specific label value? For example to allow label matching with other metrics?

The reason I suggested a more generic translate function is that it'd cover another issue we have with a system which removes all dashes from a name. Though, the cardinality of values is way lower, so we might be able to solve that with rules.

I agree that value_ is misleading and ambiguous. I meant label_value_ here to make clear it's not replacing full labels but only certain characters of a single label value.

[juliusv]

Whoops, you're right, that wouldn't actually help for your use case. The question then is whether we want upper/lower methods at all or something more complex, or nothing.

[beorn7]

How about adding something to the syntax of the replacement parameter in the label_replace() function? We just have $<number> so far as special syntax. We could implement a subset of bash-style parameter expansion, starting just with the upper-/lower-case feature, which could easily be expanded as needed.

So

• ${1^} uppercases the 1st character of the 1st match
• ${1^^} uppercases the all characters of the 1st match
• ${1,} lowercases the 1st character of the 1st match
• ${1,,} leoercases the all characters of the 1st match

If needed one day, we could implement the full Bash-style ${parameter^pattern} etc. or any other parameter expansions as described in man bash.

[brian-brazil]

We just have $ so far as special syntax.

We're using Go's standard ExpandString function for regexes. Let's not badly reinvent perl just for one user's broken setup.

If we were to allow something this powerful, the sanest way would be to give direct access to console templates from promql.

[beorn7]

If we were to allow something this powerful, the sanest way would be to give direct access to console templates from promql.

That would be way more consistent. Do you have an idea how to do this concretely?

[brian-brazil]

Something like label_template(vector, dst_label, template)

It's already possible to access console templates via ALERTS, so we may push users that way rather than making it too easy to get them as there will be performance issues.

Console templates don't currently have case functions, but they're easy to add.

[beorn7]

That makes sense to me.

[juliusv]

Currently that'd introduce a circular dependency between the template and promql packages, because of the query template function. So introducing this would require refactoring some things into a third package. Just as an additional thing to consider when thinking about whether it's worth it.

[grobie]

I like the proposal, it's basically a more flexible label_replace.

[finkr]

I was hoping to be able to use a sed like syntax in label_replace, but unfortunately, the re2 library don't support syntax like \l, \L...\E, \u, \U...\E.

[skupjoe]

I was looking for this today +1

[brian-brazil]

Can you expand on why you were looking for it? Label values are meant to be opaque in Prometheus.

[bigangryrobot]

I ran into a need for this today and wanted to raise my thoughts. While I would agree with the theory of fixing at source instead of midstream, when you have over 100 prometheus instances in production, pushing label fixes brings time, risk and even assumes that you have control of the source when you may not, like in the case of federating other teams metrics. Being able to quickly address label changes so that evaluations will fire as expected is really essential.

My use case is mainly the severity labels i find in my infrastructure, where I have some teams using [pP0-9] (eg p1 or P1) or other variants on things

[roidelapluie]

My use case is mainly the severity labels i find in my infrastructure, where I have some teams using [pP0-9] (eg p1 or P1) or other variants on things

That sounds like something already supported.

metric_relabel_configs:
  source_labels: [priority]
  target_label: priority
  regex: 'p(.)'
  replacement: 'P${1}'

[bigangryrobot]

hmm thats actually a super slick solution, thank you. Though i wonder what other edge cases ill run into hah

[finkr]

Some use case:

You want to join two metrics from two exporters (a virtual machine name, and the guest operating system hostname), one of them is uppercase the other one is lowercase.

Same with mac address, uuid, etc .

You have multiple metrics with a label that is sometime uppercase, and sometime it's lowercase ( like an URL which are user input), you want to sum them and normalize the timeserie label.

Or you just want to normalize a label to lowecase.

[roidelapluie]

Some use case:

You want to join two metrics from two exporters (a virtual machine name, and the guest operating system hostname), one of them is uppercase the other one is lowercase.

Machine names should have fixed case.

Same with mac address, uuid, etc .

mac addresses and uuids should be lowercase.

You have multiple metrics with a label that is sometime uppercase, and sometime it's lowercase ( like an URL which are user input), you want to sum them and normalize the timeserie label.

User input should NEVER be in labels at it creates too much cardinality.

[skupjoe]

Some use case:
You want to join two metrics from two exporters (a virtual machine name, and the guest operating system hostname), one of them is uppercase the other one is lowercase.

Machine names should have fixed case.

Sometimes you do not have control over this and need to normalize with lower().

[ruccoon]

can we use upper/lowercase function in relabel_configs now? like

      - action: labelmap
        regex: __meta_ec2_tag_(.+)
        replacement:lower($1)

[xenji]

Some use case:
You want to join two metrics from two exporters (a virtual machine name, and the guest operating system hostname), one of them is uppercase the other one is lowercase.

Machine names should have fixed case.

Sometimes you do not have control over this and need to normalize with lower().

Example: You use AWS MSK and their JMX exporter feature. You don't have any access to their config and unfortunately the do not normalize the label names, so I get kafka_controller_ControllerChannelManager_95thPercentile, which basically breaks any existing dashboard and forces me to reimplment everything.

I see your point in this not being your problem at all... still, I would like Prometheus to be a tool that I can also use for non-perfect world environments.

[roidelapluie]

It seems there is no consensus for a PromQL solution.

I wonder if we could start with action: lower and action: upper in relabel configs. It would only change the label values, I feel like labels names should not require this.

[roidelapluie]

#10641 introduces the relabel actions.