Relabel Regex Match Groups not working in 1.6.2

[bfosberry]

What did you do?

Set up a Kubernetes SD scraper with a relabel regex

What did you expect to see?

Match groups get detected and used

What did you see instead? Under which circumstances?

Regex match groups are not being detected for relabels

Environment

Linux/Docker, local test env and on prod

• System information:

Linux 4.9.27-moby x86_64

• Prometheus version:

1.6.2

• Prometheus configuration file:

    - source_labels: [__meta_kubernetes_node_address_NodeInternalIP]
      target_label: 'instance'

If I use this config element then my "instance" label becomes "IP:10250", which is what I expect. I have node_exporter running on my k8s nodes, so I want to rewrite this as ":9100" before scraping, so I need to use regex.

    - source_labels: [__meta_kubernetes_node_address_NodeInternalIP]
      replacement: ':9100'
      target_label: 'instance'

If I use this config it'll, as expected, replace the entire ip:port with a static string, :9100

    - source_labels: [__meta_kubernetes_node_address_NodeInternalIP]
      regex: '([^:]+).*'
      replacement: '${1}:9100'
      target_label: 'instance'

If I use a match group to preserve the IP it does not match and the default instance is used.

I've done a decent amount of testing with this, and it seems that any time I try to use a match group, no matter how simple, it fails to match and the label either doesn't get updated or doesn't get created. This happens for both $1 and ${1} formats for the replacement field.

[brian-brazil]

Can you provide the exact value of the __meta_kubernetes_node_address_NodeInternalIP label in a case where it's failing?

[bfosberry]

Totally, doing a straight replace with no regex results in the following label:

instance="193.67.23.6:10250" (ip changed for anonymity)

I am assuming that the value of NodeInternalIP is 193.67.23.6:10250

[bfosberry]

If I change the regex to something simpler:

      regex: '.*(\d+).*'

Then the match fails and the instance label contains the default value, which is an internal hostname, not the IP + port

[brian-brazil]

This smells to me like that label isn't being set and/or is empty. If you hover over the labels on the target status page you can see what the discovered labels are.

[bfosberry]

You were absolutely right, I wasnt aware of that targets page, there was another field, address which was defaulting in some of the time which let me to think it was the regex failing.

If anyone comes across this issue again, the problem is that the docs are out of date:

__meta_kubernetes_node_address_NodeInternalIP

should be

__meta_kubernetes_node_address_InternalIP

thanks so much for the help!

[brian-brazil]

__address__ is the default value for instance, see https://www.robustperception.io/controlling-the-instance-label/

[bfosberry]

So without any relabelling the instance was being set to the hostname, not address, which is what led me to think it was the matching that was failing.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.