Support differentiation between empty label and non-existing labels in k8

[discordianfish]

What did you do?
I've tried to use the Kubernetes SD and relabling to only scrape master nodes. In Kubernetes, master nodes are tagged with the label: node-role.kubernetes.io/master: "" which becomes __meta_kubernetes_node_label_node_role_kubernetes_io_master="" in Prometheus after sanitizing.

Unfortunately this doesn't seem to be possible with relabling because it's not possible to differentiate between a nonexisting and empty label. This for example will always match, since if the key doesn't exist the regexp gets matched against "".

• Prometheus version: 2.1

[brian-brazil]

This is not something we can really handle in relabelling, however service discovery could provide additional metadata to make this possible.

[cstyan]

What would the solution look like here? A new Role in discovery/kubernetes.go to get nodes but drop ones without ode-role.kubernetes.io/master: ""? RoleMaster? Or should we set an explicit value true for that meta label?

[brian-brazil]

We'd either add a new metadata entry that is a list with a list of all labels/annotations that are present, or a label per k8 label that has the value true.

[discordianfish]

Not sure how the "label per k8 label that has the value true" should work (the value is "" for these labels) but a entry with a list of all labels/annotations sounds like a good idea. 👍

[brian-brazil]

I'm saying we could represent this as either a list, or a map. A map may be easier for users to use, but is more verbose.

[discordianfish]

Ah got it.. hrm Isee, yeah checking if something is set or not would be indeed easier to do with a map. I'm fine either way.

[Quentin-M]

+1

This makes it hard to monitor master and worker nodes separately.

[brancz]

At least for all the "official" *.kubernetes.io type annotations/labels, this should be ok to have as metadata in the SD.

[nevill]

@brancz Unfortunately, node-role.kubernetes.io/master is not a member of well_known_labels (kubelet, scheduler, etc). It's usually set during mark control plane phase when kubeadm is running. Shall we still consider it as official ?

[Quentin-M]

@nevill Not because it is not part of the codebase, that it is not the upstream standard.