Customize relabel configs

[dmgtn]

Prometheus Operator v0.17.0, Prometheus v2.2.1.

Problem

Prometheus Operator already does an awesome "automagic" with relabeling, but lack of ability to make any customization — is it good or bad?

• For now, Prometheus Operator adds following labels automatically: endpoint, instance, namespace, pod, and service. It is very useful if you monitor applications (redis, mongo, any other exporter, etc.), but not system components (kubelet, node-exporter, kube-scheduler, ...,) — system components do not need most of the labels (endpoint, namespace, pod, and service). Why shall we have waste labels on all metrics from system components? It makes things much harder, when you are writing alerts (you shall drop excess labels to make alerts meaningful) and when you are writing queries (it's harder to find meaningful labels between waste ones).
• We are trying to distinguish a type of an environment, and we add a special label tier=production|system|development|... to namespaces — we want to have that label attached to every metrics from applications (it's easy with relabel_configs). Yes, it is possible to do complex "joins" with kube-state-metrics, but we want to have simple and straight queries, that are easy to read and modify.
• We want to work with nodes by their name, not by their address (node_exporters are using the IP as instance name (should be hostname?) #135). So we want to add __meta_kubernetes_pod_node_name to all metrics from kubelet and node-exporter.
• Yes, there are metricRelabelings, but there are no access to __meta_* labels at this phase (or am I missing something?), so we can't do a lot.

Feature request

• Support for skipping unnecessary "auto" labels (disableAutoLabels: [] in ServiceMonitor/Endpoint?)
• Support for custom relabeling_configs (probably already done in Support for custom relabeling_configs #923).

[adamdecaf]

I want to attach a metric_relabel_config on a ServiceMonitor. It'd be nice to apply that against all metrics coming from the monitor.

[ryanash999]

I am running into the exact same issues. Based on my testing you are correct that metricRelabeling has no access to any label which hasn't already been built, i.e __meta. I wanted to update our instance to use the __meta_kubernetes_pod_node_name exactly how you referenced it.

[brancz]

you can use the new additionalConfig feature which lets you use all the features of the prometheus operator but gives you the ability to specify complete scrape configs including relabel configs. We felt that is a fair trade off of allowing everything users may want and a restricted environment that may be necessary with ServiceMonitors.

[dmgtn]

@brancz, no, please, no! Ability to choose what labels to add to targets and what not to add is very essential! If you force any predefined algorithm without an ability to customize — you limit your users. And if the only way to override that behavior is to complete disable automatic and use additionalConfig — what is the operator for? In such scenario, we don't need the operator at al!

[dmgtn]

As a user of prometheus-operator, I want to be able to customize labels to make the following things possible.

[brancz]

You're looking to actually add arbitrary relabeling, which is not something we intend to add as it creates more confusion with users who are new to Prometheus. If that's the level of customization you need, then please use the additional config. Is there any technical reason why you must have those label names, or is it merely a preference? I don't see an actual benefit over the labelings that you get out of the box and I'd prefer to have a homogeneous landscape of label names and values as it provides a much better base in order to share alerts and dashboarding.

[dmgtn]

I explained reasons in the first comment.

1. Ability to add extra labels from Kubernetes sd (for now it's impossible because there is no access to __meta_* on the metrics_relabeling stage).
2. When we scrape kubelet, node_exporter (and some other system components, that are deployed as daemonset's) — we do not need either pod names, neither IP-addresses — we need node names, because they are meaningful.

[dmgtn]

And, by the way, if we already have metrics_relabeling — why not add relabeling?

[huydinhle]

I really hope that this issue got reopened and worked on. Like @distol has said, this is a very essential functionality that prometheus-operator user should be able to have

[luisdavim]

@brancz why would this be confusing to users? I think that it is essential to be able to specify what labels to attach to the metrics without having to specify the whole configuration in additionalConfig can't we have something like additionalLabels?

[brancz]

That already exists as the targetLabels feature.

[luisdavim]

oh I was not aware of that, will this append tags to the list or replace the original list?

[brancz]

The targetLabels selected will be additive to the already existing labels. The whole point of all of this is that we can have a streamlined Kubernetes experience to have common alerting rules, dashboards etc. without having to mess with labelings, which is why we want to restrict differences in infrastructure as much as we can.

[hgraca]

@brancz Im trying to use the targetLabels to get the node name on node_memory_MemTotal output but it doesn't seem to work.

I get this back atm:

node_memory_MemTotal{
    endpoint="https",
    instance="10.1.94.43:9100",
    job="node-exporter",
    namespace="monitoring",
    pod="node-exporter-nk4pq",
    service="node-exporter"
}

and I would like to add to it something like node="kube-master-01" cozz the IP address means nothing to me.

I tried this:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: https
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: node-exporter
  targetLabels:
    - __meta_kubernetes_node_name

and this:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: https
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: node-exporter
  metricRelabelings:
    sourceLabels: __meta_kubernetes_node_name
    targetLabel: node

But nothing works :(

Any idea what is going wrong, or where I can check how to do it?

Tkx in advance, and I apologise if this is not the best place to ask this but I can't find anywhere else...

[brancz]

We're discussing that for any node specific targets this should just always be relabelled onto the target as it is a frequently requested feature. (Re: #1548 (comment))

Quick reminder though, that with the node-exporter you will never get this because that meta label does not exist for it, it's a normal daemonset with normal pods, which do not have that label during discovery. Only the kubelet has it with node discovery. In any other case you should do a join with kube-state-metrics metrics to get the Pod's node.

[paalkr]

@hgraca , you can have a look at how I managed to get the node labels added to my targets over at
camilb/prometheus-kubernetes#88

[primeroz]

@brancz proposing to use the "additional config" for such an important feature is crazy. Why would i even use the operator if i have to replace all servicemonitors with my custom configs ?

Is it really so hard to allow for extending the "relabel_configs" the same way that the "metric_relabel_configs" is done ?

[paalkr]

@primeroz

I totally agree with your concern. I can't understand why there is so much resistant to include support for __meta tags, role: node and customizing relabel_configs. If people are happy with the defaults, fine. But why not provide the flexibility to customize if needed?

[mxinden]

For your information #1879 just merged. Let me know if that solves the issues mentioned here.

[shahaf600]

anyone used it yet ? #1879

[daohoangson]

For future Googler, this is my monitor yaml which works with quay.io/coreos/prometheus-operator:v0.25.0:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: xxx
spec:
  endpoints:
  - port: http-metrics
    interval: 30s
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      targetLabel: instance
  namespaceSelector:
    matchNames:
    - xxx
  selector:
    matchLabels:
      service: xxx

[puneetmathur-87]

with respect to node-exporter o we have any work around for now to get the node name label in the metrics without additionalScrape config ?

[msiebuhr]

My prometheus-knowledge is a bit rusty, so I might be misunderstanding something here about the many stages of relabelling and how things are pieced together. My motivation so far:

I'm on a team running an internal Kubernets platform used for ~100 of services belonging to all sorts of formal and informal teams. To keep track, we've begun tagging a project-id (or project_id when dashes are no-go) in metadata wherever possible; short-term, we just want to know where alerts should go, but long-term we would also like to build dashboards that can show devs/PM's key data, cost-estimates and so on.

But for Kubernetes, we have hit a problem with prometheus and the prometheus-operator:

1. Our automation can set a label on all Kubernetes namespaces with the project-id, e.g. k8s.example.com/project-id: j0y1dd. (After that devs, get ~unrestricted access to the namespace, so we don't control much in there.)
2. Have the Prometheus Kubernetes Service Discovery pick up the namespace-labels. Kubernetes SD: Optionally attach namespace and node metadata to targets prometheus/prometheus#9510 blocks ATM, so I have Kyverno copy the label to all newly created pods (ex. https://kyverno.io/policies/other/copy-namespace-labels/copy-namespace-labels/). Not pretty, but it does get the label near where I scrape it, without getting each and every dev-team to manually label their pods correctly and enforce it on our end.
3. Attach a relabel-config to all ServiceMonitors that sticks the __meta_kube...label_k8s_example_com_project_id label onto the scraped data as project_id. (AFAIK, this issue). Again, I would like to keep it simple and "just works". Having devs copy/paste in a stub is again error-prone and hard to evolve for us - we'd have to re-do the whole thing if/when anything changes.

From my point-of-view, allowing us to do cluster-wide relabeling on scrape feels like a very obvious simplification for us and our users. I'm quite surprised it's being rejected on grounds of adding complexity!?

Alternatives:

• Like we use Kyverno to move Kubernetes-labels from namespace to pods, I've considered having it re-write all ServiceMonitors to include the required relabelings-stanza. but (1) we want to get out of Kyverno (2) hard to evolve over time (3) tends to make Argo and friends unhappy.
• I have the namespace-to-project-id map externally; write giant rewrite-rules that match against namespace label and attach the matching project_id. But it's my impression Prometheus isn't happy with several hundred rewrite-rules, so I'm a bit hesitant to go that way.

Anybody else got ideas?

[simonpasquier]

@msiebuhr you should look into ScrapeClass which allows to append global config to service/pod monitors.