Customize relabel configs

[dmgtn]

Prometheus Operator v0.17.0, Prometheus v2.2.1.

Problem

Prometheus Operator already does an awesome "automagic" with relabeling, but lack of ability to make any customization — is it good or bad?

• For now, Prometheus Operator adds following labels automatically: endpoint, instance, namespace, pod, and service. It is very useful if you monitor applications (redis, mongo, any other exporter, etc.), but not system components (kubelet, node-exporter, kube-scheduler, ...,) — system components do not need most of the labels (endpoint, namespace, pod, and service). Why shall we have waste labels on all metrics from system components? It makes things much harder, when you are writing alerts (you shall drop excess labels to make alerts meaningful) and when you are writing queries (it's harder to find meaningful labels between waste ones).
• We are trying to distinguish a type of an environment, and we add a special label tier=production|system|development|... to namespaces — we want to have that label attached to every metrics from applications (it's easy with relabel_configs). Yes, it is possible to do complex "joins" with kube-state-metrics, but we want to have simple and straight queries, that are easy to read and modify.
• We want to work with nodes by their name, not by their address (node_exporters are using the IP as instance name (should be hostname?) #135). So we want to add __meta_kubernetes_pod_node_name to all metrics from kubelet and node-exporter.
• Yes, there are metricRelabelings, but there are no access to __meta_* labels at this phase (or am I missing something?), so we can't do a lot.

Feature request

• Support for skipping unnecessary "auto" labels (disableAutoLabels: [] in ServiceMonitor/Endpoint?)
• Support for custom relabeling_configs (probably already done in Support for custom relabeling_configs #923).

[adamdecaf]

I want to attach a metric_relabel_config on a ServiceMonitor. It'd be nice to apply that against all metrics coming from the monitor.

[ryanash999]

I am running into the exact same issues. Based on my testing you are correct that metricRelabeling has no access to any label which hasn't already been built, i.e __meta. I wanted to update our instance to use the __meta_kubernetes_pod_node_name exactly how you referenced it.

[brancz]

you can use the new additionalConfig feature which lets you use all the features of the prometheus operator but gives you the ability to specify complete scrape configs including relabel configs. We felt that is a fair trade off of allowing everything users may want and a restricted environment that may be necessary with ServiceMonitors.

[dmgtn]

@brancz, no, please, no! Ability to choose what labels to add to targets and what not to add is very essential! If you force any predefined algorithm without an ability to customize — you limit your users. And if the only way to override that behavior is to complete disable automatic and use additionalConfig — what is the operator for? In such scenario, we don't need the operator at al!

[dmgtn]

As a user of prometheus-operator, I want to be able to customize labels to make the following things possible.

[brancz]

You're looking to actually add arbitrary relabeling, which is not something we intend to add as it creates more confusion with users who are new to Prometheus. If that's the level of customization you need, then please use the additional config. Is there any technical reason why you must have those label names, or is it merely a preference? I don't see an actual benefit over the labelings that you get out of the box and I'd prefer to have a homogeneous landscape of label names and values as it provides a much better base in order to share alerts and dashboarding.

[dmgtn]

I explained reasons in the first comment.

1. Ability to add extra labels from Kubernetes sd (for now it's impossible because there is no access to __meta_* on the metrics_relabeling stage).
2. When we scrape kubelet, node_exporter (and some other system components, that are deployed as daemonset's) — we do not need either pod names, neither IP-addresses — we need node names, because they are meaningful.

[dmgtn]

And, by the way, if we already have metrics_relabeling — why not add relabeling?

[huydinhle]

I really hope that this issue got reopened and worked on. Like @distol has said, this is a very essential functionality that prometheus-operator user should be able to have

[luisdavim]

@brancz why would this be confusing to users? I think that it is essential to be able to specify what labels to attach to the metrics without having to specify the whole configuration in additionalConfig can't we have something like additionalLabels?

[brancz]

That already exists as the targetLabels feature.

[luisdavim]

oh I was not aware of that, will this append tags to the list or replace the original list?

[brancz]

The targetLabels selected will be additive to the already existing labels. The whole point of all of this is that we can have a streamlined Kubernetes experience to have common alerting rules, dashboards etc. without having to mess with labelings, which is why we want to restrict differences in infrastructure as much as we can.

[hgraca]

@brancz Im trying to use the targetLabels to get the node name on node_memory_MemTotal output but it doesn't seem to work.

I get this back atm:

node_memory_MemTotal{
    endpoint="https",
    instance="10.1.94.43:9100",
    job="node-exporter",
    namespace="monitoring",
    pod="node-exporter-nk4pq",
    service="node-exporter"
}

and I would like to add to it something like node="kube-master-01" cozz the IP address means nothing to me.

I tried this:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: https
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: node-exporter
  targetLabels:
    - __meta_kubernetes_node_name

and this:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: https
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: node-exporter
  metricRelabelings:
    sourceLabels: __meta_kubernetes_node_name
    targetLabel: node

But nothing works :(

Any idea what is going wrong, or where I can check how to do it?

Tkx in advance, and I apologise if this is not the best place to ask this but I can't find anywhere else...

[brancz]

We're discussing that for any node specific targets this should just always be relabelled onto the target as it is a frequently requested feature. (Re: #1548 (comment))

Quick reminder though, that with the node-exporter you will never get this because that meta label does not exist for it, it's a normal daemonset with normal pods, which do not have that label during discovery. Only the kubelet has it with node discovery. In any other case you should do a join with kube-state-metrics metrics to get the Pod's node.

[paalkr]

@hgraca , you can have a look at how I managed to get the node labels added to my targets over at
camilb/prometheus-kubernetes#88

[primeroz]

@brancz proposing to use the "additional config" for such an important feature is crazy. Why would i even use the operator if i have to replace all servicemonitors with my custom configs ?

Is it really so hard to allow for extending the "relabel_configs" the same way that the "metric_relabel_configs" is done ?

[paalkr]

@primeroz

I totally agree with your concern. I can't understand why there is so much resistant to include support for __meta tags, role: node and customizing relabel_configs. If people are happy with the defaults, fine. But why not provide the flexibility to customize if needed?

[mxinden]

For your information #1879 just merged. Let me know if that solves the issues mentioned here.

[shahaf600]

anyone used it yet ? #1879

[daohoangson]

For future Googler, this is my monitor yaml which works with quay.io/coreos/prometheus-operator:v0.25.0:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: xxx
spec:
  endpoints:
  - port: http-metrics
    interval: 30s
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      targetLabel: instance
  namespaceSelector:
    matchNames:
    - xxx
  selector:
    matchLabels:
      service: xxx

[puneetmathur-87]

with respect to node-exporter o we have any work around for now to get the node name label in the metrics without additionalScrape config ?