Can not see relabeled labels in queries

[widgetpl]

What did you expect to see?
relabeled labels in queries

What did you see instead? Under which circumstances?
can not see relabeled labels in queries

• Prometheus version:

  quay.io/prometheus/prometheus:v1.3.0

• Prometheus configuration file:

    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints

      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)(?::\d+);(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_label: [__meta_kubernetes_pod_host_ip]
        action: replace
        target_label: host_ip
      - source_labels: [__meta_kubernetes_service_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

During quierying i can not see relabeled labels in job configuration, f.e querying node_load1 i get:

node_load1{app="node-exporter",instance="10.116.166.4:9100",job="kubernetes-service-endpoints",kubernetes_name="node-exporter",name="node-exporter"}	3.21
node_load1{app="node-exporter",instance="10.116.171.30:9100",job="kubernetes-service-endpoints",kubernetes_name="node-exporter",name="node-exporter"}	2.87
node_load1{app="node-exporter",instance="10.116.179.11:9100",job="kubernetes-service-endpoints",kubernetes_name="node-exporter",name="node-exporter"}	4.37
node_load1{app="node-exporter",instance="10.116.135.29:9100",job="kubernetes-service-endpoints",kubernetes_name="node-exporter",name="node-exporter"}	2.13

so i don't see my label:

- source_label: [__meta_kubernetes_pod_host_ip]
        action: replace
        target_label: host_ip

[brancz]

That's interesting! I just tried this on a local minikube cluster and I don't seem to have the meta label __meta_kubernetes_pod_host_ip available either. I'll investigate!

As a sidenote the replace action is the default, so you can leave it out in this case if you want to.

[brancz]

I was a bit too quick there. It does seem to be there for me: __meta_kubernetes_pod_host_ip="192.168.99.100". I'll try to reproduce with the exact same config.

[widgetpl]

I use kubernetes 1.3.4 btw

[widgetpl]

and __meta_kubernetes_pod_host_ip is not the only one label that i don't see there is also __meta_kubernetes_service_namespace and probably few others.
I use node-exporter as a daemon in kubernetes from CoreOS manifest

[brancz]

There is a typo in your config source_label -> source_labels 🙂

[widgetpl]

what about

- source_labels: [__meta_kubernetes_service_namespace]
        action: replace
        target_label: kubernetes_namespace

[brancz]

That would actually just be __meta_kubernetes_namespace as selection in kubernetes is namespace scoped, so it doesn't make much sense to include __meta_kubernetes_endpoints_namespace, __meta_kubernetes_service_namespace and __meta_kubernetes_pod_namespace just to get the same value in all of them 🙂

[brancz]

Is there anything else that doesn't seem to work @widgetpl ?

[widgetpl]

I am trying to reload Prometheus configuration with curl -XPOST http://172.16.239.3:9090//-/reload but it seems that it dosen't work. I havent setup any persistent storage for Prometheus yet so probably I will loose all the data. What about __meta_kubernetes_endpoints_namespace, __meta_kubernetes_service_namespace and __meta_kubernetes_pod_namespace i don't even see them in Prometheus before relabeling in Kubernetes 1.3.4 but they are used in default configuration what I see is __meta_kubernetes_namespace instead

[brancz]

Yes that's what I said, __meta_kubernetes_endpoints_namespace, __meta_kubernetes_service_namespace and __meta_kubernetes_pod_namespace don't exist. It's only __meta_kubernetes_namespace. Prometheus doesn't reload your config if there is an error. Have a look at the logs and the prometheus_config_last_reload_successful as well as the prometheus_config_last_reload_success_timestamp_seconds metrics exposed by Prometheus. But to make sure before trying to reload you can always use promtool to check whether your config is valid.

[widgetpl]

thanks for advice about promtool as i haven't heard about that. It looks that everything is ok now so issue can be closed.

[widgetpl]

@brancz just one more question. I have updated configuration in ConfigMap and checked it with promtool inside Prometheus pod but i don't see any updates in Prometheus /targets UI.
Does curl -XPOST http://172.16.239.3:9090//-/reload is the correct way of reloading configuration?

[brancz]

I'm not sure it makes a difference but I use http://localhost:9090/-/reload, so maybe there is one / to much in your url. And make sure that Prometheus successfully loaded the config by looking at the logs.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.