Question: Configure Prometheus to scrape metrics from k8s nginx-containers with nginx-exporter

[Blasterdick]

Can't get how to scrape pod-level labels only for pods with openresty and nginx-exporter containers deployed (port 9113).

Now I have default config, where job 'kubernetes-service-endpoints' scrapes metrics from nginx-exporter out-of-box, but I can't get how to scrape labels, such as pod_name, etc.

  - job_name: 'kubernetes-service-endpoints'
      scheme: http
      kubernetes_sd_configs:
      - api_servers:
        - 'https://kubernetes.default.svc'
        in_cluster: true
        role: endpoint
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)(?::\d+);(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_service_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

Also, I have such a job:

    - job_name: 'kubernetes-nginx-exporter'
      static_configs:
        - targets: ['nginx-exporter.default.svc.cluster.local:9113']

Which scrapes no container or pod labels at all (eventually?)

Thanks in advance for any kind of help.

[brancz]

The static configuration will never know about pods, and the static config points to a service, meaning there can be multiple pods load balanced behind it.

If you go to the /targets site of the Prometheus UI you can hover over the labels and it will display you all the meta labels belonging to the target, for targets discovered through kubernetes service discovery. Unfortunately endpoints do not have a direct reference to a pod yet.

We are soon going to look into improving kubernetes service discovery, so stay tuned for that.

In the mean time, maybe you can discover pods directly through pod discovery and group the job labels via relabelling.

[Blasterdick]

Hi @brancz,

Thanks for a prompt reply.

I'll give another try to a pod discovery, but, if you please, can you explain, how should I use relabelling to get in /targets#job-kubernetes-nginx-exporter only pods with 9113 port opened (not the trashy output for all the pods with all opened ports, which I got previously)?

Thanks in advance.

[brancz]

You could do the same thing as your configuration does with services/endpoints, where only endpoints of a service are kept where the service has the prometheus.io/scrape: true annotation. All annotations of a pod become a meta label in Prometheus, as I mentioned above you can see which ones there are by hovering over the labels on the targets page in the Prometheus UI.

[Blasterdick]

@brancz Thank you, mate. Closing issue.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.