Targets are missing when multiple scrape configs using kubernetes_sd are defined

[simonpasquier]

What did you do?

Configure Prometheus with several scrape configurations all using kubernetes_sd:

• one for scraping the prometheus endpoints
• one for scraping the alertmanager endpoints
• one for scraping the alertbuffer endpoints
• one for scraping my application endpoints

What did you expect to see?

All endpoints are detected by the service discovery and Prometheus scrapes them.

What did you see instead? Under which circumstances?

Instead I see only one group of endpoints (Prometheus endpoints for instance). If there's a change for one pod (eg it gets recreated after deletion) then it gets detected by Prometheus but the previous targets are gone.

The same Prometheus configuration works perfectly fine with the 2.0.0 version.

Environment

• System information:

Linux 4.4.0-104-generic x86_64

• Prometheus version:

prometheus, version 2.0.0 (branch: master, revision: ec94df4)
build user: pasquier-s@simon-trusty
build date: 20180104-09:35:39
go version: go1.9.2

• Alertmanager version:

N/A

• Prometheus configuration file:

      global:
        scrape_interval: 10s
      rule_files:
        - 'prometheus.rules'
      alerting:
        alertmanagers:
        - scheme: https
          tls_config:
            # the certificate is generated for the service, not the endpoint itself
            insecure_skip_verify: true
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
          - role: endpoints
            namespaces:
              names:
              - "${NAMESPACE}"
          relabel_configs:
          - source_labels:
            - __meta_kubernetes_service_name
            - __meta_kubernetes_endpoint_ready
            action: keep
            regex: alertmanager;true
      scrape_configs:
      - job_name: 'prometheus'
        scheme: https
        tls_config:
          # the certificate is generated for the service, not the endpoint itself
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - "${NAMESPACE}"
        relabel_configs:
        # metrics are pulled through the OAuth proxy container
        - source_labels:
          - __meta_kubernetes_service_name
          - __meta_kubernetes_endpoint_ready
          - __meta_kubernetes_pod_container_name
          action: keep
          regex: prometheus;true;proxy
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: instance
      - job_name: 'alertmanager'
        scheme: https
        tls_config:
          # the certificate is generated for the service, not the endpoint itself
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - "${NAMESPACE}"
        relabel_configs:
        - source_labels:
          - __meta_kubernetes_service_name
          - __meta_kubernetes_endpoint_ready
          - __meta_kubernetes_pod_container_name
          action: keep
          regex: alertmanager;true;am-proxy
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: instance
      - job_name: 'alertbuffer'
        scheme: https
        tls_config:
          # the certificate is generated for the service, not the endpoint itself
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - "${NAMESPACE}"
        relabel_configs:
        - source_labels:
          - __meta_kubernetes_service_name
          - __meta_kubernetes_endpoint_ready
          - __meta_kubernetes_pod_container_name
          action: keep
          regex: alertbuffer;true;proxy
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: instance
      - job_name: 'endpoints'
        tls_config:
          insecure_skip_verify: true
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - "${NAMESPACE}"
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_endpoint_ready]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: (.+)(?::\d+);(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: service_name

[simonpasquier]

cc @krasi-georgiev

[krasi-georgiev]

I could replicate and it also happens with a single kube scrape job.

the initial sync send all targets , but any additional sync seems to send only the updates.

[simonpasquier]

The corresponding PR is #3654

[simonpasquier]

Fixed by #3654.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.