Missing some metrics in prometheus

[yuzhiquan]

What did you do?
i install prometheus and grafana in k8s, and i want some graph monitoring the status of CPU、Mem, the Node and the container.
What did you expect to see?

What did you see instead? Under which circumstances?
But the container's graph is blank, and without data, the Node's graph is ok.
and the metrics in prometheus does not has the metrics such as container_xxxx_xxxx

so, i visit the nodeIP:4194/metrics, i got this:

ersion & cadvisor revision.

TYPE cadvisor_version_info gauge

cadvisor_version_info{cadvisorRevision="",cadvisorVersion="",dockerVersion="17.09.0-ce",kernelVersion="4.4.0-62-generic",osVersion="Ubuntu 16.04.3 LTS"} 1

HELP container_cpu_load_average_10s Value of container cpu load average over the last 10 seconds.

TYPE container_cpu_load_average_10s gauge

container_cpu_load_average_10s{container_name="",id="/",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/docker",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/docker/11a1cb87706e9b4f16ce5884ee7b0da46a298d4f85c0068a129056b3aef91efd",image="calico/node:v2.6.2",name="calico-node",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/init.scope",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/kube-proxy",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/kubepods",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/kubepods/besteffort",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/kubepods/besteffort/poda6606d8d-e0b4-11e7-9793-00163e0c0192",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/kubepods/besteffort/podd3f3ab95-e098-11e7-9793-00163e0c0192",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/kubepods/besteffort/podfa4dda06-e491-11e7-9793-00163e0c0192",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/kubepods/besteffort/podfaa5ae07-e491-11e7-9793-00163e0c0192",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/kubepods/burstable",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/accounts-daemon.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/aegis.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/agentwatch.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/apparmor.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/atd.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/calico-node.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/cloud-config.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/cloud-final.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/cloud-init-local.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/cloud-init.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/console-setup.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/cron.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/dbus.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/docker.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/grub-common.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/ifup@eth0.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/irqbalance.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/keyboard-setup.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/kmod-static-nodes.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/kube-proxy.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/kubelet.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/networking.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/nscd.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/ntp.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/ondemand.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/rc-local.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/resolvconf.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/rsyslog.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/run-r0ee7a8f62da743e4a5e8639aaebe4e0b.scope",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/run-r4ddcb99c044d4cb1a59f6fb993db2aaa.scope",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/run-rac2bfac20382421ab42bddae02068252.scope",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/run-rb13e8936a2584deea5c32f81058dd692.scope",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/setvtrgb.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/ssh.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/sysstat.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/system-getty.slice",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-journal-flush.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-journald.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-logind.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-modules-load.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-random-seed.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-remount-fs.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-sysctl.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-tmpfiles-setup-dev.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-tmpfiles-setup.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-udev-trigger.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-udevd.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-update-utmp.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/systemd-user-sessions.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/system.slice/uuidd.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/user.slice",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/user.slice/user-0.slice",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/user.slice/user-0.slice/session-1203.scope",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="",id="/user.slice/user-0.slice/user@0.service",image="",name="",namespace="",pod_name=""} 0
container_cpu_load_average_10s{container_name="POD",id="/kubepods/besteffort/poda6606d8d-e0b4-11e7-9793-00163e0c0192/f15d7337adfd997e365ade1d9606122ff026f3fdf320e4080fff4f12384d75bb",image="mirrorgooglecontainers/pause-amd64:3.0",name="k8s_POD_mc-fs-crypted-5fbb99dbd6-sc9tz_staging_a6606d8d-e0b4-11e7-9793-00163e0c0192_1",namespace="staging",pod_name="mc-fs-crypted-5fbb99dbd6-sc9tz"} 0
container_cpu_load_average_10s{container_name="POD",id="/kubepods/besteffort/podd3f3ab95-e098-11e7-9793-00163e0c0192/3820e6b18214e821cc458f1dfa52879bbc77ef2cb86b8ca53374dfc7af59456b",image="mirrorgooglecontainers/pause-amd64:3.0",name="k8s_POD_heapster-6956dd7956-52l7k_kube-system_d3f3ab95-e098-11e7-9793-00163e0c0192_1",namespace="kube-system",pod_name="heapster-6956dd7956-52l7k"} 0
container_cpu_load_average_10s{container_name="POD",id="/kubepods/besteffort/podfa4dda06-e491-11e7-9793-00163e0c0192/43efaf47d46104b21eafffe10607a58916f6099d571f0cf04c6a81e7ed3a551b",image="mirrorgooglecontainers/pause-amd64:3.0",name="k8s_POD_prometheus-node-exporter-fn9cd_monitoring_fa4dda06-e491-11e7-9793-00163e0c0192_0",namespace="monitoring",pod_name="prometheus-node-exporter-fn9cd"} 0
container_cpu_load_average_10s{container_name="POD",id="/kubepods/besteffort/podfaa5ae07-e491-11e7-9793-00163e0c0192/c4494774f5b53cb63c669166b99808ec5ab2c2e5bb1b9a4772120c7ecea49d5f",image="mirrorgooglecontainers/pause-amd64:3.0",name="k8s_POD_node-directory-size-metrics-rgf46_monitoring_faa5ae07-e491-11e7-9793-00163e0c0192_0",namespace="monitoring",pod_name="node-directory-size-metrics-rgf46"} 0
container_cpu_load_average_10s{container_name="caddy",id="/kubepods/besteffort/podfaa5ae07-e491-11e7-9793-00163e0c0192/949887dfb82288f9b334908c4d75ddba7cb74cb4366c0797c99d95829244d441",image="dockermuenster/caddy@sha256:34899a9cf74fd3943b128e2ce65e08068ec7dcbf3631c5f08c64b483fbf8171b",name="k8s_caddy_node-directory-size-metrics-rgf46_monitoring_faa5ae07-e491-11e7-9793-00163e0c0192_0",namespace="monitoring",pod_name="node-directory-size-metrics-rgf46"} 0
container_cpu_load_average_10s{container_name="crypted",id="/kubepods/besteffort/poda6606d8d-e0b4-11e7-9793-00163e0c0192/0e87abb82923fb6fcb20c16341e190839377a0af307b92047de47e6ce8fd8991",image="harbor.csg-bjv.megvii-inc.com/megvii/fs-crypted@sha256:8579027bed75f35a2f57246f46754003f17d6bc3cc3940852a2f634b4db52de2",name="k8s_crypted_mc-fs-crypted-5fbb99dbd6-sc9tz_staging_a6606d8d-e0b4-11e7-9793-00163e0c0192_1",namespace="staging",pod_name="mc-fs-crypted-5fbb99dbd6-sc9tz"} 0
container_cpu_load_average_10s{container_name="heapster",id="/kubepods/besteffort/podd3f3ab95-e098-11e7-9793-00163e0c0192/634ea11c5ed62c171798e803f74e76f8d503f6cd77d503876709e58db579f015",image="mirrorgooglecontainers/heapster-amd64@sha256:3dff9b2425a196aa51df0cebde0f8b427388425ba84568721acf416fa003cd5c",name="k8s_heapster_heapster-6956dd7956-52l7k_kube-system_d3f3ab95-e098-11e7-9793-00163e0c0192_1",namespace="kube-system",pod_name="heapster-6956dd7956-52l7k"} 0
container_cpu_load_average_10s{container_name="prometheus-node-exporter",id="/kubepods/besteffort/podfa4dda06-e491-11e7-9793-00163e0c0192/79751fa851833cc01f162f11ae6aaec039136068b10c66ddf90ac7a30be9318c",image="prom/node-

Environment
k8s: 1.8.4
grafana: 4.2
prometheus: 1.8.2

• System information:

  insert output of uname -srm here
  Linux 4.4.0-62-generic x86_64

• Prometheus version:

  insert output of prometheus --version here

• Alertmanager version:

  insert output of alertmanager --version here (if relevant to the issue)

• Prometheus configuration file:

insert configuration here

A scrape configuration for running Prometheus on a Kubernetes cluster.
# This uses separate scrape configs for cluster components (i.e. API server, node)
# and services to allow each to use different authentication configs.
#
# Kubernetes labels will be added as Prometheus labels on metrics via the
# `labelmap` relabeling action.
#
# If you are using Kubernetes 1.7.2 or earlier, please take note of the comments
# for the kubernetes-cadvisor job; you will need to edit or remove this job.

# Scrape config for API servers.
#
# Kubernetes exposes API servers as endpoints to the default/kubernetes
# service so this uses `endpoints` role and uses relabelling to only keep
# the endpoints associated with the default/kubernetes service using the
# default named port `https`. This works for single API server deployments as
# well as HA API server deployments.
apiVersion: v1
data:
  prometheus.yaml: |
    global:
      scrape_interval: 10s
      scrape_timeout: 10s
      evaluation_interval: 10s
    rule_files:
      - "/etc/prometheus-rules/*.rules"
    scrape_configs:
    - job_name: 'kubernetes-apiservers'

      kubernetes_sd_configs:
      - role: endpoints

      # Default to scraping over https. If required, just disable this or change to
      # `http`.
      scheme: https

      # This TLS & bearer token file config is used to connect to the actual scrape
      # endpoints for cluster components. This is separate to discovery auth
      # configuration because discovery & scraping are two separate concerns in
      # Prometheus. The discovery auth config is automatic if Prometheus runs inside
      # the cluster. Otherwise, more config options have to be provided within the
      # <kubernetes_sd_config>.
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        # If your node certificates are self-signed or use a different CA to the
        # master CA, then disable certificate verification below. Note that
        # certificate verification is an integral part of a secure infrastructure
        # so this should only be disabled in a controlled environment. You can
        # disable certificate verification by uncommenting the line below.
        #
        # insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      # Keep only the default/kubernetes service endpoints for the https port. This
      # will add targets for each API server which Kubernetes adds an endpoint to
      # the default/kubernetes service.
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

    # Scrape config for nodes (kubelet).
    #
    # Rather than connecting directly to the node, the scrape is proxied though the
    # Kubernetes apiserver.  This means it will work if Prometheus is running out of
    # cluster, or can't connect to nodes for some other reason (e.g. because of
    # firewalling).
    - job_name: 'kubernetes-nodes'

      # Default to scraping over https. If required, just disable this or change to
      # `http`.
      scheme: https

      # This TLS & bearer token file config is used to connect to the actual scrape
      # endpoints for cluster components. This is separate to discovery auth
      # configuration because discovery & scraping are two separate concerns in
      # Prometheus. The discovery auth config is automatic if Prometheus runs inside
      # the cluster. Otherwise, more config options have to be provided within the
      # <kubernetes_sd_config>.
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics

    # Scrape config for Kubelet cAdvisor.
    #
    # This is required for Kubernetes 1.7.3 and later, where cAdvisor metrics
    # (those whose names begin with 'container_') have been removed from the
    # Kubelet metrics endpoint.  This job scrapes the cAdvisor endpoint to
    # retrieve those metrics.
    #
    # In Kubernetes 1.7.0-1.7.2, these metrics are only exposed on the cAdvisor
    # HTTP endpoint; use "replacement: /api/v1/nodes/${1}:4194/proxy/metrics"
    # in that case (and ensure cAdvisor's HTTP server hasn't been disabled with
    # the --cadvisor-port=0 Kubelet flag).
    #
    # This job is not necessary and should be removed in Kubernetes 1.6 and
    # earlier versions, or it will cause the metrics to be scraped twice.
    - job_name: 'kubernetes-cadvisor'

      # Default to scraping over https. If required, just disable this or change to
      # `http`.
      scheme: https

      # This TLS & bearer token file config is used to connect to the actual scrape
      # endpoints for cluster components. This is separate to discovery auth
      # configuration because discovery & scraping are two separate concerns in
      # Prometheus. The discovery auth config is automatic if Prometheus runs inside
      # the cluster. Otherwise, more config options have to be provided within the
      # <kubernetes_sd_config>.
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

    # Scrape config for service endpoints.
    #
    # The relabeling allows the actual service scrape endpoint to be configured
    # via the following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape services that have a value of `true`
    # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
    # to set this to `https` & most likely set the `tls_config` of the scrape config.
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: If the metrics are exposed on a different port to the
    # service then set this appropriately.
    - job_name: 'kubernetes-service-endpoints'

      kubernetes_sd_configs:
      - role: endpoints

      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

    # Example scrape config for probing services via the Blackbox Exporter.
    #
    # The relabeling allows the actual service scrape endpoint to be configured
    # via the following annotations:
    #
    # * `prometheus.io/probe`: Only probe services that have a value of `true`
    - job_name: 'kubernetes-services'

      metrics_path: /probe
      params:
        module: [http_2xx]

      kubernetes_sd_configs:
      - role: service

      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-exporter.example.com:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: kubernetes_name

    # Example scrape config for probing ingresses via the Blackbox Exporter.
    #
    # The relabeling allows the actual ingress scrape endpoint to be configured
    # via the following annotations:
    #
    # * `prometheus.io/probe`: Only probe services that have a value of `true`
    - job_name: 'kubernetes-ingresses'

      metrics_path: /probe
      params:
        module: [http_2xx]

      kubernetes_sd_configs:
        - role: ingress

      relabel_configs:
        - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
          regex: (.+);(.+);(.+)
          replacement: ${1}://${2}${3}
          target_label: __param_target
        - target_label: __address__
          replacement: blackbox-exporter.example.com:9115
        - source_labels: [__param_target]
          target_label: instance
        - action: labelmap
          regex: __meta_kubernetes_ingress_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_ingress_name]
          target_label: kubernetes_name

    # Example scrape config for pods
    #
    # The relabeling allows the actual pod scrape endpoint to be configured via the
    # following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the
    # pod's declared ports (default is a port-free target if none are declared).
    - job_name: 'kubernetes-pods'

      kubernetes_sd_configs:
      - role: pod

      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: prometheus-core
  namespace: monitoring

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

wait for help , thanks!

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.