Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus-master can not query any metrics when labelmap and hashmod are both configured in Prometheus-slave #4872

Open
heayin opened this Issue Nov 16, 2018 · 0 comments

Comments

Projects
None yet
1 participant
@heayin
Copy link

heayin commented Nov 16, 2018

I am using Prometheus Federation in k8s.

The master configuration is as follows:

global:
  scrape_interval: 10s
  scrape_timeout: 10s
scrape_configs:
- job_name: 'dc_prometheus'
  honor_labels: true
  metrics_path: /federate
  params:
    match[]:
    - '{__name__=~".+"}'
  static_configs:
    - targets:
      - 10.181.12.19:30092
      - 10.181.12.19:30093

"10.181.12.19:30093" is the ip-address of slave1.
"10.181.12.19:30092" is the ip-address of slave0.

The slave1 configuration is as follows:

global:
  external_labels:
    slave: 1
  scrape_interval: 10s
  scrape_timeout: 10s
scrape_configs:
- job_name: 'kubernetes-pod'

  kubernetes_sd_configs:
  - role: pod

  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__]
    modulus: 2
    target_label: __tmp_hash
    action: hashmod
  - source_labels: [__tmp_hash]
    regex:     ^1$
    action: keep

I can query metrics successfully in slave1. But I can query nothing in master.
When querying in master, I tcpdumped "10.181.12.19:30093" . It showed that the slave sended the metrics to master.

However, I can query successfully in master when labelmap is removed in slave1 configuration .
The configuration is follows:

global:
  external_labels:
    slave: 1
  scrape_interval: 10s
  scrape_timeout: 10s
scrape_configs:
- job_name: 'kubernetes-pod'

  kubernetes_sd_configs:
  - role: pod

  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__]
    modulus: 2
    target_label: __tmp_hash
    action: hashmod
  - source_labels: [__tmp_hash]
    regex:     ^1$
    action: keep
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.