Parameter labels (__param_X) lost on configuration reload

[raineth]

I am attempting to use the SNMP exporter, and my relabel_configs matches the example in the snmp_extractor README exactly.

My configuration works, but upon a configuration reload (SIGHUP) the query parameters are lost: they disappear from the endpoint's configuration on the status page, the actual queries sent to the snmp_exporter no longer have the 'address' parameter, and the targets are marked unhealthy because snmp_exporter is now (rightly) returning 400s.

[fabxc]

@brian-brazil mind giving this a look and whether you can reproduce.

[brian-brazil]

I've noticed some bug in this area previously, but haven't tracked it down.

[iwinux]

Got the same problem with Prometheus v0.16.0rc1 and blackbox_exporter: param target (which is added via relabel_configs) is missing after sending SIGHUP to Prometheus server.

The relabeling config is almost the same as the example from https://github.com/prometheus/blackbox_exporter.

[iwinux]

Here's how to reproduce:

config

# prometheus.yml
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

scrape_configs:
  - job_name: blackbox-icmp
    metrics_path: /probe
    params:
      module: [icmp]
    target_groups:
      - targets:
        - host1
        - host2
    relabel_configs:
      - source_labels: [__address__]
        regex: '([^:]+)(:\d+)?'
        target_label: __param_target
        replacement: ${1}

      - source_labels: [__param_target]
        regex: '(.*)'
        target_label: instance
        replacement: ${1}

      - source_labels: []
        regex: '.*'
        target_label: __address__
        replacement: localhost:9115

steps

1. start up Prometheus server at localhost:9100

2. verify that blackbox-icmp targets are all up:

   curl -s "http://localhost:9100/api/v1/query?query=up&time=`date --utc '+%Y-%m-%dT%H:%M:%S.000Z'`" | jq '.data.result[] | .metric.instance + " " + .metric.job + " " + .value[1]'

   you should see output like this:

   host1 blackbox-icmp 1
   host1 blackbox-icmp 1
   

3. killall -HUP prometheus

4. repeat step 2 (waiting for a few seconds?), this time you'll see:

   host1 blackbox-icmp 0
   host1 blackbox-icmp 0
   

5. restart Prometheus server and those targets will come up as healthy again.

[fabxc]

Awesome, thanks!

[brian-brazil]

The problem here is fullLabels() not including the parameters. I'm preparing a fix.

[fabxc]

Cool, thanks!

On Sat, Sep 26, 2015 at 5:33 PM Brian Brazil notifications@github.com
wrote:

The problem here is fullLabels() not including the parameters. I'm
preparing a fix.

—
Reply to this email directly or view it on GitHub
#1100 (comment)
.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.