Issue with relabelling after checkpointing is run

[chriswiggins]

What did you do?

Configured prometheus to scrape an SNMP Exporter endpoint

What did you expect to see?

Working continuously

What did you see instead? Under which circumstances?

As soon as prometheus does the following checkpointing from the log, the target relabelling on the SNMP targets is incorrectly set. On a restart, the target is set to the SNMP device IP, however after the checkpointing the target is set to the device running the SNMP exporter

time="2017-02-02T20:54:53Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:611"
time="2017-02-02T20:54:54Z" level=info msg="Done checkpointing in-memory metrics and chunks in 345.972951ms." source="persistence.go:638"

Environment

• System information:

  Linux 3.13.0-32-generic x86_64

• Prometheus version:

prometheus, version 1.5.0 (branch: master, revision: d840f2c400629a846b210cf58d65b9fbae0f1d5c)
  build user:       root@a04ed5b536e3
  build date:       20170123-13:56:24
  go version:       go1.7.4

• Prometheus configuration file:
  127.1.2.3 is just an example IP of the exporter

global:
  scrape_interval:     1m # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 1m # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
  external_labels:
      monitor: 'prom-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    honor_labels: true
    static_configs:
      - targets: ['localhost:9090', 'localhost:9100']
        labels:
          hostname: prometheus

  - job_name: 'ubnt-ap-snmp'
    scrape_interval: 1m
    file_sd_configs:
      - files:
        - /config/files/ubnt-ap-snmp.json
    metrics_path: /snmp
    params:
      module: [ubnt-ap]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.1.2.3:9116  # SNMP exporter.

  - job_name: 'netonix-sw-snmp'
    scrape_interval: 1m
    file_sd_configs:
      - files:
        - /config/files/netonix-sw-snmp.json
    metrics_path: /snmp
    params:
      module: [netonix-sw]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.1.2.3:9116  # SNMP exporter.

Sample of what is in the JSON files:

[{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} },
{"targets": ["x.x.x.x"], "labels": { "hostname" "hostname-here", "location": "TheLocation"} }]

• Logs:

See above section with the output.

I've tried to enable debug logging with -log.level=debug however this doesn't provide any more information.

If you'd like me to send more detailed information without replacement of IPs etc then I'm happy to do so.

Really appreciate the help!

[brian-brazil]

Sounds like a dupe of #2377. The checkpointing is unrelated.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.