relabeling ignores __name__ when replacing __address__

[amnonh]

Bug Report

What did you do?
I'm using my server ip address configuration for node_exporter.
To be able to support both node_exporter 0.14 and node_exporter 0.17 I'm renaming the metrics to 0.14.

  relabel_configs:
    - source_labels: [__address__]
      regex:  '(.*):\d+'
      target_label: instance
      replacement: '${1}'
    - source_labels: [__address__]
      regex:  '(.*):\d+'
      target_label: __address__
      replacement: '${1}:9100'
    - source_labels: [__name__]
      regex:  'node_disk_read_bytes_total'
      target_label: __name__
      replacement: 'node_disk_bytes_read'

What did you expect to see?
both address and names will be replaced.

What did you see instead? Under which circumstances?
Unfortunately, the names were not replaced, but the address was correct.

To verify it, I've changed the name to be replaced in the metric_relabel_configs part.
This did work.

  relabel_configs:
    - source_labels: [__address__]
      regex:  '(.*):\d+'
      target_label: instance
      replacement: '${1}'
    - source_labels: [__address__]
      regex:  '(.*):\d+'
      target_label: __address__
      replacement: '${1}:9100'
  metric_relabel_configs:
    - source_labels: [__name__]
      regex:  'node_disk_read_bytes_total'
      target_label: __name__
      replacement: 'node_disk_bytes_read'

Environment
Prometheus with dokcer container.

• System information:

  insert output of uname -srm here

• Prometheus version:
  checked with 2.5 and 2.7.1

• Prometheus configuration file:

global:
  scrape_interval: 5s # By default, scrape targets every 5 second.
  scrape_timeout: 4s # Timeout before trying to scape a target again

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'scylla-monitor'
rule_files:
  - /etc/prometheus/prometheus.rules.yml
    #
# Alerting specifies settings related to the Alertmanager.
alerting:
        #  alert_relabel_configs:
        #    [ - <relabel_config> ... ]
  alertmanagers:
  - static_configs:
    - targets:
        - AM_ADDRESS

scrape_configs:
- job_name: scylla
  honor_labels: false
  file_sd_configs:
    - files:
      - /etc/scylla.d/prometheus/scylla_servers.yml
  relabel_configs:
    - source_labels: [__address__]
      regex:  '(.*):.+'
      target_label: instance
      replacement: '${1}'

- job_name: node_exporter
  honor_labels: false
  file_sd_configs:
    - files:
      - /etc/scylla.d/prometheus/node_exporter_servers.yml
  relabel_configs:
    - source_labels: [__address__]
      regex:  '(.*):\d+'
      target_label: instance
      replacement: '${1}'
    - source_labels: [__address__]
      regex:  '(.*):\d+'
      target_label: __address__
      replacement: '${1}:9100'
    - source_labels: [__name__]
      regex:  'node_disk_read_bytes_total'
      target_label: __name__
      replacement: 'node_disk_bytes_read'
    - source_labels: [__name__]
      regex:  'node_disk_written_bytes_total'
      target_label: __name__
      replacement: 'node_disk_bytes_written'
    - source_labels: [__name__]
      regex:  'node_disk_reads_completed_total'
      target_label: __name__
      replacement: 'node_disk_reads_completed'
    - source_labels: [__name__]
      regex:  'node_disk_writes_completed_total'
      target_label: __name__
      replacement: 'node_disk_writes_completed'
    - source_labels: [__name__]
      regex:  'node_filesystem_avail_bytes'
      target_label: __name__
      replacement: 'node_filesystem_avail'
    - source_labels: [__name__]
      regex:  'node_network_receive_bytes_total'
      target_label: __name__
      replacement: 'node_network_receive_bytes'
    - source_labels: [__name__]
      regex:  'node_network_receive_packets_total'
      target_label: __name__
      replacement: 'node_network_receive_packets'
    - source_labels: [__name__]
      regex:  'node_network_transmit_bytes_total'
      target_label: __name__
      replacement: 'node_network_transmit_bytes'
    - source_labels: [__name__]
      regex:  'node_network_transmit_packets_total'
      target_label: __name__
      replacement: 'node_network_transmit_packets'
    - source_labels: [__name__]
      regex:  'node_filesystem_size_bytes'
      target_label: __name__
      replacement: 'node_filesystem_size'

- job_name: scylla_manager
  honor_labels: false
  file_sd_configs:
    - files:
      - /etc/scylla.d/prometheus/scylla_manager_servers.yml
  metric_relabel_configs:
    - source_labels: [host]
      target_label: instance

- job_name: 'prometheus'
  # Override the global default and scrape targets from this job every 5 seconds.
  scrape_interval: 5s
  static_configs:
    - targets:
      - localhost:9090

[simonpasquier]

To verify it, I've changed the name to be replaced in the metric_relabel_configs part.

This is expected, relabel_configs applies to the label set of a target before it gets scraped so __name__ isn't known yet. You need to use metric_relabel_configs indeed.

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.