consul-sd started failing with 2.4.3

[davidkarlsen]

Bug Report

What did you do?
Upgrade from v2.4.2 to v2.4.3

What did you expect to see?
That consul sd kept on working

What did you see instead? Under which circumstances?
The targets found through consul disappeared

Environment

• System information:
  N/A

• Prometheus version:
  v2.4.3

• Prometheus configuration file:

 - job_name: 'consul'
    consul_sd_configs:
      - server: 'alp-aot-ccm02.unix.cosng.net:8500'
        datacenter: 'production'
    metric_relabel_configs:
      - source_labels: [container_label_container_group]
        target_label: container_group
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*prom_monitored.*
        action: keep
      - source_labels: [__meta_consul_service]
        target_label: job 
      - source_labels: [__meta_consul_tags]
        regex: .*,alias-([^,]+),.*
        replacement: '${1}'
        target_label: alias
      - source_labels: [__meta_consul_tags]
        regex: .*,metrics_path=([^,]+),.*
        replacement: '${1}'
        target_label: __metrics_path__
      - source_labels: [__meta_consul_tags]
        regex: .*,finodsgroup=([^,]+),.*
        replacement: '${1}'
        target_label: finodsgroup
      - source_labels: [__meta_consul_tags]
        regex: .*,container_group=([^,]+),.*
        replacement: '${1}'
        target_label: container_group

[juliusv]

Can you confirm that it's still working with 2.4.2? These are all code changes from 2.4.2 -> 2.4.3:

v2.4.2...v2.4.3

It's really few changes and I have a hard time seeing how anything in there could have affected Consul SD...

[davidkarlsen]

Yes - works fine if I downgrade

[juliusv]

@davidkarlsen So strange - do you see any pre-relabeling targets being discovered under /service-discovery with 2.4.3?

[davidkarlsen]

Ah - I see what's going on now - it has trouble resolving the name of my consul server.
Did the base-image change in any way?

[davidkarlsen]

ah - now this is embarrassing - docker adds the hostname to /etc/hosts which messes things up - something has changed here - but it's a docker problem at my side. Sorry for the noise!

[juliusv]

@davidkarlsen No worries, glad that it's not a Prometheus problem :)

[simonpasquier]

This has been introduced by prometheus/busybox#16