version 1.4.0 broke our service discovery

[fuzzyami]

What did you do?
we've accidentally upgraded our prometheus setup to 1.4.0 (from 1.3.1)
What did you expect to see?
the list of targets from all our dc, which we're getting via consul
What did you see instead? Under which circumstances?
the list of targets we got from consul was partial. we only saw a subset of the targets. and different subset everytime we restarted prometheus

Environment
prom/prometheus image over docker

• System information:
  insert output of uname -srm here
  Linux 3.13.0-100-generic x86_64

• Prometheus version:
  we were using the master image (prom/prometheus). problem disappered with prom/prometheus:v.1.3.1
  insert output of prometheus -version here

• Alertmanager version:

  insert output of alertmanager -version here (if relevant to the issue)

• Prometheus configuration file:
  (I've sanitized the dc names here to xxx(number))

global:
  scrape_interval: 15s
  evaluation_interval: 15s

  external_labels:
    monitor: 'prometheus-server'

rule_files:
  
  - alerts/service_down.rules
  - alerts/bandwidth_status.rules
  - alerts/media_connectivity.rules
  - alerts/mlb_status.rules
  - alerts/conferences_status.rules
  - alerts/app_deploy.rules
  - alerts/node_status.rules
  - alerts/media_server_status.rules
  - alerts/streaming_connectivity_ratio.rules

  
scrape_configs:
  - job_name: 'consul'
    consul_sd_configs:
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx1'
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx2'
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx3'
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx4'
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx5'
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx6'
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx7'
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx8'
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx9'
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx10'
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx11'
       - server:   '0.0.0.0:8500'
         datacenter: 'xxx12'

    relabel_configs:

      - source_labels: ['__meta_consul_tags']
        regex: '(.*),ignore-at-prometheus,(.*)'
        action: drop


      - source_labels: ['__meta_consul_tags']
        regex: '(.*),decommissioned,(.*)'
        action: drop

      - source_labels: ['__address__']
        separator:     ':'
        regex:         '(.*):(8300)'
        action:        drop


      - source_labels: ['__meta_consul_service']
        regex:         '(.*)telegraf-(.*)'
        target_label: '__meta_consul_service'
        replacement:   '$1$2'

      - source_labels: ['__meta_consul_service']
        regex:         '(.*)'
        target_label: 'job'
        replacement:   '$1'

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

[brian-brazil]

Can you try head? A fix for consul SD was just submitted.

[fuzzyami]

@brian-brazil, Could you provide a docker image I can use?

[brian-brazil]

It's not released yet. You'll have to build one yourself.

[fuzzyami]

tested with head. looks good.

Version	1.4.0
Revision	d95e61d418ec98cdb8b14fecca39c03e70fa56a5

[brian-brazil]

Great, thanks!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.