2.0.0-rc1: targets page hangs and new targets cease to be scraped

[kinghrothgar]

What did you do?
Left Prometheus running for many days using DNS SD for target discovery

What did you expect to see?
It to keep working and scraping new instances that appeared in DNS SRV requests

What did you see instead? Under which circumstances?
I restarted a group of dns_sd targets which cause them to receive new dns domains. I noticed that no metrics from these new targets were showing up. I attempted to targets page in the web-ui and it hung and would not load. I couldn't think of any way to troubleshoot it more. Restarting Prometheus fixed the problem.

Environment

• System information:

Linux 3.10.0-693.2.2.el7.x86_64 x86_64
Centos 7

• Prometheus version:

prometheus, version 2.0.0-rc.1 (branch: HEAD, revision: 5ab8834befbd92241a88976c790ace7543edcd59)
  build user:       root@1f56dd8b6f7b
  build date:       20171017-12:34:15
  go version:       go1.9.1

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'admiral-submissions'
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - "alertmanager.services.admiral:9093"

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
   - /etc/prometheus/rules/*.rules

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
      - targets: ['localhost:9090']

    file_sd_configs:
      - files: ["/etc/prometheus/target_files/*.yml"]

  - job_name: 'debug'
    metrics_path: /debug/metrics
    dns_sd_configs:
      - names:
        - 'ergoproxy.debug.admiral'
        - 'hoothoot.debug.admiral'
        - 'visitor-api.debug.admiral'
    relabel_configs:
      - source_labels: [__meta_dns_name]
        regex: ([a-z0-9-]+).debug.admiral
        target_label: job
        replacement: $1
      # todo: get rid of hostname and just use instance
      - source_labels: [__address__]
        regex: ([a-z0-9-]+)-([a-z0-9]+)-.*
        target_label: hostname
        replacement: $2
      - source_labels: [__address__]
        regex: ([a-z0-9-]+)-([a-z0-9]+)-.*
        target_label: instance
        replacement: $2
      - source_labels: [__address__]
        regex: ([a-z0-9-]+)-([a-z0-9]+)-.*
        target_label: datacenter
        replacement: $1

• Logs:
  There were no logs of any kind :(

[fabxc]

Please try out rc.2 and let us know whether the problem persists. Several bugs were fixed and we cannot feasibly debug old pre-release versions.

[gouthamve]

Closing this as there is no update and the issue is likely fixed. Please re-open if the issue still persists in the 2.0 stable release.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.