Crash with serverset_sd_configs

[boiler]

What did you do?

Run latest prometheus binary with "serverset_sd_configs" option in config

What did you expect to see?

Working daemon

What did you see instead? Under which circumstances?

panic: runtime error

Environment

• System information:

    Linux 3.10.0-514.26.2.el7.x86_64 x86_64
  

• Prometheus version:

    2.0.0-beta.0
  

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'telegraf'
    serverset_sd_configs: # zookeeper service discovery
      - servers: ['10.123.0.11','10.123.0.12']
        paths: ['/discovery/prod/telegraf']

• Logs:

INFO[0000] Starting prometheus (version=2.0.0-beta.0, branch=master, revision=2b5d9159537cbd123219296121e05244e26c0940)  source="main.go:202"
INFO[0000] Build context (go=go1.8.3, user=root@fc24486243df, date=20170712-12:21:13)  source="main.go:203"
INFO[0000] Host details (Linux 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 ******* (none))  source="main.go:204"
INFO[0000] Starting tsdb                                 source="main.go:216"
INFO[0000] tsdb started                                  source="main.go:222"
INFO[0000] Loading configuration file prometheus.yml     source="main.go:351"
INFO[0000] Listening on 0.0.0.0:9090                     source="web.go:286"
INFO[0000] Starting target manager...                    source="targetmanager.go:67"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x70 pc=0x15c0f32]

goroutine 196 [running]:
github.com/prometheus/prometheus/util/treecache.ZookeeperLogger.Printf(0x0, 0x0, 0x1c1596f, 0xf, 0xc420164570, 0x1, 0x1)
        /go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:53 +0x22
github.com/prometheus/prometheus/util/treecache.(*ZookeeperLogger).Printf(0xc420164410, 0x1c1596f, 0xf, 0xc420164570, 0x1, 0x1)
        <autogenerated>:1 +0x88
github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).connect(0xc42016c140, 0xe, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:324 +0x39f
github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).loop(0xc42016c140)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:393 +0x43
github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.Connect.func1(0xc42016c140)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:218 +0x2f
created by github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.Connect
        /go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:222 +0x6db

[JustinVenus]

I hit the same issue today w/ prometheus-2.0.0-beta.2

[JustinVenus]

@boiler I don't know how to open a pull request against the tag v2.0.0-beta.2, but this patch fixes the issue for me.

v2.0.0-beta.2...JustinVenus:issues/2976

$ ./prometheus --config.file=/Users/jvenus/Downloads/prometheus.yml
INFO[0000] Starting prometheus (version=2.0.0-beta.2, branch=issues/2976, revision=c684392f318ea9b176a48ca497c43ddd8103f9dd)  source="main.go:204"
INFO[0000] Build context (go=go1.8.3, user=jvenus@jvenus-signal-mbp13.local, date=20170822-20:22:17)  source="main.go:205"
INFO[0000] Host details (darwin)                         source="main.go:206"
INFO[0000] Starting tsdb                                 source="main.go:218"
INFO[0000] tsdb started                                  source="main.go:224"
INFO[0000] Loading configuration file /Users/jvenus/Downloads/prometheus.yml  source="main.go:357"
INFO[0000] Server is Ready to receive requests.          source="main.go:334"
INFO[0000] Listening on 0.0.0.0:9090                     source="web.go:327"
INFO[0000] Starting target manager...                    source="targetmanager.go:67"
^CWARN[0002] Received SIGTERM, exiting gracefully...       source="main.go:340"
INFO[0002] See you next time!                            source="main.go:347"
INFO[0002] Stopping target manager...                    source="targetmanager.go:81"
INFO[0002] Target manager stopped.                       source="targetmanager.go:93"
INFO[0002] Stopping rule manager...                      source="manager.go:454"
INFO[0002] Rule manager stopped.                         source="manager.go:460"
INFO[0002] Stopping notification handler...              source="notifier.go:471"

global: {evaluation_interval: 1m, scrape_interval: 1m}
rule_files: []
scrape_configs:
- job_name: namerd
  metrics_path: /admin/metrics/prometheus
  serverset_sd_configs:
  - paths: [/sd/nobody/prod/namerd]
    servers: ['172.31.2.242:2181', '172.31.0.131:2181', '172.31.5.115:2181', '172.31.5.100:2181',
      '172.31.11.164:2181']

/CC @brian-brazil

[brian-brazil]

Open it against master, it'll get merged forward.

[boiler]

Fix in master. Issue resolved.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.