Crash on reload

[theonlydoo]

What did you do?

1. use fileservdisc
2. remove all files (throws "normal errors")
3. append new files
4. reload prometheus
5. watch it crash

What did you expect to see?

1. a reloaded prometheus

What did you see instead? Under which circumstances?

déc. 19 10:59:55 foobar prometheus[14283]: level=info ts=2017-12-19T09:59:55.579535345Z caller=main.go:394 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
déc. 19 10:59:55 foobar systemd[1]: Reloaded Prometheus Node Exporter.
déc. 19 10:59:55 foobar prometheus[14283]: level=error ts=2017-12-19T09:59:55.59282889Z caller=file.go:121 component="target manager" discovery=file msg="Error adding file watcher" err="too many open files"
déc. 19 10:59:55 foobar prometheus[14283]: panic: runtime error: invalid memory address or nil pointer dereference
déc. 19 10:59:55 foobar prometheus[14283]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xab0f82]
déc. 19 10:59:55 foobar prometheus[14283]: goroutine 24618 [running]:
déc. 19 10:59:55 foobar prometheus[14283]: github.com/prometheus/prometheus/vendor/gopkg.in/fsnotify%2ev1.(*Watcher).isClosed(0x0, 0xc4f1d3ed78)
déc. 19 10:59:55 foobar prometheus[14283]:         /go/src/github.com/prometheus/prometheus/vendor/gopkg.in/fsnotify.v1/inotify.go:67 +0x22

Environment

[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
MemoryAccounting=true
MemoryLimit=24G
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
User=prometheus
Restart=always
RestartSec=2
EnvironmentFile=/etc/default/prometheus
ExecStart=/usr/bin/prometheus $ARGS
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill $MAINPID
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target

• System information:

$ uname -srm
Linux 4.9.0-3-amd64 x86_64

• Prometheus version:

prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2

[krasi-georgiev]

@brian-brazil this looks like a duplicate of #3446
with an open PR for it.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.