panic: runtime error when the bind port has been used

[songjiayang]

What did you do?

I meet a panic error when I start prometheus-1.8.0.darwin-amd642 if the bind port had been used.

What did you expect to see?

Like old version just logging error and exit progress.

What did you see instead? Under which circumstances?

INFO[0000] Starting prometheus (version=1.8.0, branch=HEAD, revision=3569eef8b1bc062bb5df43181b938277818f365b) source="main.go:87"
INFO[0000] Build context (go=go1.9.1, user=root@bd4857492255, date=20171006-22:16:15) source="main.go:88"
INFO[0000] Host details (darwin) source="main.go:89"
INFO[0000] Loading configuration file prometheus.yml source="main.go:254"
INFO[0000] Listening on :9090 source="web.go:341"
INFO[0000] Loading series map and head chunks... source="storage.go:428"
INFO[0000] 1831 series loaded. source="storage.go:439"
INFO[0000] Server is Ready to receive requests. source="main.go:230"
ERRO[0000] Error starting web server, exiting gracefully: listen tcp :9090: bind: address already in use source="main.go:240"
INFO[0000] See you next time! source="main.go:243"
INFO[0000] Stopping target manager... source="targetmanager.go:77"
INFO[0000] Stopping rule manager... source="manager.go:388"
INFO[0000] Starting target manager... source="targetmanager.go:63"
INFO[0000] Rule manager stopped. source="manager.go:394"
INFO[0000] Stopping notification handler... source="notifier.go:418"
INFO[0000] Stopping local storage... source="storage.go:457"
INFO[0000] Stopping maintenance loop... source="storage.go:459"
INFO[0000] Maintenance loop stopped. source="storage.go:1463"
INFO[0000] Stopping series quarantining... source="storage.go:463"
INFO[0000] Series quarantining stopped. source="storage.go:1912"
INFO[0000] Stopping chunk eviction... source="storage.go:467"
INFO[0000] Chunk eviction stopped. source="storage.go:1158"
INFO[0000] Checkpointing in-memory metrics and chunks... source="persistence.go:633"
INFO[0000] Done checkpointing in-memory metrics and chunks in 22.634392ms. source="persistence.go:665"
INFO[0000] Checkpointing fingerprint mappings... source="persistence.go:1526"
INFO[0000] Done checkpointing fingerprint mappings in 652.656µs. source="persistence.go:1549"
INFO[0000] Local storage stopped. source="storage.go:484"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x21be9bd]

goroutine 1 [running]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Stop(0xc4203ee480)
/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:83 +0xad
main.Main(0x0)
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:244 +0x16d8
main.main()
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:44 +0x22

Environment

1.8.0.darwin-amd642

• System information:

  Darwin 16.7.0 x86_64

• Prometheus version:

  prometheus, version 1.8.0 (branch: HEAD, revision: 3569eef)
  build user: root@bd4857492255
  build date: 20171006-22:16:15
  go version: go1.9.1

[brian-brazil]

I can't reproduce this on my workstation with release-1.8.

[songjiayang]

so strange, your workstation system is what

[tattsun]

I can hardly reproduce it with release-1.8. But, it seems to be a timing issue.

The problem is that rarely TargetManager#Stop is called before TargetManager#Run.
(No guarantee of calling order)

Inserting time.Sleep will reproduce this problem 100%.

tattsun@b21d935

[songjiayang]

@tattsun thanks for reproducing it. So need to fix it ?

[songjiayang]

Thank guys.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.