Panic in labels hashing (Prometheus 2)

[grobie]

What did you do?

Started a couple of Prometheus servers v2.0.0-beta.5 (plus a tiny patch to send an Accept header again, see #3208)

What did you expect to see?

No crashes.

What did you see instead? Under which circumstances?

Panic after a couple of hours of running.

Environment

• System information:

  Linux 4.4.10+soundcloud x86_64

• Prometheus version:

prometheus, version 2.0.0-beta.5.sc0 (branch: grobie/http-accept, revision: 5964c0dc6f460d12ba8df62cde5849a62b24648d)
  build user:       grobie@grobook
  build date:       20170922-16:11:02
  go version:       go1.9

• Logs:

2017-09-23_01:04:22.59135 unexpected fault address 0x7f8b0598ebaa
2017-09-23_01:04:22.59139 fatal error: fault
2017-09-23_01:04:22.59458 [signal SIGSEGV: segmentation violation code=0x1 addr=0x7f8b0598ebaa pc=0x45d356]
2017-09-23_01:04:22.59461 
2017-09-23_01:04:22.59462 goroutine 44283 [running]:
2017-09-23_01:04:22.59462 runtime.throw(0x1babb6f, 0x5)
2017-09-23_01:04:22.59462       /usr/lib/go/src/runtime/panic.go:605 +0x95 fp=0xcb83175308 sp=0xcb831752e8 pc=0x42bc55
2017-09-23_01:04:22.59464 runtime.sigpanic()
2017-09-23_01:04:22.59464       /usr/lib/go/src/runtime/signal_unix.go:374 +0x227 fp=0xcb83175358 sp=0xcb83175308 pc=0x4426c7
2017-09-23_01:04:22.59464 runtime.memmove(0xcb83175412, 0x7f8b0598ebaa, 0xa)
2017-09-23_01:04:22.59465       /usr/lib/go/src/runtime/memmove_amd64.s:172 +0x136 fp=0xcb83175360 sp=0xcb83175358 pc=0x45d356
2017-09-23_01:04:22.59466 github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/labels.Labels.Hash(0xc5fc1b4500, 0x8, 0x8, 0x7f8bcb697d90)
2017-09-23_01:04:22.59466       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/labels/labels.go:64 +0x191 fp=0xcb83175800 sp=0xcb83175360 pc=0x1567861
2017-09-23_01:04:22.59466 github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Add(0xc7e6b766e0, 0xc5fc1b4500, 0x8, 0x8, 0x15eac43733d, 0x3ff0000000000000, 0xc42c6864b0, 0x0, 0x46da42)
2017-09-23_01:04:22.59468       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:393 +0x66 fp=0xcb83175878 sp=0xcb83175800 pc=0x1585a76
2017-09-23_01:04:22.59468 github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Add(0xc85f6dc0e0, 0xc5fc1b4500, 0x8, 0x8, 0x15eac43733d, 0x3ff0000000000000, 0x1a1e520, 0xc42c6864b8, 0x28002c6864b0)
2017-09-23_01:04:22.59469       <autogenerated>:1 +0x7b fp=0xcb831758d0 sp=0xcb83175878 pc=0x15a7d0b
2017-09-23_01:04:22.59469 github.com/prometheus/prometheus/storage/tsdb.appender.Add(0x28a1c60, 0xc85f6dc0e0, 0xc5fc1b4500, 0x8, 0x8, 0x15eac43733d, 0x3ff0000000000000, 0xc8c96b4501, 0xc608055a20, 0xcb83175a30)
2017-09-23_01:04:22.59470       /home/grobie/code/go/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:192 +0xa8 fp=0xcb83175968 sp=0xcb831758d0 pc=0x15aafb8
2017-09-23_01:04:22.59471 github.com/prometheus/prometheus/storage/tsdb.(*appender).Add(0xc8f009efb0, 0xc5fc1b4500, 0x8, 0x8, 0x15eac43733d, 0x3ff0000000000000, 0x3, 0x6, 0x7)
2017-09-23_01:04:22.59472       <autogenerated>:1 +0x8b fp=0xcb831759c8 sp=0xcb83175968 pc=0x15abd6b
2017-09-23_01:04:22.59472 github.com/prometheus/prometheus/storage.(*fanoutAppender).Add(0xc729891640, 0xc5fc1b4500, 0x8, 0x8, 0x15eac43733d, 0x3ff0000000000000, 0x425430, 0x0, 0x0)
2017-09-23_01:04:22.59473       /home/grobie/code/go/src/github.com/prometheus/prometheus/storage/fanout.go:112 +0x7b fp=0xcb83175a40 sp=0xcb831759c8 pc=0x155a5eb
2017-09-23_01:04:22.59473 github.com/prometheus/prometheus/rules.(*Group).Eval.func1(0x1bafb16, 0x8, 0xc628038600, 0xbe698aa3c9d93ca0, 0x1a37884941da, 0x296bd00, 0x6, 0x28a0f60, 0xc628038580)
2017-09-23_01:04:22.59474       /home/grobie/code/go/src/github.com/prometheus/prometheus/rules/manager.go:334 +0x6f1 fp=0xcb83175d60 sp=0xcb83175a40 pc=0x1600651
2017-09-23_01:04:22.59474 github.com/prometheus/prometheus/rules.(*Group).Eval(0xc628038600, 0xbe698aa3c9d93ca0, 0x1a37884941da, 0x296bd00)
2017-09-23_01:04:22.59475       /home/grobie/code/go/src/github.com/prometheus/prometheus/rules/manager.go:375 +0xbf fp=0xcb83175de0 sp=0xcb83175d60 pc=0x15fbf4f
2017-09-23_01:04:22.59476 github.com/prometheus/prometheus/rules.(*Group).run.func1()
2017-09-23_01:04:22.59476       /home/grobie/code/go/src/github.com/prometheus/prometheus/rules/manager.go:182 +0x81 fp=0xcb83175e30 sp=0xcb83175de0 pc=0x15ffd81
2017-09-23_01:04:22.59477 github.com/prometheus/prometheus/rules.(*Group).run(0xc628038600)
2017-09-23_01:04:22.59477       /home/grobie/code/go/src/github.com/prometheus/prometheus/rules/manager.go:207 +0x23c fp=0xcb83175fb0 sp=0xcb83175e30 pc=0x15fb21c
2017-09-23_01:04:22.59478 github.com/prometheus/prometheus/rules.(*Manager).ApplyConfig.func1.1(0xc4200751c0, 0xc628038600)
2017-09-23_01:04:22.59478       /home/grobie/code/go/src/github.com/prometheus/prometheus/rules/manager.go:510 +0x46 fp=0xcb83175fd0 sp=0xcb83175fb0 pc=0x1601ec6
2017-09-23_01:04:22.59480 runtime.goexit()
2017-09-23_01:04:22.59480       /usr/lib/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xcb83175fd8 sp=0xcb83175fd0 pc=0x45ca71
2017-09-23_01:04:22.59480 created by github.com/prometheus/prometheus/rules.(*Manager).ApplyConfig.func1
2017-09-23_01:04:22.59481       /home/grobie/code/go/src/github.com/prometheus/prometheus/rules/manager.go:505 +0x56
2017-09-23_01:04:22.59481 
2017-09-23_01:04:22.59481 goroutine 1 [select, 418 minutes]:
2017-09-23_01:04:22.59482 main.main()
2017-09-23_01:04:22.59482       /home/grobie/code/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:368 +0x443f
2017-09-23_01:04:22.59483 
2017-09-23_01:04:22.59483 goroutine 17 [syscall, 451 minutes]:
2017-09-23_01:04:22.59483 os/signal.signal_recv(0x2895420)
2017-09-23_01:04:22.59484       /usr/lib/go/src/runtime/sigqueue.go:131 +0xa6
2017-09-23_01:04:22.59484 os/signal.loop()
2017-09-23_01:04:22.59484       /usr/lib/go/src/os/signal/signal_unix.go:22 +0x22
2017-09-23_01:04:22.59485 created by os/signal.init.0
2017-09-23_01:04:22.59486       /usr/lib/go/src/os/signal/signal_unix.go:28 +0x41
2017-09-23_01:04:22.59486 
2017-09-23_01:04:22.59486 goroutine 20 [chan receive]:
2017-09-23_01:04:22.59487 github.com/prometheus/prometheus/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x296c060)
2017-09-23_01:04:22.59487       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:879 +0x9f
2017-09-23_01:04:22.59488 created by github.com/prometheus/prometheus/vendor/github.com/golang/glog.init.0
2017-09-23_01:04:22.59488       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:410 +0x203
2017-09-23_01:04:22.59488 
2017-09-23_01:04:22.59489 goroutine 200 [select, 418 minutes]:
2017-09-23_01:04:22.59490 main.main.func2(0xc420156240, 0xc420156180, 0xc42015a3c0, 0x2886420, 0xc42042fd10, 0xc42065e0a0, 0x5, 0x5, 0xc42044c600)
2017-09-23_01:04:22.59490       /home/grobie/code/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:285 +0x121
2017-09-23_01:04:22.59490 created by main.main
2017-09-23_01:04:22.59491       /home/grobie/code/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:282 +0x3d96
2017-09-23_01:04:22.59491 
2017-09-23_01:04:22.59491 goroutine 50656 [IO wait]:
2017-09-23_01:04:22.59492 internal/poll.runtime_pollWait(0x7f8bb85fd400, 0x72, 0x0)
2017-09-23_01:04:22.59492       /usr/lib/go/src/runtime/netpoll.go:173 +0x57
2017-09-23_01:04:22.59493 internal/poll.(*pollDesc).wait(0xc6ffbfc318, 0x72, 0xffffffffffffff00, 0x288fc60, 0x28835c0)
2017-09-23_01:04:22.59493       /usr/lib/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
2017-09-23_01:04:22.59494 internal/poll.(*pollDesc).waitRead(0xc6ffbfc318, 0xc6ffbff000, 0x1000, 0x1000)
2017-09-23_01:04:22.59494       /usr/lib/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
2017-09-23_01:04:22.59495 internal/poll.(*FD).Read(0xc6ffbfc300, 0xc6ffbff000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59495       /usr/lib/go/src/internal/poll/fd_unix.go:125 +0x18a
2017-09-23_01:04:22.59495 net.(*netFD).Read(0xc6ffbfc300, 0xc6ffbff000, 0x1000, 0x1000, 0x42d91b, 0xc6db8e0a40, 0x458b60)
2017-09-23_01:04:22.59496       /usr/lib/go/src/net/fd_unix.go:202 +0x52
2017-09-23_01:04:22.59496 net.(*conn).Read(0xc4ee8cbc30, 0xc6ffbff000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59497       /usr/lib/go/src/net/net.go:176 +0x6d
2017-09-23_01:04:22.59497 net/http.(*persistConn).Read(0xc4edd857a0, 0xc6ffbff000, 0x1000, 0x1000, 0xc6975eb9a0, 0xc6e31cbf78, 0x456bc0)
2017-09-23_01:04:22.59498       /usr/lib/go/src/net/http/transport.go:1391 +0x140
2017-09-23_01:04:22.59498 bufio.(*Reader).fill(0xc6f4ac5aa0)
2017-09-23_01:04:22.59499       /usr/lib/go/src/bufio/bufio.go:97 +0x11a
2017-09-23_01:04:22.59499 bufio.(*Reader).Peek(0xc6f4ac5aa0, 0x1, 0x0, 0x0, 0x0, 0xc7706ed740, 0x0)
2017-09-23_01:04:22.59499       /usr/lib/go/src/bufio/bufio.go:129 +0x3a
2017-09-23_01:04:22.59500 net/http.(*persistConn).readLoop(0xc4edd857a0)
2017-09-23_01:04:22.59500       /usr/lib/go/src/net/http/transport.go:1539 +0x185
2017-09-23_01:04:22.59501 created by net/http.(*Transport).dialConn
2017-09-23_01:04:22.59501       /usr/lib/go/src/net/http/transport.go:1186 +0xa2e
2017-09-23_01:04:22.59502 
2017-09-23_01:04:22.59502 goroutine 27 [select, 418 minutes, locked to thread]:
2017-09-23_01:04:22.59502 runtime.gopark(0x1c3be90, 0x0, 0x1bad58a, 0x6, 0x18, 0x1)
2017-09-23_01:04:22.59503       /usr/lib/go/src/runtime/proc.go:277 +0x12c
2017-09-23_01:04:22.59503 runtime.selectgo(0xc420800f50, 0xc420156300)
2017-09-23_01:04:22.59504       /usr/lib/go/src/runtime/select.go:395 +0x1138
2017-09-23_01:04:22.59504 runtime.ensureSigM.func1()
2017-09-23_01:04:22.59505       /usr/lib/go/src/runtime/signal_unix.go:511 +0x220
2017-09-23_01:04:22.59505 runtime.goexit()
2017-09-23_01:04:22.59506       /usr/lib/go/src/runtime/asm_amd64.s:2337 +0x1
2017-09-23_01:04:22.59506 
2017-09-23_01:04:22.59506 goroutine 1280754 [select]:
2017-09-23_01:04:22.59507 net/http.(*persistConn).writeLoop(0xc683b12b40)
2017-09-23_01:04:22.59507       /usr/lib/go/src/net/http/transport.go:1759 +0x165
2017-09-23_01:04:22.59507 created by net/http.(*Transport).dialConn
2017-09-23_01:04:22.59508       /usr/lib/go/src/net/http/transport.go:1187 +0xa53
2017-09-23_01:04:22.59508 
2017-09-23_01:04:22.59508 goroutine 1043163 [select]:
2017-09-23_01:04:22.59509 net/http.(*persistConn).writeLoop(0xc60a97e000)
2017-09-23_01:04:22.59509       /usr/lib/go/src/net/http/transport.go:1759 +0x165
2017-09-23_01:04:22.59509 created by net/http.(*Transport).dialConn
2017-09-23_01:04:22.59510       /usr/lib/go/src/net/http/transport.go:1187 +0xa53
2017-09-23_01:04:22.59511 
2017-09-23_01:04:22.59511 goroutine 43861 [select, 418 minutes]:
2017-09-23_01:04:22.59511 github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc4569e6f00, 0x7f8bcb600bc8, 0xc62770c800)
2017-09-23_01:04:22.59512       /home/grobie/code/go/src/github.com/prometheus/prometheus/discovery/discovery.go:203 +0x2b3
2017-09-23_01:04:22.59512 github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f8bcb600bc8, 0xc62770c800, 0xc4205a0f00, 0xc627716de0)
2017-09-23_01:04:22.59513       /home/grobie/code/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:121 +0x43
2017-09-23_01:04:22.59513 created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
2017-09-23_01:04:22.59513       /home/grobie/code/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:118 +0x5a3
2017-09-23_01:04:22.59514 
2017-09-23_01:04:22.59514 goroutine 204 [runnable]:
2017-09-23_01:04:22.59514 github.com/prometheus/prometheus/notifier.(*Notifier).Run(0xc42041c240)
2017-09-23_01:04:22.59515       /home/grobie/code/go/src/github.com/prometheus/prometheus/notifier/notifier.go:308 +0x100
2017-09-23_01:04:22.59515 created by main.main
2017-09-23_01:04:22.59516       /home/grobie/code/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:330 +0x3f43
2017-09-23_01:04:22.59516 
2017-09-23_01:04:22.59516 goroutine 42862 [select]:
2017-09-23_01:04:22.59517 github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc627924eb0, 0x7f8bcb600bc8, 0xc6278aa600)
2017-09-23_01:04:22.59517       /home/grobie/code/go/src/github.com/prometheus/prometheus/discovery/discovery.go:203 +0x2b3
2017-09-23_01:04:22.59518 github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f8bcb600bc8, 0xc6278aa600, 0xc4205a0f00, 0xc6278d4f00)
2017-09-23_01:04:22.59518       /home/grobie/code/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:121 +0x43
2017-09-23_01:04:22.59519 created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
2017-09-23_01:04:22.59519       /home/grobie/code/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:118 +0x5a3
2017-09-23_01:04:22.59519 
2017-09-23_01:04:22.59520 goroutine 217 [IO wait, 480 minutes]:
2017-09-23_01:04:22.59520 internal/poll.runtime_pollWait(0x7f8bcb63bdf0, 0x72, 0x0)
2017-09-23_01:04:22.59521       /usr/lib/go/src/runtime/netpoll.go:173 +0x57
2017-09-23_01:04:22.59521 internal/poll.(*pollDesc).wait(0xc4205a0818, 0x72, 0xffffffffffffff00, 0x288fc60, 0x28835c0)
2017-09-23_01:04:22.59521       /usr/lib/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
2017-09-23_01:04:22.59523 internal/poll.(*pollDesc).waitRead(0xc4205a0818, 0xc420b04000, 0x8000, 0x8000)
2017-09-23_01:04:22.59523       /usr/lib/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
2017-09-23_01:04:22.59524 internal/poll.(*FD).Read(0xc4205a0800, 0xc420b04000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59525       /usr/lib/go/src/internal/poll/fd_unix.go:125 +0x18a
2017-09-23_01:04:22.59525 net.(*netFD).Read(0xc4205a0800, 0xc420b04000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59525       /usr/lib/go/src/net/fd_unix.go:202 +0x52
2017-09-23_01:04:22.59526 net.(*conn).Read(0xc4202d40b8, 0xc420b04000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59526       /usr/lib/go/src/net/net.go:176 +0x6d
2017-09-23_01:04:22.59526 bufio.(*Reader).Read(0xc4204365a0, 0xc420b16038, 0x9, 0x9, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59527       /usr/lib/go/src/bufio/bufio.go:213 +0x30b
2017-09-23_01:04:22.59527 io.ReadAtLeast(0x28850a0, 0xc4204365a0, 0xc420b16038, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59528       /usr/lib/go/src/io/io.go:309 +0x86
2017-09-23_01:04:22.59528 io.ReadFull(0x28850a0, 0xc4204365a0, 0xc420b16038, 0x9, 0x9, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59528       /usr/lib/go/src/io/io.go:327 +0x58
2017-09-23_01:04:22.59529 github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.readFrameHeader(0xc420b16038, 0x9, 0x9, 0x28850a0, 0xc4204365a0, 0x0, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59529       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
2017-09-23_01:04:22.59530 github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420b16000, 0x0, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59530       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
2017-09-23_01:04:22.59531 github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc4205560f0, 0x0, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59531       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport/http_util.go:608 +0x2f
2017-09-23_01:04:22.59532 github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc4204eec00)
2017-09-23_01:04:22.59532       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport/http2_client.go:1080 +0x47
2017-09-23_01:04:22.59533 created by github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport.newHTTP2Client
2017-09-23_01:04:22.59533       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport/http2_client.go:267 +0xbe4
2017-09-23_01:04:22.59534 
2017-09-23_01:04:22.59534 goroutine 207 [IO wait]:
2017-09-23_01:04:22.59534 internal/poll.runtime_pollWait(0x7f8bcb63beb0, 0x72, 0xffffffffffffffff)
2017-09-23_01:04:22.59535       /usr/lib/go/src/runtime/netpoll.go:173 +0x57
2017-09-23_01:04:22.59535 internal/poll.(*pollDesc).wait(0xc4203c0598, 0x72, 0xc420aa5b00, 0x0, 0x0)
2017-09-23_01:04:22.59536       /usr/lib/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
2017-09-23_01:04:22.59536 internal/poll.(*pollDesc).waitRead(0xc4203c0598, 0xffffffffffffff00, 0x0, 0x0)
2017-09-23_01:04:22.59536       /usr/lib/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
2017-09-23_01:04:22.59537 internal/poll.(*FD).Accept(0xc4203c0580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59537       /usr/lib/go/src/internal/poll/fd_unix.go:334 +0x1e2
2017-09-23_01:04:22.59537 net.(*netFD).accept(0xc4203c0580, 0xc4201cf880, 0x456db0, 0xc4201566b8)
2017-09-23_01:04:22.59538       /usr/lib/go/src/net/fd_unix.go:238 +0x42
2017-09-23_01:04:22.59538 net.(*TCPListener).accept(0xc42090e2c0, 0x403a73, 0xc420156660, 0xc420aa5d28)
2017-09-23_01:04:22.59538       /usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
2017-09-23_01:04:22.59539 net.(*TCPListener).Accept(0xc42090e2c0, 0xc420aa5d28, 0xc420aa5d58, 0x434d4b, 0xc420aa5d30)
2017-09-23_01:04:22.59539       /usr/lib/go/src/net/tcpsock.go:247 +0x49
2017-09-23_01:04:22.59540 github.com/prometheus/prometheus/vendor/golang.org/x/net/netutil.(*limitListener).Accept(0xc42066a200, 0x1c35098, 0xc4208f49c0, 0x28a76c0, 0xc9e1da0060)
2017-09-23_01:04:22.59540       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/netutil/listen.go:30 +0x53
2017-09-23_01:04:22.59541 github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc4208f49c0, 0x0, 0x0)
2017-09-23_01:04:22.59542       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x95
2017-09-23_01:04:22.59542 github.com/prometheus/prometheus/web.(*Handler).Run(0xc42044c600, 0x7f8bcb600bc8, 0xc420075180, 0x0, 0x0)
2017-09-23_01:04:22.59542       /home/grobie/code/go/src/github.com/prometheus/prometheus/web/web.go:443 +0xcba
2017-09-23_01:04:22.59543 main.main.func4(0xc420156600, 0xc42044c600, 0x7f8bcb600bc8, 0xc420075180)
2017-09-23_01:04:22.59543       /home/grobie/code/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:344 +0x3f
2017-09-23_01:04:22.59543 created by main.main
2017-09-23_01:04:22.59544       /home/grobie/code/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:344 +0x409a
2017-09-23_01:04:22.59544 
2017-09-23_01:04:22.59544 goroutine 218 [select, 480 minutes]:
2017-09-23_01:04:22.59545 github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc4204eec00)
2017-09-23_01:04:22.59545       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport/http2_client.go:1168 +0x142
2017-09-23_01:04:22.59546 created by github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport.newHTTP2Client
2017-09-23_01:04:22.59546       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport/http2_client.go:297 +0xd1a
2017-09-23_01:04:22.59547 
2017-09-23_01:04:22.59547 goroutine 226 [chan receive (nil chan), 480 minutes]:
2017-09-23_01:04:22.59547 github.com/prometheus/prometheus/prompb.RegisterAdminHandlerFromEndpoint.func1.1(0x7f8bcb63c0a8, 0xc420044008, 0xc420161520, 0xc420630360, 0x9)
2017-09-23_01:04:22.59548       /home/grobie/code/go/src/github.com/prometheus/prometheus/prompb/rpc.pb.gw.go:67 +0x4c
2017-09-23_01:04:22.59548 created by github.com/prometheus/prometheus/prompb.RegisterAdminHandlerFromEndpoint.func1
2017-09-23_01:04:22.59549       /home/grobie/code/go/src/github.com/prometheus/prometheus/prompb/rpc.pb.gw.go:66 +0x1b7
2017-09-23_01:04:22.59549 
2017-09-23_01:04:22.59549 goroutine 225 [select, 480 minutes]:
2017-09-23_01:04:22.59550 github.com/prometheus/prometheus/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc420161860)
2017-09-23_01:04:22.59550       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/clientconn.go:908 +0x1de
2017-09-23_01:04:22.59550 github.com/prometheus/prometheus/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc420161860)
2017-09-23_01:04:22.59551       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/clientconn.go:637 +0x1c8
2017-09-23_01:04:22.59552 created by github.com/prometheus/prometheus/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
2017-09-23_01:04:22.59552       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/clientconn.go:628 +0x749
2017-09-23_01:04:22.59552 
2017-09-23_01:04:22.59553 goroutine 271 [chan receive]:
2017-09-23_01:04:22.59553 github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
2017-09-23_01:04:22.59553       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:184
2017-09-23_01:04:22.59554 github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42066a360, 0x1c3b830, 0xc65a1cc280, 0x28a0ce0, 0xc420530090)
2017-09-23_01:04:22.59554       <autogenerated>:1 +0x65
2017-09-23_01:04:22.59555 net/http.(*Server).Serve(0xc42046c000, 0x289f620, 0xc42066a360, 0x0, 0x0)
2017-09-23_01:04:22.59555       /usr/lib/go/src/net/http/server.go:2695 +0x1b2
2017-09-23_01:04:22.59556 github.com/prometheus/prometheus/web.(*Handler).Run.func5(0xc42046c000, 0x289f620, 0xc42066a360, 0xc42044c600)
2017-09-23_01:04:22.59556       /home/grobie/code/go/src/github.com/prometheus/prometheus/web/web.go:433 +0x46
2017-09-23_01:04:22.59557 created by github.com/prometheus/prometheus/web.(*Handler).Run
2017-09-23_01:04:22.59557       /home/grobie/code/go/src/github.com/prometheus/prometheus/web/web.go:432 +0xc57
2017-09-23_01:04:22.59558 
2017-09-23_01:04:22.59558 goroutine 243 [select]:
2017-09-23_01:04:22.59558 github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*SegmentWAL).run(0xc42031a000, 0x2540be400)
2017-09-23_01:04:22.59559       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.go:627 +0x125
2017-09-23_01:04:22.59559 created by github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.OpenSegmentWAL
2017-09-23_01:04:22.59559       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.go:205 +0x478
2017-09-23_01:04:22.59560 
2017-09-23_01:04:22.59560 goroutine 53409 [select]:
2017-09-23_01:04:22.59560 net/http.(*persistConn).writeLoop(0xc68e7c97a0)
2017-09-23_01:04:22.59561       /usr/lib/go/src/net/http/transport.go:1759 +0x165
2017-09-23_01:04:22.59561 created by net/http.(*Transport).dialConn
2017-09-23_01:04:22.59561       /usr/lib/go/src/net/http/transport.go:1187 +0xa53
2017-09-23_01:04:22.59562 goroutine 272 [chan receive, 480 minutes]:
2017-09-23_01:04:22.59562 github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
2017-09-23_01:04:22.59563       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:184
2017-09-23_01:04:22.59564 github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42066a220, 0x1c37c48, 0xc42024e7e0, 0x289f620, 0xc42066a220)
2017-09-23_01:04:22.59564       <autogenerated>:1 +0x65
2017-09-23_01:04:22.59564 github.com/prometheus/prometheus/vendor/google.golang.org/grpc.(*Server).Serve(0xc42024e7e0, 0x289f620, 0xc42066a220, 0x0, 0x0)
2017-09-23_01:04:22.59565       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/server.go:432 +0x189
2017-09-23_01:04:22.59565 github.com/prometheus/prometheus/web.(*Handler).Run.func6(0xc42024e7e0, 0x289f620, 0xc42066a220, 0xc42044c600)
2017-09-23_01:04:22.59566       /home/grobie/code/go/src/github.com/prometheus/prometheus/web/web.go:438 +0x46
2017-09-23_01:04:22.59566 created by github.com/prometheus/prometheus/web.(*Handler).Run
2017-09-23_01:04:22.59566       /home/grobie/code/go/src/github.com/prometheus/prometheus/web/web.go:437 +0xca3
2017-09-23_01:04:22.59567 
2017-09-23_01:04:22.59567 goroutine 305 [IO wait, 480 minutes]:
2017-09-23_01:04:22.59567 internal/poll.runtime_pollWait(0x7f8bcb63bd30, 0x72, 0x0)
2017-09-23_01:04:22.59568       /usr/lib/go/src/runtime/netpoll.go:173 +0x57
2017-09-23_01:04:22.59568 internal/poll.(*pollDesc).wait(0xc42040a598, 0x72, 0xffffffffffffff00, 0x288fc60, 0x28835c0)
2017-09-23_01:04:22.59569       /usr/lib/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
2017-09-23_01:04:22.59569 internal/poll.(*pollDesc).waitRead(0xc42040a598, 0xc420b16100, 0x9, 0x9)
2017-09-23_01:04:22.59570       /usr/lib/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
2017-09-23_01:04:22.59570 internal/poll.(*FD).Read(0xc42040a580, 0xc420b161f8, 0x9, 0x9, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59570       /usr/lib/go/src/internal/poll/fd_unix.go:125 +0x18a
2017-09-23_01:04:22.59571 net.(*netFD).Read(0xc42040a580, 0xc420b161f8, 0x9, 0x9, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59571       /usr/lib/go/src/net/fd_unix.go:202 +0x52
2017-09-23_01:04:22.59572 net.(*conn).Read(0xc4201440f0, 0xc420b161f8, 0x9, 0x9, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59572       /usr/lib/go/src/net/net.go:176 +0x6d
2017-09-23_01:04:22.59572 github.com/prometheus/prometheus/vendor/golang.org/x/net/netutil.(*limitListenerConn).Read(0xc420e01650, 0xc420b161f8, 0x9, 0x9, 0x400000801, 0x80100000000, 0x4)
2017-09-23_01:04:22.59573       <autogenerated>:1 +0x5a
2017-09-23_01:04:22.59574 github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*bufferedReader).Read(0xc4200cacf8, 0xc420b161f8, 0x9, 0x9, 0x0, 0x10, 0xc4200444a0)
2017-09-23_01:04:22.59574       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/buffer.go:42 +0x123
2017-09-23_01:04:22.59575 io.ReadAtLeast(0x28862a0, 0xc4200cacf8, 0xc420b161f8, 0x9, 0x9, 0x9, 0x411108, 0x10, 0x1a3f200)
2017-09-23_01:04:22.59575       /usr/lib/go/src/io/io.go:309 +0x86
2017-09-23_01:04:22.59576 io.ReadFull(0x28862a0, 0xc4200cacf8, 0xc420b161f8, 0x9, 0x9, 0x60b6173c870d0101, 0xefff100000004, 0x4)
2017-09-23_01:04:22.59576       /usr/lib/go/src/io/io.go:327 +0x58
2017-09-23_01:04:22.59577 github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.readFrameHeader(0xc420b161f8, 0x9, 0x9, 0x28862a0, 0xc4200cacf8, 0x0, 0x0, 0xc4200444a0, 0x0)
2017-09-23_01:04:22.59577       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
2017-09-23_01:04:22.59577 github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420b161c0, 0x2891e20, 0xc4200444a0, 0x0, 0x0)
2017-09-23_01:04:22.59578       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
2017-09-23_01:04:22.59578 github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.matchHTTP2Field(0x28862a0, 0xc4200cacf8, 0x1bb4d3a, 0xc, 0x1bba11f, 0x10, 0x7f8bcb6516b0)
2017-09-23_01:04:22.59579       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/matchers.go:145 +0x14e
2017-09-23_01:04:22.59579 github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.HTTP2HeaderField.func1(0x28862a0, 0xc4200cacf8, 0xc420e01650)
2017-09-23_01:04:22.59580       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/matchers.go:111 +0x59
2017-09-23_01:04:22.59580 github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*cMux).serve(0xc4208f49c0, 0x28a76c0, 0xc420e01650, 0xc4201566c0, 0xc420044190)
2017-09-23_01:04:22.59581       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:143 +0x228
2017-09-23_01:04:22.59581 created by github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*cMux).Serve
2017-09-23_01:04:22.59581       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:133 +0x16a
2017-09-23_01:04:22.59582 
2017-09-23_01:04:22.59582 goroutine 1321884 [select]:
2017-09-23_01:04:22.59582 net/http.(*persistConn).writeLoop(0xc9e2600000)
2017-09-23_01:04:22.59583       /usr/lib/go/src/net/http/transport.go:1759 +0x165
2017-09-23_01:04:22.59583 created by net/http.(*Transport).dialConn
2017-09-23_01:04:22.59583       /usr/lib/go/src/net/http/transport.go:1187 +0xa53
2017-09-23_01:04:22.59584 
2017-09-23_01:04:22.59584 goroutine 43929 [semacquire]:
2017-09-23_01:04:22.59584 sync.runtime_notifyListWait(0xc627734b68, 0x2cae)
2017-09-23_01:04:22.59585       /usr/lib/go/src/runtime/sema.go:507 +0x110
2017-09-23_01:04:22.59585 sync.(*Cond).Wait(0xc627734b58)
2017-09-23_01:04:22.59586       /usr/lib/go/src/sync/cond.go:56 +0x80
2017-09-23_01:04:22.59586 github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc627734b40, 0xc627a284f0, 0x0, 0x0, 0x0, 0x0)
2017-09-23_01:04:22.59586       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:438 +0xb6
2017-09-23_01:04:22.59587 github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop(0xc627bfe680)
2017-09-23_01:04:22.59587       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:147 +0x40
2017-09-23_01:04:22.59588 github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.processLoop)-fm()
2017-09-23_01:04:22.59588       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x2a
2017-09-23_01:04:22.59589 github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc6279e4ef8)
2017-09-23_01:04:22.59589       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96 +0x5e
2017-09-23_01:04:22.59590 github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc6279e4ef8, 0x3b9aca00, 0x0, 0x1827401, 0xc627b226c0)
2017-09-23_01:04:22.59591       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0xa1
2017-09-23_01:04:22.59591 github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc6279e4ef8, 0x3b9aca00, 0xc627b226c0)
2017-09-23_01:04:22.59591       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d
2017-09-23_01:04:22.59592 github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).Run(0xc627bfe680, 0xc627b226c0)
2017-09-23_01:04:22.59592       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x22e
2017-09-23_01:04:22.59592 github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc627a2a280, 0xc627b226c0)
2017-09-23_01:04:22.59593       /home/grobie/code/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:213 +0x380
2017-09-23_01:04:22.59594 created by github.com/prometheus/prometheus/discovery/kubernetes.(*Discovery).Run
2017-09-23_01:04:22.59594       /home/grobie/code/go/src/github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:205 +0x497
2017-09-23_01:04:22.59594 
2017-09-23_01:04:22.59595 goroutine 41162 [select]:
2017-09-23_01:04:22.59595 github.com/prometheus/prometheus/discovery/dns.(*Discovery).Run(0xc6279a2080, 0x7f8bcb600bc8, 0xc6279a2100, 0xc6279a8180)
2017-09-23_01:04:22.59595       /home/grobie/code/go/src/github.com/prometheus/prometheus/discovery/dns/dns.go:106 +0x1b2
2017-09-23_01:04:22.59596 created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
2017-09-23_01:04:22.59596       /home/grobie/code/go/src/github.com/prometheus/prometheus/discovery/discovery.go:250 +0x26f
2017-09-23_01:04:22.59597 
2017-09-23_01:04:22.59597 goroutine 56459 [select]:
2017-09-23_01:04:22.59597 net/http.(*persistConn).writeLoop(0xc697b40360)
2017-09-23_01:04:22.59598       /usr/lib/go/src/net/http/transport.go:1759 +0x165
2017-09-23_01:04:22.59598 created by net/http.(*Transport).dialConn
2017-09-23_01:04:22.59598       /usr/lib/go/src/net/http/transport.go:1187 +0xa53
2017-09-23_01:04:22.59599 
2017-09-23_01:04:22.59599 goroutine 49703 [select]:
2017-09-23_01:04:22.59599 net/http.(*persistConn).writeLoop(0xc452fa4ea0)
2017-09-23_01:04:22.59600       /usr/lib/go/src/net/http/transport.go:1759 +0x165
2017-09-23_01:04:22.59600 created by net/http.(*Transport).dialConn
2017-09-23_01:04:22.59601       /usr/lib/go/src/net/http/transport.go:1187 +0xa53
2017-09-23_01:04:22.59601 
2017-09-23_01:04:22.59601 goroutine 43898 [select]:
2017-09-23_01:04:22.59602 github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc6278a6f70, 0xc6278b0540, 0xc627950910, 0xc6278ad3d0, 0xc6278ad3b8, 0x5, 0x2885960, 0xc6278a8540)
2017-09-23_01:04:22.59602       /home/grobie/code/go/src/github.com/prometheus/prometheus/discovery/discovery.go:273 +0x369
2017-09-23_01:04:22.59602 created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
2017-09-23_01:04:22.59603       /home/grobie/code/go/src/github.com/prometheus/prometheus/discovery/discovery.go:252 +0x2e0
2017-09-23_01:04:22.59603 
2017-09-23_01:04:22.59604 goroutine 57012 [select]:
2017-09-23_01:04:22.59604 net/http.(*persistConn).writeLoop(0xc697dd4480)
2017-09-23_01:04:22.59604       /usr/lib/go/src/net/http/transport.go:1759 +0x165
2017-09-23_01:04:22.59605 created by net/http.(*Transport).dialConn
2017-09-23_01:04:22.59605       /usr/lib/go/src/net/http/transport.go:1187 +0xa53
2017-09-23_01:04:22.59606 
...

[grobie]

One thing to note, as I compiled Prometheus locally, it uses go1.9. If there are known issues with go1.9, I can recompile using go1.8.

On another note, the whole crashdump is >50k lines.

[grobie]

This happened the third time now.

[gouthamve]

Hmm, so this is coming from the rules/alerts. I don't think this is an issue with 1.9 as nothing seems to have changed that could cause this. I have been running an instance with a bunch of different rules since last night but still cannot reproduce this.

Any chance you can share your rules?

[grobie]

It has been stable for the last day, I fear this will become difficult to debug. I shared the config and rules via PM.

[grobie]

Our Prometheus server monitoring our Kubernetes infrastructure hasn't hit this problem again during the last 7d. After the initial 3 panics during the first day, it has been stable.

[fabxc]

@grobie considered fixed?

[grobie]

None of our servers have experienced such a panic since rc.0.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.