Fatal error in persistence.go #2387

nicklan · Feb 1, 2017

Prometheus appears to have crashed persisting a memory chunk. Logs we could recover below. Let me know if you need more info. Thanks!

Environment

System information:
Linux 4.3.3-coreos x86_64
AWS m4.10xlarge
Prometheus version:

prometheus, version 1.4.1 (branch: master, revision: 2a89e8733f240d3cd57a6520b52c36ac4744ce12)
  build user:       root@e685d23d8809
  build date:       20161128-09:59:22
  go version:       go1.7.3

Logs:

fatal error: fault
goroutine 214 [running]:
runtime.sigpanic()
	/go/src/github.com/prometheus/prometheus/storage/local/persistence.go:661 +0x5bc fp=0xc67e680cf8 sp=0xc67e680c78
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).loop(0xc42028a640)
	/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc67e680fb0 sp=0xc67e680fa8

	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:230 +0x1c22

	/go/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:879 +0x7a

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x5e1

	/go/src/github.com/prometheus/prometheus/rules/manager.go:171 +0x282
os/signal.signal_recv(0x0)
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22

	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:183 +0x2e4

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:96 +0x249

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x5e1

	/usr/local/go/src/runtime/proc.go:259 +0x13a
runtime.selectgo(0xc4205c0730)
	/usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3

	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:165 +0x358

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x247

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x83c

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x263
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
goroutine 206 [select, 310 minutes]:
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
goroutine 207 [select]:
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
goroutine 208 [select, 310 minutes]:
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
goroutine 209 [select, 484 minutes]:
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
goroutine 210 [select]:
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
goroutine 83 [select]:
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
goroutine 84 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table.(*Writer).writeBlock(0xc87ca74300, 0xc87ca74358, 0x2, 0x85ae76, 0xc95bb7dd30, 0xcc2f3fa980, 0x1a)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table/writer.go:221 +0x5a
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*tWriter).append(0xc9401f8d20, 0xcc2f3fa980, 0x1a, 0x20, 0xcadc438020, 0x4360bb, 0x4360c3, 0x3, 0x1)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:396 +0xa8
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransact(0xc4203c0780, 0x1ad28a7, 0xb, 0x2659040, 0xccc0d317c0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:577 +0x6ce

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x263

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x247

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x5e1

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:96 +0x249

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x83c

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x263

	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:696 +0x3d9
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*Transaction).Commit(0xca60f46000, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:259 +0x6b3
github.com/prometheus/prometheus/storage/local/index.(*LabelPairFingerprintIndex).IndexBatch(0xc420c584b0, 0xc81e6f5740, 0xc42039cd80, 0xed022c2f0)
	/go/src/github.com/prometheus/prometheus/storage/local/persistence.go:1257 +0x108
github.com/prometheus/prometheus/storage/local.(*persistence).run(0xc42039cd80)
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:344 +0x12f

	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1639 +0x45f

	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:900 +0x838

	/go/src/github.com/prometheus/prometheus/notifier/notifier.go:205 +0x1b8

	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:279 +0x2ae

	/usr/local/go/src/runtime/sema.go:47 +0x30
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run(0xc4206751f0)
	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:216 +0x167d
net.runtime_pollWait(0x7f1328c25f00, 0x72, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*netFD).accept(0xc67e68a0e0, 0x0, 0x2654700, 0xc706dc9c60)
	/usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net/http.tcpKeepAliveListener.Accept(0xc67e6d8000, 0x1bbd1d8, 0xc961838680, 0x2664100, 0xc67e678120)
	/usr/local/go/src/net/http/server.go:2273 +0x1ce
github.com/prometheus/prometheus/web.(*Handler).Run(0xc42038ea90)
	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:223 +0x16f0

	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1211 +0x3bc

	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:183 +0x2e4
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
goroutine 96 [chan receive, 6870 minutes]:
github.com/prometheus/prometheus/discovery/kubernetes.(*Kubernetes).Run(0xc67e6bcd20, 0x7f1328c99750, 0xc67e68c340, 0xc67e69a5a0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:233 +0x2b4
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc67e6a41e0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc67e6c2000, 0x7f1328c99750, 0xc67e6b0000)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:116 +0x43

	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:357 +0xdf6
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).RunUntil.func1()
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:84 +0x19
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.Until(0xc58ed69b60, 0x3b9aca00, 0xc67e69a540)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:216 +0x1c4
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc4fe871bd0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
goroutine 229 [select]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f1328c99750, 0xc67e6b0dc0, 0xc4206751f0, 0xc67e6bd020)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:119 +0x632
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc67e6662c0, 0xc67e69a5a0, 0xc67e6c2500, 0xc67e6662e0, 0xc67e6b9620, 0xc, 0x264e740, 0xc67e6bcd20)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:270 +0x325
sync.runtime_notifyListWait(0xc67e768028, 0x3e998)
	/usr/local/go/src/sync/cond.go:57 +0x80
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Controller).processLoop(0xc67e68a150)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/controller.go:102 +0x2a
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil(0xc420c09ee0, 0x3b9aca00, 0x0, 0x179fc01, 0xc67e69a540)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:47 +0x4d
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*sharedIndexInformer).Run(0xc67e68e380, 0xc67e69a540)
	/go/src/github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:145 +0x1adc
sync.runtime_notifyListWait(0xc5b2db2238, 0x337)
	/usr/local/go/src/sync/cond.go:57 +0x80
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Controller).processLoop(0xc67e72e070)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/controller.go:102 +0x2a
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil(0xc67e771ee0, 0x3b9aca00, 0x0, 0x179fc01, 0xc67e69a540)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:47 +0x4d

	/usr/local/go/src/runtime/sema.go:267 +0x122
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*DeltaFIFO).Pop(0xc67e766000, 0xc67e666330, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/controller.go:126 +0x3c
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil.func1(0xc67e6fdee0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:85 +0xad
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Controller).Run(0xc67e68e4d0, 0xc67e69a540)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:184 +0x322

	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:357 +0xdf6
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).RunUntil.func1()
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:84 +0x19
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.Until(0xc67e67a420, 0x3b9aca00, 0xc67e69a540)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:216 +0x1c4
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).watchHandler(0xc67e698080, 0x2659c80, 0xca810720f0, 0xc7ff1afc60, 0xc67edd2840, 0xc67e69a540, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:328 +0xa6f
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil.func1(0xc67e65a140)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:85 +0xad
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).RunUntil
goroutine 326 [semacquire]:
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*processorListener).pop(0xc67eaba900, 0xc67e604a20)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:258 +0x27e
net.runtime_pollWait(0x7f1328c25d80, 0x72, 0x1a)
	/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*netFD).Read(0xc67e72e2a0, 0xc67ebd6000, 0x8000, 0x8000, 0x0, 0x2657440, 0xc420018150)
	/usr/local/go/src/net/net.go:173 +0x70
crypto/tls.(*Conn).readRecord(0xc67e784700, 0x1bbd917, 0xc67e784808, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1113 +0x116
bufio.(*Reader).Read(0xc67e6b3320, 0xc67e92e578, 0x9, 0x9, 0x0, 0xc4205a8800, 0x11d4)
	/usr/local/go/src/io/io.go:307 +0xa4
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.readFrameHeader(0xc67e92e578, 0x9, 0x9, 0x264dd80, 0xc67e6b3320, 0x0, 0xc600000000, 0x21cb, 0x4000)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/frame.go:464 +0xa4
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0xc67e938ea0)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/transport.go:492 +0x89b
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc67e8d4140, 0x7f1328c99750, 0xc67e6b1800)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:116 +0x43

	/go/src/github.com/prometheus/prometheus/discovery/kubernetes/node.go:98 +0x524
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
goroutine 289 [select]:
sync.runtime_notifyListWait(0xc67dd8a868, 0xf0cb1)
	/usr/local/go/src/sync/cond.go:57 +0x80
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Controller).processLoop(0xc67e6177a0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/controller.go:102 +0x2a
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil(0xc67e685ee0, 0x3b9aca00, 0x0, 0x179fc01, 0xc67e604a20)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:47 +0x4d
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*sharedIndexInformer).Run(0xc67e617730, 0xc67e604a20)
	/go/src/github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:192 +0x580
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).watchHandler(0xc67e3c3080, 0x2659c80, 0xc947d2e150, 0xca1f805c70, 0xc67e658f00, 0xc67e604a20, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:328 +0xa6f
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil.func1(0xc67e63f360)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:85 +0xad
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).RunUntil
goroutine 341 [select]:
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).ListAndWatch
goroutine 231 [select]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f1328c99750, 0xc67e8fe040, 0xc4206751f0, 0xc67e6bde60)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:119 +0x632
github.com/prometheus/prometheus/discovery/kubernetes.(*Pod).Run(0xc67e63cfc0, 0x7f1328c99750, 0xc67e6410c0, 0xc67e605260)
	/go/src/github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:171 +0xda5
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
goroutine 297 [semacquire]:
sync.(*Cond).Wait(0xc67dd8a908)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/delta_fifo.go:407 +0xc5
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Controller).(github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.processLoop)-fm()
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:84 +0x19
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.Until(0xc67e76aee0, 0x3b9aca00, 0xc67e605200)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/controller.go:102 +0x1af
created by github.com/prometheus/prometheus/discovery/kubernetes.(*Kubernetes).Run
goroutine 298 [select]:
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).ListAndWatch(0xc67e3c3280, 0xc67e605200, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:213 +0x33
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil(0xc67e63f760, 0x3b9aca00, 0x0, 0xc67e583001, 0xc67e605200)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:47 +0x4d

	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:279 +0x2ae

	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:279 +0x2ae

	/go/src/github.com/prometheus/prometheus/discovery/kubernetes/pod.go:106 +0x74c
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc67e755840, 0xc67e6b2ae0, 0xc67e8d4be0, 0xc67e755860, 0xc67e755830, 0xc, 0x264e740, 0xc67e9044b0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:270 +0x325
sync.runtime_notifyListWait(0xc67e944028, 0x80ca)
	/usr/local/go/src/sync/cond.go:57 +0x80
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Controller).processLoop(0xc67e7356c0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/controller.go:102 +0x2a
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil(0xc67e682ee0, 0x3b9aca00, 0x0, 0x179fc01, 0xc67e6b2a80)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:47 +0x4d
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*sharedIndexInformer).Run(0xc67e735650, 0xc67e6b2a80)
	/go/src/github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:166 +0xd3c
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).watchHandler(0xc67e930880, 0x2659c80, 0xca949a2540, 0xc828279c70, 0xc683e3d080, 0xc67e6b2a80, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:328 +0xa6f
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil.func1(0xc67e936620)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:85 +0xad
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).RunUntil
goroutine 309 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 308 [select]:
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).ListAndWatch
goroutine 325 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc67f53af00, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc67f53af50, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc67f53afa0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc67f53aff0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc67f53b040, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc67f53b090, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc67f53b0e0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc67f53b130, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc67f53b180, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc67f53b1d0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 384 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 385 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 386 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 405 [select]:
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*sharedIndexInformer).AddEventHandler
goroutine 266 [select]:
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*sharedIndexInformer).AddEventHandler
goroutine 354 [select]:
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).ListAndWatch
goroutine 419 [select]:
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*sharedIndexInformer).AddEventHandler
goroutine 420 [semacquire]:
sync.(*Cond).Wait(0xc6814f8398)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:365 +0xd3
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*sharedIndexInformer).AddEventHandler
goroutine 421 [select]:
sync.runtime_notifyListWait(0xc6814f84a8, 0x36b)
	/usr/local/go/src/sync/cond.go:57 +0x80
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*processorListener).pop(0xc6814f8480, 0xc67e69a540)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:258 +0x27e
sync.runtime_notifyListWait(0xc67e6c0728, 0x7f96)
	/usr/local/go/src/sync/cond.go:57 +0x80
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*processorListener).pop(0xc67e6c0700, 0xc67e6b2a80)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:258 +0x27e
sync.runtime_notifyListWait(0xc6866b9aa8, 0x7f86)
	/usr/local/go/src/sync/cond.go:57 +0x80
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*processorListener).pop(0xc6866b9a80, 0xc67e605200)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:258 +0x27e
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6869e7040, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6869e7090, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6869e71d0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6869e7220, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/usr/local/go/src/net/http/transport.go:1840 +0x93b
net/http.(*Client).doFollowingRedirects(0xc67e6bd6b0, 0xc7ece1c5a0, 0x1bbd388, 0x3, 0xc67f30a901, 0x41413f)
	/usr/local/go/src/net/http/client.go:184 +0x1ea
github.com/prometheus/prometheus/retrieval.(*targetScraper).scrape(0xc6876eb560, 0x7f13178196d0, 0xca355cb1a0, 0xed022c32c, 0x2dfda385, 0x2700300, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:423 +0x4af

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 459 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 460 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 461 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 462 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 465 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 466 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 468 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 469 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 470 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 471 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878ca050, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878ca0a0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878ca140, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878ca190, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878ca280, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878ca2d0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878ca370, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878ca410, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878ca460, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878ca4b0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 507 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 508 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 510 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 512 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 514 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 515 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 516 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 520 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 521 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 522 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb130, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb180, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb1d0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb220, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb2c0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
net/http.(*persistConn).roundTrip(0xc5b3f5e700, 0xca172bc7a0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/transport.go:380 +0x4ee
net/http.(*Client).send(0xc67e6bd6b0, 0xcb4d81d2c0, 0x0, 0x0, 0x0, 0xcb0c9d30f0, 0x0, 0x1)
	/usr/local/go/src/net/http/client.go:528 +0x5e5
github.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp.Do(0x7f13178196d0, 0xc4c8c27860, 0xc67e6bd6b0, 0xcb4d81d1d0, 0x89, 0x0, 0xcb4d81d1d0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:323 +0x162
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 530 [select]:
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
goroutine 531 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb4a0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb4f0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb540, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb590, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb5e0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb680, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb6d0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb770, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb810, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cb860, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc

	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:436 +0x3cc
goroutine 560 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cbd60, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cbdb0, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:240 +0x3e5
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc6878cbe00, 0x6fc23ac00, 0x2540be400, 0x0)

beorn7 · Feb 1, 2017

That dump looks weird to me. I cannot really read from it where the panic occurs.
Is that just the stderr output?

This might be something further down, Go runtime problem or hardware problem.
Is that happening repeatedly? If yes, can you upgrade to 1.5, which uses a newer Go version.

guoshimin · Feb 1, 2017

(I'm @nicklan's coworker)

We are using the official prom/prometheus:v1.4.1 docker container. What version of Go was that built with? This is the first time we've seen it. It may or may not have happened before, but definitely not frequently.

beorn7 · Feb 6, 2017

You can check the Go version on the "Status → Runtime & Build Information" page.

I cannot read anything useful from the panic above. I'll close this issue. Please re-open if it happens again, and provide a normal goroutine dump.

lock · Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

beorn7 closed this Feb 6, 2017

lock bot locked and limited conversation to collaborators Mar 24, 2019

prometheus/prometheus

Fatal error in persistence.go #2387

Fatal error in persistence.go #2387

nicklan commented Feb 1, 2017

This comment has been minimized.

beorn7 commented Feb 1, 2017

This comment has been minimized.

guoshimin commented Feb 1, 2017

This comment has been minimized.

beorn7 commented Feb 6, 2017

beorn7 closed this Feb 6, 2017

This comment has been minimized.

lock bot commented Mar 24, 2019

lock bot locked and limited conversation to collaborators Mar 24, 2019

prometheus/prometheus

Join GitHub today

Fatal error in persistence.go #2387

Comments

nicklan commented Feb 1, 2017

This comment has been minimized.

beorn7 commented Feb 1, 2017

This comment has been minimized.

guoshimin commented Feb 1, 2017

This comment has been minimized.

beorn7 commented Feb 6, 2017

beorn7 closed this Feb 6, 2017

This comment has been minimized.

lock bot commented Mar 24, 2019

lock bot locked and limited conversation to collaborators Mar 24, 2019