Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Isolate corrupted series files instead of panicking. #877

Closed
TheTincho opened this Issue Jul 6, 2015 · 15 comments

Comments

Assignees

@beorn7 beorn7

Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@TheTincho @beorn7 @mdirkse @atombender @brian-brazil
@TheTincho
Copy link
Contributor

TheTincho commented Jul 6, 2015

While looking though data with the prometheus console, I get many errors in the log like this:

2015/07/06 08:37:15 http: panic serving 127.0.0.1:40562: unknown chunk encoding: 255
goroutine 661098 [running]:
net/http.func·011()
    /usr/local/go/src/net/http/server.go:1130 +0xbb
github.com/prometheus/prometheus/storage/local.newChunkForEncoding(0x7fca38e1beff, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/chunk.go:286 +0x265
github.com/prometheus/prometheus/storage/local.(*persistence).loadChunks(0xc2082b7d00, 0xb88cfd1aad811e79, 0xc2419e8000, 0x48, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:437 +0x51d
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loadChunks(0xc2080661c0, 0xb88cfd1aad811e79, 0xc2419e8000, 0x48, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:1093 +0xa0
github.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunks(0xc21f1ed1a0, 0xc225664280, 0x49, 0x49, 0xb88cfd1aad811e79, 0xc2080661c0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:323 +0x4c5
github.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunksForRange(0xc21f1ed1a0, 0x14e3f2e8450, 0x14e63443010, 0xb88cfd1aad811e79, 0xc2080661c0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:413 +0x5cc
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).preloadChunksForRange(0xc2080661c0, 0xb88cfd1aad811e79, 0x14e3f2e8450, 0x14e63443010, 0x45d964b800, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:611 +0x36b
github.com/prometheus/prometheus/storage/local.(*memorySeriesPreloader).PreloadRange(0xc2581a9da0, 0xb88cfd1aad811e79, 0x14e3f2e8450, 0x14e63443010, 0x45d964b800, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/preload.go:34 +0x82
github.com/prometheus/prometheus/promql.(*Analyzer).Prepare(0xc2218e5b00, 0x7fca1f5fc608, 0xc25a01f4a0, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/analyzer.go:136 +0x4ef
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc20800bdc0, 0x7fca1f5fc608, 0xc25a01f4a0, 0xc2218e5a80, 0xc233f4b0e0, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:411 +0x39d
github.com/prometheus/prometheus/promql.(*Engine).exec(0xc20800bdc0, 0xc2218e5a80, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:376 +0x5a7
github.com/prometheus/prometheus/promql.(*query).Exec(0xc2218e5a80, 0xc229dc3570)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:212 +0x31
github.com/prometheus/prometheus/web/api/legacy.(*API).QueryRange(0xc20801f4e0, 0x7fca1e6444e0, 0xc23fbb5160, 0xc25b999380)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/api/legacy/query.go:144 +0x92a
github.com/prometheus/prometheus/web/api/legacy.*API.QueryRange·fm(0x7fca1e6444e0, 0xc23fbb5160, 0xc25b999380)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/api/legacy/api.go:39 +0x45
net/http.HandlerFunc.ServeHTTP(0xc208101250, 0x7fca1e6444e0, 0xc23fbb5160, 0xc25b999380)
    /usr/local/go/src/net/http/server.go:1265 +0x41
github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x7fca38e1c070, 0xc208101250, 0x7fca34c53988, 0xc2294965f0, 0xc25b999380)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x9a
github.com/prometheus/prometheus/util/httputil.(*CompressionHandler).ServeHTTP(0xc208101260, 0x7fca34c53988, 0xc2294965f0, 0xc25b999380)
    <autogenerated>:5 +0xbd
net/http.Handler.ServeHTTP·fm(0x7fca34c53988, 0xc2294965f0, 0xc25b999380)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/web.go:153 +0x57
github.com/prometheus/client_golang/prometheus.func·008(0x7fca34c53950, 0xc230d6a6e0, 0xc25b999380)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:144 +0x1b0
github.com/prometheus/prometheus/util/route.func·001(0x7fca34c53950, 0xc230d6a6e0, 0xc25b999380, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/route/route.go:49 +0x427
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc2080d6180, 0x7fca34c53950, 0xc230d6a6e0, 0xc25b999380)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/julienschmidt/httprouter/router.go:299 +0x18e
github.com/prometheus/prometheus/util/route.(*Router).ServeHTTP(0xc20801f4a0, 0x7fca34c53950, 0xc230d6a6e0, 0xc25b999380)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/route/route.go:90 +0x49
net/http.serverHandler.ServeHTTP(0xc23a2cc7e0, 0x7fca34c53950, 0xc230d6a6e0, 0xc25b999380)
    /usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc230d6a0a0)
    /usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1751 +0x35e

And some times this one:

2015/07/03 11:54:57 http: panic serving 127.0.0.1:23506: runtime error: slice bounds out of range
goroutine 2779141 [running]:
net/http.func·011()
    /usr/local/go/src/net/http/server.go:1130 +0xbb
github.com/prometheus/prometheus/storage/local.(*deltaEncodedChunk).unmarshalFromBuf(0xc270845e20, 0xc298e1bf30, 0x4510, 0x4510)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/delta.go:238 +0x10f
github.com/prometheus/prometheus/storage/local.(*persistence).loadChunks(0xc2082b4900, 0x14fbde92b6c46404, 0xc270883000, 0x65, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:438 +0x5b4
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loadChunks(0xc2080621c0, 0x14fbde92b6c46404, 0xc270883000, 0x65, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:1093 +0xa0
github.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunks(0xc273a3a300, 0xc270882800, 0x7a, 0x7a, 0x14fbde92b6c46404, 0xc2080621c0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:323 +0x4c5
github.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunksForRange(0xc273a3a300, 0x14e30799800, 0x14e548f43c0, 0x14fbde92b6c46404, 0xc2080621c0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:413 +0x5cc
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).preloadChunksForRange(0xc2080621c0, 0x14fbde92b6c46404, 0x14e30799800, 0x14e548f43c0, 0x45d964b800, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:611 +0x36b
github.com/prometheus/prometheus/storage/local.(*memorySeriesPreloader).PreloadRange(0xc29ad35e00, 0x14fbde92b6c46404, 0x14e30799800, 0x14e548f43c0, 0x45d964b800, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/preload.go:34 +0x82
github.com/prometheus/prometheus/promql.(*Analyzer).Prepare(0xc299f6f7c0, 0x7fcce27fe100, 0xc2974be4e0, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/analyzer.go:136 +0x4ef
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc208057d80, 0x7fcce27fe100, 0xc2974be4e0, 0xc299f6f740, 0xc2974d5950, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:411 +0x39d
github.com/prometheus/prometheus/promql.(*Engine).exec(0xc208057d80, 0xc299f6f740, 0x0, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:376 +0x5a7
github.com/prometheus/prometheus/promql.(*query).Exec(0xc299f6f740, 0xc2974b8a80)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:212 +0x31
github.com/prometheus/prometheus/web/api/legacy.(*API).QueryRange(0xc20801f4e0, 0x7fcce27fe028, 0xc29acbf480, 0xc29a458680)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/api/legacy/query.go:144 +0x92a
github.com/prometheus/prometheus/web/api/legacy.*API.QueryRange·fm(0x7fcce27fe028, 0xc29acbf480, 0xc29a458680)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/api/legacy/api.go:39 +0x45
net/http.HandlerFunc.ServeHTTP(0xc2080d9270, 0x7fcce27fe028, 0xc29acbf480, 0xc29a458680)
    /usr/local/go/src/net/http/server.go:1265 +0x41
github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x7fcceb638070, 0xc2080d9270, 0x7fcce8b73488, 0xc297b7f400, 0xc29a458680)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x9a
github.com/prometheus/prometheus/util/httputil.(*CompressionHandler).ServeHTTP(0xc2080d9280, 0x7fcce8b73488, 0xc297b7f400, 0xc29a458680)
    <autogenerated>:5 +0xbd
net/http.Handler.ServeHTTP·fm(0x7fcce8b73488, 0xc297b7f400, 0xc29a458680)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/web.go:153 +0x57
github.com/prometheus/client_golang/prometheus.func·008(0x7fcce8b73450, 0xc29acd45a0, 0xc29a458680)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:144 +0x1b0
github.com/prometheus/prometheus/util/route.func·001(0x7fcce8b73450, 0xc29acd45a0, 0xc29a458680, 0x0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/route/route.go:49 +0x427
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc2080d4180, 0x7fcce8b73450, 0xc29acd45a0, 0xc29a458680)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/julienschmidt/httprouter/router.go:299 +0x18e
github.com/prometheus/prometheus/util/route.(*Router).ServeHTTP(0xc20801f4a0, 0x7fcce8b73450, 0xc29acd45a0, 0xc29a458680)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/route/route.go:90 +0x49
net/http.serverHandler.ServeHTTP(0xc23387d800, 0x7fcce8b73450, 0xc29acd45a0, 0xc29a458680)
    /usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc29acd4500)
    /usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1751 +0x35e

I suspect this might be some DB corruption, as the process has been killed many times (usually through SIGTERM though). I have also seen the whole server come down because of other errors that might be related. For example:

panic: Invalid number of bytes for time delta

goroutine 116 [running]:
github.com/prometheus/prometheus/storage/local.(*doubleDeltaEncodedChunkIterator).timestampAtIndex(0xc291382190, 0x2, 0x400)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/doubledelta.go:514 +0x1c2
github.com/prometheus/prometheus/storage/local.(*doubleDeltaEncodedChunkIterator).lastTimestamp(0xc291382190, 0x7fcce8b63000)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/doubledelta.go:520 +0x34
github.com/prometheus/prometheus/storage/local.(*chunkDesc).lastTime(0xc293ebca00, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/chunk.go:142 +0xc3
github.com/prometheus/prometheus/storage/local.func·019()
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:642 +0x7bd
github.com/prometheus/prometheus/storage/local.(*persistence).checkpointSeriesMapAndHeads(0xc2082b4900, 0xc208453260, 0xc20801ef00, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:657 +0x5bc
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loop(0xc2080621c0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:844 +0x24d
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:270 +0x51f

goroutine 1 [select, 1376 minutes]:
main.Main(0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:149 +0x183b
main.main()
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:46 +0x1f

goroutine 16 [syscall, 1369 minutes]:
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 89 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc208062380)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c

goroutine 88 [chan receive, 1369 minutes]:
main.func·005()
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:105 +0xbd
created by main.Main
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:108 +0xe1e

goroutine 91 [select, 1377 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b2420)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc

goroutine 92 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b2420)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6

goroutine 93 [select, 1377 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b2420)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf

goroutine 94 [select, 1377 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b2420)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9

goroutine 95 [select, 1377 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b2420)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03

goroutine 96 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc208062540)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c

goroutine 97 [select, 1374 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b2580)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc

goroutine 98 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b2580)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6

goroutine 99 [select, 1374 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b2580)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf

goroutine 100 [select, 1377 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b2580)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9

goroutine 101 [select, 1377 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b2580)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03

goroutine 102 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc2080628c0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c

goroutine 103 [select, 7 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b2840)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc

goroutine 104 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b2840)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6

goroutine 105 [select, 7 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b2840)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf

goroutine 106 [select, 7 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b2840)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9

goroutine 107 [select, 7 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b2840)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03

goroutine 108 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc2080627e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c

goroutine 109 [select, 1377 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b2b00)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc

goroutine 110 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b2b00)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6

goroutine 111 [select, 1377 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b2b00)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf

goroutine 112 [select, 1377 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b2b00)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9

goroutine 113 [select, 1377 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b2b00)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03

goroutine 114 [select]:
github.com/prometheus/prometheus/storage/local.(*persistence).processIndexingQueue(0xc2082b4900)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:1345 +0xba0
github.com/prometheus/prometheus/storage/local.(*persistence).run(0xc2082b4900)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:282 +0x28
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:246 +0x16a

goroutine 115 [select]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).handleEvictList(0xc2080621c0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:623 +0x46e
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:269 +0x502

goroutine 121 [select]:
github.com/prometheus/prometheus/rules.(*Manager).Run(0xc208011c00)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/rules/manager.go:142 +0x3c6
created by main.Main
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:131 +0x11a8

goroutine 118 [select]:
github.com/prometheus/prometheus/storage/local.func·043()
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:761 +0x300
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughMemoryFingerprints
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:777 +0xfb

goroutine 119 [select]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc2080621c0, 0x0, 0x3ff0000000000000, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:727 +0x199
github.com/prometheus/prometheus/storage/local.func·044()
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:800 +0x285
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughArchivedFingerprints
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:820 +0xfb

goroutine 122 [chan receive, 1376 minutes]:
github.com/prometheus/prometheus/notification.(*NotificationHandler).Run(0xc20805a280)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/notification/notification.go:187 +0x81
created by main.Main
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:134 +0x11eb

goroutine 133 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b905a0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 124 [IO wait]:
net.(*pollDesc).Wait(0xc2338206f0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2338206f0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc233820690, 0x0, 0x7fcceb637e78, 0xc27e9b8f80)
    /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc231d53318, 0x61e11e, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc231d53318, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1976 +0x4c
net/http.(*Server).Serve(0xc23387d800, 0x7fcce8bb4438, 0xc231d53318, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1728 +0x92
net/http.(*Server).ListenAndServe(0xc23387d800, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1718 +0x154
net/http.ListenAndServe(0xb41980, 0x5, 0x7fcce8bb3348, 0xc20801f4a0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1808 +0xba
github.com/prometheus/prometheus/web.(*Handler).Run(0xc20800a9b0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/web.go:188 +0x158
created by main.Main
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:142 +0x1294

goroutine 204 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc2264d0630, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 132 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b903f0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 131 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90240, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 134 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b906c0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 135 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90870, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 136 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90a20, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 137 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90bd0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 138 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90d80, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 139 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90ea0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 140 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90fc0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 141 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b910e0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 142 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91200, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 143 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91320, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 144 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91440, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 145 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91560, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 146 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91680, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 147 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b917a0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 148 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b918c0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 149 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b919e0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 150 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91b00, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 151 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91c20, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 152 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91dd0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 153 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91ef0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 154 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c090, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 155 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c1b0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 156 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c360, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 157 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c5a0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 158 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c6c0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 159 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c7e0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 160 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699cab0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 161 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699cbd0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 162 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699ccf0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 163 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699cf30, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 164 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d050, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 165 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d200, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 166 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d320, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 167 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d440, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 168 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d560, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 169 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d680, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 170 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d7a0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 171 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d950, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 172 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699da70, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 173 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699db90, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 174 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699dcb0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 175 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699def0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 176 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c090, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 177 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c240, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 178 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c360, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 179 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c480, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 180 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c5a0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 181 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c6c0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 182 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c7e0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 183 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c900, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 184 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679cab0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 185 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679cc60, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 186 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679cd80, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 187 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679cea0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 188 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679cfc0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 189 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d0e0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 190 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d200, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 191 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d3b0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 192 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d560, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 193 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d680, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 194 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d7a0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 195 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d8c0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 196 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679da70, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 197 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679db90, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 198 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679dcb0, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 199 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679de60, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 200 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc2264d0120, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 201 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc2264d0240, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 202 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc2264d0360, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 203 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc2264d0480, 0x7fcceb638d80, 0xc20801f3e0)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c

goroutine 209 [syscall, 1376 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 2779142 [chan send, 3 minutes]:
github.com/prometheus/client_golang/prometheus.computeApproximateRequestSize(0xc29a458680, 0xc2974be2a0, 0x28f)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:173 +0x1b9
created by github.com/prometheus/client_golang/prometheus.func·008
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:143 +0x15a

goroutine 2779137 [chan send, 3 minutes]:
github.com/prometheus/client_golang/prometheus.computeApproximateRequestSize(0xc28b13f860, 0xc295c1b980, 0x28f)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:173 +0x1b9
created by github.com/prometheus/client_golang/prometheus.func·008
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:143 +0x15a

goroutine 2779139 [chan send, 3 minutes]:
github.com/prometheus/client_golang/prometheus.computeApproximateRequestSize(0xc258fe7450, 0xc295c1be00, 0x28f)
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:173 +0x1b9
created by github.com/prometheus/client_golang/prometheus.func·008
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:143 +0x15a

goroutine 2779151 [IO wait, 1 minutes]:
net.(*pollDesc).Wait(0xc29ad0a060, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc29ad0a060, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc29ad0a000, 0xc270996000, 0x1000, 0x1000, 0x0, 0x7fcceb637e78, 0xc265ebcdf8)
    /usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2708fc000, 0xc270996000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0xdc
net/http.(*liveSwitchReader).Read(0xc27094a048, 0xc270996000, 0x1000, 0x1000, 0xc2708c58b0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc2709d6480, 0xc270996000, 0x1000, 0x1000, 0x7fcceb638da8, 0x0, 0x0)
    /usr/local/go/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc2709da000)
    /usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc2709da000, 0x5c320a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc2709da000, 0x0, 0x0, 0x0, 0xc234d5db00, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc2709c4090, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc2709c4090, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc2709da000, 0xc26ca951e0, 0x0, 0x0)
    /usr/local/go/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc27094a000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc27094a000)
    /usr/local/go/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1751 +0x35e

goroutine 106767 [IO wait, 1353 minutes]:
net.(*pollDesc).Wait(0xc2422715d0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2422715d0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc242271570, 0xc23c842000, 0x1000, 0x1000, 0x0, 0x7fcceb637e78, 0xc241bab548)
    /usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc243968fd8, 0xc23c842000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0xdc
net/http.(*liveSwitchReader).Read(0xc260f25ee8, 0xc23c842000, 0x1000, 0x1000, 0xc24da718b0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc233bf80a0, 0xc23c842000, 0x1000, 0x1000, 0x7fcceb638da8, 0x0, 0x0)
    /usr/local/go/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc23dc86000)
    /usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc23dc86000, 0x5c320a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc23dc86000, 0x0, 0x0, 0x0, 0xc234d6e200, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc243849f20, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc243849f20, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc23dc86000, 0xc24c8a4dd0, 0x0, 0x0)
    /usr/local/go/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc260f25ea0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc260f25ea0)
    /usr/local/go/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1751 +0x35e

goroutine 2727234 [chan send]:
github.com/prometheus/prometheus/storage/local.func·030()
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:129 +0xec
created by github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter
    /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:134 +0xfb

@beorn7 beorn7 self-assigned this Jul 6, 2015

@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jul 21, 2015

This really looks like severe data corruption. One might argue if panicking is the right response. If corruption of this kind happens, there is probably nothing to rescue... On the other hand, the affected series could be used to the orphaned directory for forensics...

I'll leave this open for the record, with a changed title, but not high priority to deal with it right now.

@beorn7 beorn7 changed the title panics due to storage errors Isolate corrupted series files instead of panicking. Jul 21, 2015

@mdirkse

This comment has been minimized.

Sign in to view
Copy link

mdirkse commented Sep 3, 2015

I'd second the notion that it'd be useful to know exactly where the data is corrupted and if there is anything to rescue. Also, if the server randomly fails and data gets corrupted in such a way that prometheus dies soon after it's started (as is the case at the moment for our server) then that's a problem. I'd prefer it to deduce that it has corrupt data, and then start collecting new data anyways. At least then you have some sort of monitoring.

I currently have a DB that is corrupted to the point that prometheus shuts down after a couple seconds of operation with a chunk panic. Right now I have no choice but to delete all the data and start over. Not sure what caused the corruption, probably random restarts of the process or machine.

The top of the stacktrace looks like this:

time="2015-09-03T17:21:52+02:00" level=info msg="30000 metrics queued for indexing." file=crashrecovery.go line=461 
time="2015-09-03T17:21:52+02:00" level=info msg="Indexing archived metrics." file=crashrecovery.go line=464 
time="2015-09-03T17:21:52+02:00" level=info msg="All requests for rebuilding the label indexes queued. (Actual processing may lag behind.)" file=crashrecovery.go line=483 
time="2015-09-03T17:21:52+02:00" level=warning msg="Crash recovery complete." file=crashrecovery.go line=141 
time="2015-09-03T17:21:52+02:00" level=info msg="35826 series loaded." file=storage.go line=268 
time="2015-09-03T17:21:52+02:00" level=info msg="Starting target manager..." file=targetmanager.go line=75 
time="2015-09-03T17:21:52+02:00" level=info msg="Listening on :9090" file=web.go line=186 

panic: unknown chunk encoding: 25

goroutine 683 [running]:
github.com/prometheus/prometheus/storage/local.newChunkForEncoding(0xc20f7a4719, 0x0, 0x0)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/chunk.go:287 +0x265
github.com/prometheus/prometheus/storage/local.(*persistence).loadChunks(0xc2082b5d00, 0xf85559db665a4132, 0xc20ebacea0, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:437 +0x51d
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loadChunks(0xc2080ca000, 0xf85559db665a4132, 0xc20ebacea0, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:1107 +0xa0
github.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunks(0xc20c6d5da0, 0xc20ebace70, 0x2, 0x2, 0xf85559db665a4132, 0xc2080ca000, 0x0, 0x0, 0x0, 0x0, ...)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:329 +0x4c5
github.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunksForRange(0xc20c6d5da0, 0x14f93ccf82d, 0x14f93ccf82d, 0xf85559db665a4132, 0xc2080ca000, 0x0, 0x0, 0x0, 0x0, 0x0)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:419 +0x5d7
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).preloadChunksForRange(0xc2080ca000, 0xf85559db665a4132, 0x14f93ccf82d, 0x14f93ccf82d, 0x45d964b800, 0x0, 0x0, 0x0, 0x0, 0x0)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:638 +0x36b
github.com/prometheus/prometheus/storage/local.(*memorySeriesPreloader).PreloadRange(0xc20f7adb60, 0xf85559db665a4132, 0x14f93ccf82d, 0x14f93ccf82d, 0x45d964b800, 0x0, 0x0)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/preload.go:34 +0x82
github.com/prometheus/prometheus/promql.(*Analyzer).Prepare(0xc214446740, 0x7f21dd719680, 0xc20ea2e2a0, 0x0, 0x0, 0x0, 0x0)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/promql/analyzer.go:145 +0x6b7
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc20805be80, 0x7f21dd719680, 0xc20ea2e2a0, 0xc2144466c0, 0xc20e9082a0, 0x0, 0x0, 0x0, 0x0)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:440 +0x39d
github.com/prometheus/prometheus/promql.(*Engine).exec(0xc20805be80, 0xc2144466c0, 0x0, 0x0, 0x0, 0x0)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:405 +0x5a7
github.com/prometheus/prometheus/promql.(*query).Exec(0xc2144466c0, 0xc20e908180)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:241 +0x31
github.com/prometheus/prometheus/rules.(*AlertingRule).eval(0xc2084323f0, 0x14f93ccf82d, 0xc20805be80, 0x0, 0x0, 0x0, 0x0, 0x0)
    /app/.build/root/gopath/src/github.com/prometheus/prometheus/rules/alerting.go:160 +0x15d

I did an strace on the failing process which yielded this:

stat("/prometheus/13/ca4fa04ec2417c.db", {st_mode=S_IFREG|0640, st_size=1041, ...}) = 0
unlink("/prometheus/13/ca4fa04ec2417c.db") = 0
close(23)                               = 0
write(6, "p\356\205\30\26\0\1J_\0\0\0\0\0\0\1\0\0\0\0\10\23\312O\240N\302A|", 29) = 29
write(10, "\357\264Xt\26\0\1\361\345\0\0\0\0\0\0\1\0\0\0\0\10\23\312O\240N\302A|", 29) = 29
mmap(0xc2184f0000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc2184f0000
mmap(0xc206fa0000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc206fa0000
mmap(0xc2185f0000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc2185f0000
mmap(0xc206f90000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc206f90000
pread(15, "\272 \314\0006q\20clientIdHID_Wednesday-469"..., 4058, 638939) = 4058
mmap(0xc2186f0000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc2186f0000
mmap(0xc206f80000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc206f80000
pread(15, "\206 \354\0=a\24consumerIdRID_Wednesday-5"..., 3849, 1710522) = 3849
mmap(0xc2187f0000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc2187f0000
mmap(0xc206f70000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc206f70000
mmap(0xc000042000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc000042000
futex(0xc215ae8458, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 61460051}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4bd98, FUTEX_WAIT, 0, NULL)    = 0
futex(0xc2084500d8, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 214272613}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc215ae8458, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 238835601}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc215ae8458, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 138733026}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc215ae8458, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 102111452}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc215ae8458, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 170718291}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 14072305}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 21145085}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 16775720}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 20743839}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 14572029}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 54956074}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 173570185}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 268775881}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 323268200}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 176590628}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 415511867}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 84330509}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 410686590}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 14051131}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 21140551}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 16744111}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 36696952}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 7582122}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 492263087}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 99845433}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 400015021}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 192069392}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 307784160}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 284403378}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 124899157}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 14077107}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 15739582}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4b120, FUTEX_WAKE, 1)          = 1
futex(0xc20801d958, FUTEX_WAKE, 1)      = 1
futex(0xf4ae38, FUTEX_WAIT, 0, {0, 5265673}) = -1 ETIMEDOUT (Connection timed out)
futex(0xf4bd98, FUTEX_WAIT, 0, NULLPANIC: attached pid 48264 exited with 2
 <unfinished ... exit status 2>
@mdirkse

This comment has been minimized.

Sign in to view
Copy link

mdirkse commented Sep 3, 2015

Ah, turns out that my corruption was probably due to the disk being full at a certain point in time.

@aecolley aecolley referenced this issue Nov 27, 2015

Closed

Panic in newChunkForEncoding #1243

@brian-brazil brian-brazil added the bug label Dec 16, 2015

@atombender

This comment has been minimized.

Sign in to view
Copy link
Contributor

atombender commented Jan 8, 2016

Having the same problem. Tried stracing and deleting the files that Prometheus was accessing just before crashing, several times, but it's always crashing. Is there a tool one can use to determine if a database file is valid or not? I'd rather not have to delete our entire history.
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jan 8, 2016

Currently no tool.

It will be a pretty easy fix to not crash in the case above, but my gut feeling is that those kind of corruptions will come in bulk in most cases (e.g. because you ran out of disk space), so you will have a lot of lost series anyway, and the value of your history might be questionable...

I'll definitely implement some kind of series isolation mechanism eventually, but not very soon. PRs welcome. ;)
@atombender

This comment has been minimized.

Sign in to view
Copy link
Contributor

atombender commented Jan 8, 2016

My problem was the same as the OP — ran out of disk space. I don't know anything about how Prometheus organizes its files, but presumably old history would still be valid?
@atombender

This comment has been minimized.

Sign in to view
Copy link
Contributor

atombender commented Jan 8, 2016

Considering there's apparently no way to do consistent snapshots for backup purposes, it sounds like this problem might conceivably also arise if you try to restore a backup. I'm a bit worried now.
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jan 8, 2016

Cold backups are no problem.

But even "hot" snapshots have always worked for me.

The corruption we are talking about here can only happen because of...

  1. data corruption on disk
  2. disk ran full
  3. some unknown bug in the code
@atombender

This comment has been minimized.

Sign in to view
Copy link
Contributor

atombender commented Jan 8, 2016

With "hot" snapshots, I get constant errors from tar that a file changed while it was being read — couldn't a situation arise where data inconsistencies will make Prometheus blow up?
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jan 8, 2016

Files that are incompletely written are fixed during crash recovery.

The corruptions causing the crashes are different in nature, wrong rather than incomplete data, or wrong alignment.

@beorn7 beorn7 referenced this issue Mar 2, 2016

Merged

Handle errors caused by data corruption more gracefully #1448

@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Mar 17, 2016

Fixed by #1448
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Mar 18, 2016

This bug seems to be un-closeable.... Meta bug in Github... :)

@prometheus prometheus locked and limited conversation to collaborators Mar 18, 2016

@prometheus prometheus unlocked this conversation Mar 21, 2016

@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Mar 21, 2016

I filed a support ticket about the un-closeability of this issue.

@beorn7 beorn7 closed this Mar 23, 2016

@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Mar 23, 2016

GH support made this closeable. \o/
@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 24, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.