Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nil pointer crash in compaction curator #392

Closed
juliusv opened this Issue May 21, 2014 · 3 comments

Comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@juliusv @matttproud
@juliusv
Copy link
Member

juliusv commented May 21, 2014

Brian Brazil is regularly encountering this crash in the compaction curator:

I0521 13:44:07.436742 15539 tiered.go:313] Flushing samples to disk...
I0521 13:44:07.437431 15539 tiered.go:324] Writing 3534 samples...
I0521 13:44:07.453735 15539 curator.go:336] Curating 01416687721567363777-_-8-9...
I0521 13:44:07.454042 15539 curator.go:336] Curating 01440054640060257063-_-7-1...
I0521 13:44:07.454703 15539 curator.go:336] Curating 01451099465613467354-_-3-9...
I0521 13:44:07.497744 15539 curator.go:336] Curating 01471559311769181855-_-0-9...
I0521 13:44:07.498027 15539 curator.go:336] Curating 01606155229084260876-_-7-r...
I0521 13:44:07.498608 15539 curator.go:336] Curating 01662171370440346787-_-4-9...
I0521 13:44:07.499321 15539 curator.go:336] Curating 01729910661288580458-_-6-g...
I0521 13:44:07.499994 15539 curator.go:336] Curating 01814014444408188467-_-4-s...
I0521 13:44:07.500503 15539 curator.go:336] Curating 01837249206601870817-_-4-t...
I0521 13:44:07.500824 15539 curator.go:336] Curating 01941743549851231537-_-9-t...
I0521 13:44:07.501065 15539 curator.go:336] Curating 01970935242391824013-_-5-5...
I0521 13:44:07.501608 15539 curator.go:336] Curating 02144333964455548649-_-9-s...
I0521 13:44:07.502233 15539 curator.go:336] Curating 02261264228156541201-_-0-s...
I0521 13:44:07.502780 15539 curator.go:336] Curating 02327171372117302045-_-3-r...
I0521 13:44:07.503539 15539 curator.go:336] Curating 02341660258630150082-_-6-k...
I0521 13:44:07.512260 15539 curator.go:336] Curating 02346497262117238201-_-8-e...
I0521 13:44:07.513011 15539 curator.go:336] Curating 02390177016617661977-_-2-s...
I0521 13:44:07.513819 15539 curator.go:336] Curating 02401995445572852413-_-9-s...
I0521 13:44:07.514409 15539 curator.go:336] Curating 02411497020961948501-_-4-s...
I0521 13:44:07.515142 15539 curator.go:336] Curating 02481919835234196377-_-5-s...
I0521 13:44:07.515789 15539 curator.go:336] Curating 02597508809804401091-_-6-5...
I0521 13:44:07.516332 15539 curator.go:336] Curating 02775961821224806906-_-6-r...
I0521 13:44:07.517077 15539 curator.go:336] Curating 02784830415456707011-_-7-e...
I0521 13:44:07.517794 15539 curator.go:336] Curating 02833557479258027598-_-3-t...
I0521 13:44:07.518427 15539 curator.go:336] Curating 02907528775206078261-_-0-t...
I0521 13:44:07.519081 15539 curator.go:336] Curating 02987574650877550367-_-7-9...
I0521 13:44:07.519717 15539 curator.go:336] Curating 03040287266226324621-_-5-9...
I0521 13:44:07.647656 15539 curator.go:336] Curating 03053923783400927315-_-2-e...
I0521 13:44:07.648030 15539 curator.go:336] Curating 03219724117074421696-_-7-5...
I0521 13:44:07.648341 15539 curator.go:336] Curating 03233630492714976575-_-4-5...
I0521 13:44:07.649129 15539 curator.go:336] Curating 03241100289372500203-_-5-s...
I0521 13:44:07.649711 15539 curator.go:336] Curating 03333222850181259365-_-4-5...
I0521 13:44:07.658529 15539 curator.go:336] Curating 03359367521074392271-_-5-g...
I0521 13:44:07.659117 15539 curator.go:336] Curating 03429257346714730163-_-9-t...
I0521 13:44:07.659464 15539 curator.go:336] Curating 03465632722383485859-_-7-5...
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x38 pc=0x4a5f34]

goroutine 26 [running]:
runtime.panic(0x845de0, 0x10c8b88)
        /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/prometheus/prometheus/storage/metric/tiered.(*CompactionProcessor).Apply(0xc211b65ec0, 0x7f534e395d48, 0xc211b65f00, 0x7f5348031740, 0xc21005ade0, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/processor.go:236 +0x704
github.com/prometheus/prometheus/storage/metric/tiered.(*watermarkScanner).Operate(0xc211fba990, 0x88bfa0, 0xc21307f990, 0x847cc0, 0xc213081960, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/curator.go:408 +0xa3b
github.com/prometheus/prometheus/storage/raw/leveldb.(*LevelDBPersistence).ForEach(0xc21005af60, 0x7f53480317b0, 0xc211fba990, 0x7f53480317e0, 0xc211fba990, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/raw/leveldb/leveldb.go:509 +0x332
github.com/prometheus/prometheus/storage/metric/tiered.(*Curator).Run(0xc2134c24c0, 0x45d964b800, 0x537cada7, 0x7f5348031700, 0xc211b65ec0, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/curator.go:203 +0x748
main.(*prometheus).compact(0xc211965000, 0x45d964b800, 0x1f4, 0x0, 0x0)
        /home/xxxxxxxx/workspace/prometheus/main.go:140 +0x37b
main.func·004()
        /home/xxxxxxxx/workspace/prometheus/main.go:360 +0x163
created by main.main
        /home/xxxxxxxx/workspace/prometheus/main.go:367 +0x100a

goroutine 1 [chan receive]:
main.main()
        /home/xxxxxxxx/workspace/prometheus/main.go:389 +0x10d3

goroutine 3 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x10e5360)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/golang/glog/glog.go:839 +0x50
created by github.com/golang/glog.init·1
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/golang/glog/glog.go:406 +0x276

goroutine 4 [syscall]:
runtime.goexit()
        /usr/local/go/src/pkg/runtime/proc.c:1394

goroutine 7 [syscall]:
os/signal.loop()
        /usr/local/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
        /usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x31

goroutine 23 [chan receive]:
github.com/prometheus/prometheus/notification.func·001()
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/notification/notification.go:171 +0x48
created by github.com/prometheus/prometheus/notification.(*NotificationHandler).Run
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/notification/notification.go:174 +0xa8

goroutine 22 [select]:
github.com/prometheus/prometheus/storage/metric/tiered.(*TieredStorage).Serve(0xc210057480, 0xc211956360)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:271 +0x467
created by main.main
        /home/xxxxxxxx/workspace/prometheus/main.go:352 +0xf83

goroutine 20 [select]:
github.com/prometheus/prometheus/rules.(*ruleManager).Run(0xc21005acc0)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/rules/manager.go:89 +0x2e6
created by main.main
        /home/xxxxxxxx/workspace/prometheus/main.go:285 +0x859

goroutine 16 [semacquire]:
sync.runtime_Semacquire(0xc211b7eb30)
        /usr/local/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(*WaitGroup).Wait(0xc21217efe0)
        /usr/local/go/src/pkg/sync/waitgroup.go:127 +0x14b
github.com/prometheus/prometheus/retrieval.(*TargetPool).runIteration(0xc2101af690, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:164 +0x399
github.com/prometheus/prometheus/retrieval.(*TargetPool).Run(0xc2101af690, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:61 +0xd0
created by github.com/prometheus/prometheus/retrieval.(*targetManager).TargetPoolForJob
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:73 +0x3f5

goroutine 21 [chan receive]:
github.com/prometheus/prometheus/notification.(*NotificationHandler).Run(0xc2100e7120)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/notification/notification.go:177 +0xf5
created by main.main
        /home/xxxxxxxx/workspace/prometheus/main.go:288 +0x89f

goroutine 19 [semacquire]:
sync.runtime_Semacquire(0xc211b7eb60)
        /usr/local/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(*WaitGroup).Wait(0xc211c2d2c0)
        /usr/local/go/src/pkg/sync/waitgroup.go:127 +0x14b
github.com/prometheus/prometheus/retrieval.(*TargetPool).runIteration(0xc2101af930, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:164 +0x399
github.com/prometheus/prometheus/retrieval.(*TargetPool).Run(0xc2101af930, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:61 +0xd0
created by github.com/prometheus/prometheus/retrieval.(*targetManager).TargetPoolForJob
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:73 +0x3f5

goroutine 17 [select]:
github.com/prometheus/prometheus/retrieval.(*TargetPool).Run(0xc2101af770, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:59 +0x276
created by github.com/prometheus/prometheus/retrieval.(*targetManager).TargetPoolForJob
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:73 +0x3f5

goroutine 18 [semacquire]:
sync.runtime_Semacquire(0xc211b7eb50)
        /usr/local/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(*WaitGroup).Wait(0xc211c2d1a0)
        /usr/local/go/src/pkg/sync/waitgroup.go:127 +0x14b
github.com/prometheus/prometheus/retrieval.(*TargetPool).runIteration(0xc2101af850, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:164 +0x399
github.com/prometheus/prometheus/retrieval.(*TargetPool).Run(0xc2101af850, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:61 +0xd0
created by github.com/prometheus/prometheus/retrieval.(*targetManager).TargetPoolForJob
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:73 +0x3f5

goroutine 24 [chan receive]:
github.com/prometheus/prometheus/storage/metric/tiered.func·035()
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:264 +0x48
created by github.com/prometheus/prometheus/storage/metric/tiered.(*TieredStorage).Serve
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:267 +0x18e

goroutine 25 [chan receive]:
main.(*prometheus).interruptHandler(0xc211965000)
        /home/xxxxxxxx/workspace/prometheus/main.go:100 +0x107
created by main.main
        /home/xxxxxxxx/workspace/prometheus/main.go:355 +0xfba

goroutine 27 [chan receive]:
main.func·005()
        /home/xxxxxxxx/workspace/prometheus/main.go:370 +0x50
created by main.main
        /home/xxxxxxxx/workspace/prometheus/main.go:379 +0x105a

goroutine 28 [IO wait]:
net.runtime_pollWait(0x7f534e38fbf0, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc2101afed0, 0x72, 0x7f534e38c0e8, 0xb)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc2101afed0, 0xb, 0x7f534e38c0e8)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).accept(0xc2101afe70, 0x9dc1f8, 0x0, 0x7f534e38c0e8, 0xb)
        /usr/local/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(*TCPListener).AcceptTCP(0xc210000f90, 0x541abb, 0x7f53485abd88, 0x541abb)
        /usr/local/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(*TCPListener).Accept(0xc210000f90, 0x7f534e38fc98, 0xc21212f5b8, 0xc2128cfa80, 0x0)
        /usr/local/go/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(*Server).Serve(0xc2101a4aa0, 0x7f534e38ec50, 0xc210000f90, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/server.go:1622 +0x91
net/http.(*Server).ListenAndServe(0xc2101a4aa0, 0xc2101a4aa0, 0x7f534e38ebd0)
        /usr/local/go/src/pkg/net/http/server.go:1612 +0xa0
net/http.ListenAndServe(0x7fff924efdcf, 0x5, 0x7f534e38ebd0, 0xc210000488, 0xc211963160, ...)
        /usr/local/go/src/pkg/net/http/server.go:1677 +0x6d
github.com/prometheus/prometheus/web.WebService.ServeForever(0xc210057750, 0xc211956240, 0xc21018ca80, 0xc2101b06a0, 0xc211963060, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/web.go:89 +0x629
main.func·006()
        /home/xxxxxxxx/workspace/prometheus/main.go:382 +0x3d
created by main.main
        /home/xxxxxxxx/workspace/prometheus/main.go:386 +0x10aa

goroutine 9602 [IO wait]:
net.runtime_pollWait(0x7f534e38f8a8, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc21265ba70, 0x72, 0x7f534e38c0e8, 0xb)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc21265ba70, 0xb, 0x7f534e38c0e8)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).Read(0xc21265ba10, 0xc211e4a000, 0x1000, 0x1000, 0x0, ...)
        /usr/local/go/src/pkg/net/fd_unix.go:204 +0x2a0
net.(*conn).Read(0xc21212f5b8, 0xc211e4a000, 0x1000, 0x1000, 0x7f534e383000, ...)
        /usr/local/go/src/pkg/net/net.go:122 +0xc5
net/http.(*liveSwitchReader).Read(0xc2128cfaa8, 0xc211e4a000, 0x1000, 0x1000, 0x3, ...)
        /usr/local/go/src/pkg/net/http/server.go:204 +0xa5
io.(*LimitedReader).Read(0xc21212d8a0, 0xc211e4a000, 0x1000, 0x1000, 0xc211f547c8, ...)
        /usr/local/go/src/pkg/io/io.go:398 +0xbb
bufio.(*Reader).fill(0xc211cac960)
        /usr/local/go/src/pkg/bufio/bufio.go:91 +0x110
bufio.(*Reader).ReadSlice(0xc211cac960, 0xc20000000a, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/bufio/bufio.go:274 +0x204
bufio.(*Reader).ReadLine(0xc211cac960, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/bufio/bufio.go:305 +0x63
net/textproto.(*Reader).readLineSlice(0xc211c17690, 0x7f534e383000, 0x766440, 0x7f5348573ce8, 0x428932, ...)
        /usr/local/go/src/pkg/net/textproto/reader.go:55 +0x61
net/textproto.(*Reader).ReadLine(0xc211c17690, 0xc211f55b60, 0x0, 0xc211e4b000, 0x0)
        /usr/local/go/src/pkg/net/textproto/reader.go:36 +0x27
net/http.ReadRequest(0xc211cac960, 0xc211f55b60, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/request.go:526 +0x88
net/http.(*conn).readRequest(0xc2128cfa80, 0x0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/server.go:575 +0x1bb
net/http.(*conn).serve(0xc2128cfa80)
        /usr/local/go/src/pkg/net/http/server.go:1123 +0x3b4
created by net/http.(*Server).Serve
        /usr/local/go/src/pkg/net/http/server.go:1644 +0x28b

goroutine 9846 [runnable]:
github.com/prometheus/prometheus/storage/metric/tiered.findUnindexed(0x7f534e38d818, 0xc210000b98, 0xc211eb2690, 0x7f5348591f38, 0x7f5348591ee0, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/index.go:566 +0x158
github.com/prometheus/prometheus/storage/metric/tiered.(*TotalIndexer).IndexMetrics(0xc21018cd80, 0xc211eb2690, 0x86efcf16286c77f, 0x903870)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/index.go:663 +0x46
github.com/prometheus/prometheus/storage/metric/tiered.func·015(0xc2120bba50)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/leveldb.go:334 +0x157
created by github.com/prometheus/prometheus/storage/metric/tiered.(*LevelDBPersistence).AppendSamples
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/leveldb.go:335 +0x1c0

goroutine 9603 [runnable]:
net.runtime_pollWait(0x7f534e38f758, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc211d8aca0, 0x72, 0x7f534e38c0e8, 0xb)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc211d8aca0, 0xb, 0x7f534e38c0e8)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).Read(0xc211d8ac40, 0xc212130000, 0x1000, 0x1000, 0x0, ...)
        /usr/local/go/src/pkg/net/fd_unix.go:204 +0x2a0
net.(*conn).Read(0xc21212f5f8, 0xc212130000, 0x1000, 0x1000, 0x41b61d, ...)
        /usr/local/go/src/pkg/net/net.go:122 +0xc5
bufio.(*Reader).fill(0xc211bc9f00)
        /usr/local/go/src/pkg/bufio/bufio.go:91 +0x110
bufio.(*Reader).Peek(0xc211bc9f00, 0x1, 0x2, 0x2, 0x0, ...)
        /usr/local/go/src/pkg/bufio/bufio.go:119 +0xcb
net/http.(*persistConn).readLoop(0xc2128cfb00)
        /usr/local/go/src/pkg/net/http/transport.go:687 +0xb7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/pkg/net/http/transport.go:528 +0x607

goroutine 9604 [select]:
net/http.(*persistConn).writeLoop(0xc2128cfb00)
        /usr/local/go/src/pkg/net/http/transport.go:791 +0x271
created by net/http.(*Transport).dialConn
        /usr/local/go/src/pkg/net/http/transport.go:529 +0x61e

goroutine 9608 [select]:
net/http.(*persistConn).writeLoop(0xc2128cfc00)
        /usr/local/go/src/pkg/net/http/transport.go:791 +0x271
created by net/http.(*Transport).dialConn
        /usr/local/go/src/pkg/net/http/transport.go:529 +0x61e

goroutine 9586 [runnable]:
net.runtime_pollWait(0x7f534e38f6b0, 0x77, 0x0)
        /usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc211a95df0, 0x77, 0x7f534e38c0e8, 0x73)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitWrite(0xc211a95df0, 0x73, 0x7f534e38c0e8)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:90 +0x30
net.(*netFD).connect(0xc211a95d90, 0x0, 0x0, 0x7f534e393dd0, 0xc211e3e660, ...)
        /usr/local/go/src/pkg/net/fd_unix.go:86 +0x166
net.(*netFD).dial(0xc211a95d90, 0x7f534e38ec88, 0x0, 0x7f534e38ec88, 0xc211e38f00, ...)
        /usr/local/go/src/pkg/net/sock_posix.go:121 +0x2fd
net.socket(0x8eeb00, 0x3, 0x2, 0x1, 0x0, ...)
        /usr/local/go/src/pkg/net/sock_posix.go:91 +0x40b
net.internetSocket(0x8eeb00, 0x3, 0x7f534e38ec88, 0x0, 0x7f534e38ec88, ...)
        /usr/local/go/src/pkg/net/ipsock_posix.go:136 +0x161
net.dialTCP(0x8eeb00, 0x3, 0x0, 0xc211e38f00, 0xecb0ea4e3, ...)
        /usr/local/go/src/pkg/net/tcpsock_posix.go:155 +0xef
net.dialSingle(0x8eeb00, 0x3, 0xc2100b6bb5, 0x15, 0x0, ...)
        /usr/local/go/src/pkg/net/dial.go:225 +0x3d8
net.func·015(0xecb0ea4e3, 0x20985ddc, 0x10e5140, 0xdf8475800, 0xecb0ea4e3, ...)
        /usr/local/go/src/pkg/net/dial.go:158 +0xde
net.dial(0x8eeb00, 0x3, 0x7f534e38ec20, 0xc211e38f00, 0x7f534ac41c68, ...)
        /usr/local/go/src/pkg/net/fd_unix.go:40 +0x45
net.(*Dialer).Dial(0xc2133ff600, 0x8eeb00, 0x3, 0xc2100b6bb5, 0x15, ...)
        /usr/local/go/src/pkg/net/dial.go:165 +0x3e0
net.DialTimeout(0x8eeb00, 0x3, 0xc2100b6bb5, 0x15, 0xdf8475800, ...)
        /usr/local/go/src/pkg/net/dial.go:145 +0xaa
github.com/prometheus/prometheus/utility.func·001(0x8eeb00, 0x3, 0xc2100b6bb5, 0x15, 0x0, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/utility/deadline_client.go:33 +0xb8
net/http.(*Transport).dial(0xc210061a80, 0x8eeb00, 0x3, 0xc2100b6bb5, 0x15, ...)
        /usr/local/go/src/pkg/net/http/transport.go:399 +0x89
net/http.(*Transport).dialConn(0xc210061a80, 0xc2120f6c00, 0x41fc60, 0xc210057480, 0x7f534e395b30)
        /usr/local/go/src/pkg/net/http/transport.go:444 +0x6e
net/http.func·014()
        /usr/local/go/src/pkg/net/http/transport.go:419 +0x3e
created by net/http.(*Transport).getConn
        /usr/local/go/src/pkg/net/http/transport.go:421 +0x11a

goroutine 9607 [chan receive]:
net/http.(*persistConn).readLoop(0xc2128cfc00)
        /usr/local/go/src/pkg/net/http/transport.go:778 +0x68f
created by net/http.(*Transport).dialConn
        /usr/local/go/src/pkg/net/http/transport.go:528 +0x607

goroutine 9567 [select]:
net/http.(*persistConn).roundTrip(0xc2128cfb00, 0xc212056f00, 0xc2128cfb00, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/transport.go:879 +0x6d6
net/http.(*Transport).RoundTrip(0xc210061d00, 0xc211c8a410, 0x1, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/transport.go:187 +0x391
net/http.send(0xc211c8a410, 0x7f534e38c2b8, 0xc210061d00, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/net/http/client.go:168 +0x37f
net/http.(*Client).send(0xc2101b1960, 0xc211c8a410, 0x1d, 0xc2120f65a0, 0x428a01)
        /usr/local/go/src/pkg/net/http/client.go:100 +0xd9
net/http.(*Client).doFollowingRedirects(0xc2101b1960, 0xc211c8a410, 0x9dd198, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/net/http/client.go:294 +0x671
net/http.(*Client).Do(0xc2101b1960, 0xc211c8a410, 0x6, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/client.go:129 +0x8f
github.com/prometheus/prometheus/retrieval.(*target).scrape(0xc2101a4780, 0x537cada7, 0x7f534e38e3e8, 0xc2101b1870, 0x0, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:223 +0x1ba
github.com/prometheus/prometheus/retrieval.(*target).Scrape(0xc2101a4780, 0xecb0ea4a7, 0x19ff1aa7, 0x10e5140, 0x7f534e38e3e8, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:187 +0x5a
github.com/prometheus/prometheus/retrieval.(*TargetPool).runSingle(0xc2101af690, 0xecb0ea4a7, 0x19ff1aa7, 0x10e5140, 0x7f534e38e3e8, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:124 +0xa8
github.com/prometheus/prometheus/retrieval.func·003(0x7f534e38e4c8, 0xc2101a4780)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:159 +0x69
created by github.com/prometheus/prometheus/retrieval.(*TargetPool).runIteration
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:161 +0x53d

goroutine 9573 [runnable]:
net/http.(*Transport).getConn(0xc210061a00, 0xc2120f6b70, 0xc2120f6b70, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/transport.go:424 +0x24d
net/http.(*Transport).RoundTrip(0xc210061a00, 0xc211c8a8f0, 0x1, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/transport.go:182 +0x340
net/http.send(0xc211c8a8f0, 0x7f534e38c2b8, 0xc210061a00, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/net/http/client.go:168 +0x37f
net/http.(*Client).send(0xc2101b1ed0, 0xc211c8a8f0, 0x24, 0xc2120f6b10, 0x428a01)
        /usr/local/go/src/pkg/net/http/client.go:100 +0xd9
net/http.(*Client).doFollowingRedirects(0xc2101b1ed0, 0xc211c8a8f0, 0x9dd198, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/net/http/client.go:294 +0x671
net/http.(*Client).Do(0xc2101b1ed0, 0xc211c8a8f0, 0x6, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/client.go:129 +0x8f
github.com/prometheus/prometheus/retrieval.(*target).scrape(0xc2101a4960, 0x537cada7, 0x7f534e38e3e8, 0xc2101b1870, 0x0, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:223 +0x1ba
github.com/prometheus/prometheus/retrieval.(*target).Scrape(0xc2101a4960, 0xecb0ea4a7, 0x19ff6b38, 0x10e5140, 0x7f534e38e3e8, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:187 +0x5a
github.com/prometheus/prometheus/retrieval.(*TargetPool).runSingle(0xc2101af930, 0xecb0ea4a7, 0x19ff6b38, 0x10e5140, 0x7f534e38e3e8, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:124 +0xa8
github.com/prometheus/prometheus/retrieval.func·003(0x7f534e38e4c8, 0xc2101a4960)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:159 +0x69
created by github.com/prometheus/prometheus/retrieval.(*TargetPool).runIteration
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:161 +0x53d

goroutine 9578 [chan receive]:
github.com/prometheus/prometheus/storage/metric/tiered.(*LevelDBPersistence).AppendSamples(0xc210061b00, 0xc2134cf000, 0xdce, 0xea6, 0x0, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/leveldb.go:388 +0x46c
github.com/prometheus/prometheus/storage/metric/tiered.(*TieredStorage).flushMemory(0xc210057480, 0x8bb2c97000)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:325 +0x3e4
github.com/prometheus/prometheus/storage/metric/tiered.func·036()
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:276 +0x3a
created by github.com/prometheus/prometheus/storage/metric/tiered.(*TieredStorage).Serve
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:278 +0x24d

goroutine 9569 [runnable]:
net/http.(*persistConn).roundTrip(0xc2128cfc00, 0xc212056f40, 0xc2128cfc00, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/transport.go:879 +0x6d6
net/http.(*Transport).RoundTrip(0xc210061e00, 0xc211c8a5b0, 0x1, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/transport.go:187 +0x391
net/http.send(0xc211c8a5b0, 0x7f534e38c2b8, 0xc210061e00, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/net/http/client.go:168 +0x37f
net/http.(*Client).send(0xc2101b1ba0, 0xc211c8a5b0, 0x23, 0xc2120f66c0, 0x428a01)
        /usr/local/go/src/pkg/net/http/client.go:100 +0xd9
net/http.(*Client).doFollowingRedirects(0xc2101b1ba0, 0xc211c8a5b0, 0x9dd198, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/net/http/client.go:294 +0x671
net/http.(*Client).Do(0xc2101b1ba0, 0xc211c8a5b0, 0x6, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/client.go:129 +0x8f
github.com/prometheus/prometheus/retrieval.(*target).scrape(0xc2101a4820, 0x537cada7, 0x7f534e38e3e8, 0xc2101b1870, 0x0, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:223 +0x1ba
github.com/prometheus/prometheus/retrieval.(*target).Scrape(0xc2101a4820, 0xecb0ea4a7, 0x19ff48c4, 0x10e5140, 0x7f534e38e3e8, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:187 +0x5a
github.com/prometheus/prometheus/retrieval.(*TargetPool).runSingle(0xc2101af850, 0xecb0ea4a7, 0x19ff48c4, 0x10e5140, 0x7f534e38e3e8, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:124 +0xa8
github.com/prometheus/prometheus/retrieval.func·003(0x7f534e38e4c8, 0xc2101a4820)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:159 +0x69
created by github.com/prometheus/prometheus/retrieval.(*TargetPool).runIteration
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:161 +0x53d

goroutine 9574 [select]:
net/http.(*Transport).getConn(0xc210061a80, 0xc2120f6c00, 0xc2120f6c00, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/transport.go:424 +0x24d
net/http.(*Transport).RoundTrip(0xc210061a80, 0xc211c8a9c0, 0x1, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/transport.go:182 +0x340
net/http.send(0xc211c8a9c0, 0x7f534e38c2b8, 0xc210061a80, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/net/http/client.go:168 +0x37f
net/http.(*Client).send(0xc2101b1f60, 0xc211c8a9c0, 0x24, 0xc2120f6ba0, 0x428a01)
        /usr/local/go/src/pkg/net/http/client.go:100 +0xd9
net/http.(*Client).doFollowingRedirects(0xc2101b1f60, 0xc211c8a9c0, 0x9dd198, 0x0, 0x0, ...)
        /usr/local/go/src/pkg/net/http/client.go:294 +0x671
net/http.(*Client).Do(0xc2101b1f60, 0xc211c8a9c0, 0x6, 0x0, 0x0)
        /usr/local/go/src/pkg/net/http/client.go:129 +0x8f
github.com/prometheus/prometheus/retrieval.(*target).scrape(0xc2101a49b0, 0x537cada7, 0x7f534e38e3e8, 0xc2101b1870, 0x0, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:223 +0x1ba
github.com/prometheus/prometheus/retrieval.(*target).Scrape(0xc2101a49b0, 0xecb0ea4a7, 0x19ff7125, 0x10e5140, 0x7f534e38e3e8, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:187 +0x5a
github.com/prometheus/prometheus/retrieval.(*TargetPool).runSingle(0xc2101af930, 0xecb0ea4a7, 0x19ff7125, 0x10e5140, 0x7f534e38e3e8, ...)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:124 +0xa8
github.com/prometheus/prometheus/retrieval.func·003(0x7f534e38e4c8, 0xc2101a49b0)
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:159 +0x69
created by github.com/prometheus/prometheus/retrieval.(*TargetPool).runIteration
        /home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:161 +0x53d

The line in the code is:

prometheus/storage/metric/tiered/processor.go

Line 236 in 01f652c

pendingBatch.PutRaw(k, b)

@juliusv juliusv added bug labels May 21, 2014

@matttproud

This comment has been minimized.

Sign in to view
Copy link
Member

matttproud commented May 23, 2014

This drop-in-place snippet greatly rationalizes the current compactor and
could be ported to the deletion one in a few minutes. It is annotated to
help a newbie do so:

https://gist.github.com/matttproud/ab7e6e5e12a5ed2131b4

I expect this would either fix the underlying problem or show where it
originates concretely. It is worth incorporating purely for reasons of
stabilizing the existing implementation to reduce pain for current users
and buy time for a more thoughtful re-einvisioning of the storage stack,
since good decisions are seldom made with fire on the toes.
Am 21.05.2014 08:22 schrieb "juliusv" notifications@github.com:

Brian Brazil is regularly encountering this crash in the compaction
curator:

I0521 13:44:07.436742 15539 tiered.go:313] Flushing samples to disk...
I0521 13:44:07.437431 15539 tiered.go:324] Writing 3534 samples...
I0521 13:44:07.453735 15539 curator.go:336] Curating 01416687721567363777--8-9...
I0521 13:44:07.454042 15539 curator.go:336] Curating 01440054640060257063--7-1...
I0521 13:44:07.454703 15539 curator.go:336] Curating 01451099465613467354--3-9...
I0521 13:44:07.497744 15539 curator.go:336] Curating 01471559311769181855--0-9...
I0521 13:44:07.498027 15539 curator.go:336] Curating 01606155229084260876--7-r...
I0521 13:44:07.498608 15539 curator.go:336] Curating 01662171370440346787--4-9...
I0521 13:44:07.499321 15539 curator.go:336] Curating 01729910661288580458--6-g...
I0521 13:44:07.499994 15539 curator.go:336] Curating 01814014444408188467--4-s...
I0521 13:44:07.500503 15539 curator.go:336] Curating 01837249206601870817--4-t...
I0521 13:44:07.500824 15539 curator.go:336] Curating 01941743549851231537--9-t...
I0521 13:44:07.501065 15539 curator.go:336] Curating 01970935242391824013--5-5...
I0521 13:44:07.501608 15539 curator.go:336] Curating 02144333964455548649--9-s...
I0521 13:44:07.502233 15539 curator.go:336] Curating 02261264228156541201--0-s...
I0521 13:44:07.502780 15539 curator.go:336] Curating 02327171372117302045--3-r...
I0521 13:44:07.503539 15539 curator.go:336] Curating 02341660258630150082--6-k...
I0521 13:44:07.512260 15539 curator.go:336] Curating 02346497262117238201--8-e...
I0521 13:44:07.513011 15539 curator.go:336] Curating 02390177016617661977--2-s...
I0521 13:44:07.513819 15539 curator.go:336] Curating 02401995445572852413--9-s...
I0521 13:44:07.514409 15539 curator.go:336] Curating 02411497020961948501--4-s...
I0521 13:44:07.515142 15539 curator.go:336] Curating 02481919835234196377--5-s...
I0521 13:44:07.515789 15539 curator.go:336] Curating 02597508809804401091--6-5...
I0521 13:44:07.516332 15539 curator.go:336] Curating 02775961821224806906--6-r...
I0521 13:44:07.517077 15539 curator.go:336] Curating 02784830415456707011--7-e...
I0521 13:44:07.517794 15539 curator.go:336] Curating 02833557479258027598--3-t...
I0521 13:44:07.518427 15539 curator.go:336] Curating 02907528775206078261--0-t...
I0521 13:44:07.519081 15539 curator.go:336] Curating 02987574650877550367--7-9...
I0521 13:44:07.519717 15539 curator.go:336] Curating 03040287266226324621--5-9...
I0521 13:44:07.647656 15539 curator.go:336] Curating 03053923783400927315--2-e...
I0521 13:44:07.648030 15539 curator.go:336] Curating 03219724117074421696--7-5...
I0521 13:44:07.648341 15539 curator.go:336] Curating 03233630492714976575--4-5...
I0521 13:44:07.649129 15539 curator.go:336] Curating 03241100289372500203--5-s...
I0521 13:44:07.649711 15539 curator.go:336] Curating 03333222850181259365--4-5...
I0521 13:44:07.658529 15539 curator.go:336] Curating 03359367521074392271--5-g...
I0521 13:44:07.659117 15539 curator.go:336] Curating 03429257346714730163--9-t...
I0521 13:44:07.659464 15539 curator.go:336] Curating 03465632722383485859-_-7-5...
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x38 pc=0x4a5f34]

goroutine 26 [running]:
runtime.panic(0x845de0, 0x10c8b88)
/usr/local/go/src/pkg/runtime/panic.c:266 +0xb6github.com/prometheus/prometheus/storage/metric/tiered.(_CompactionProcessor).Apply(0xc211b65ec0, 0x7f534e395d48, 0xc211b65f00, 0x7f5348031740, 0xc21005ade0, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/processor.go:236 +0x704github.com/prometheus/prometheus/storage/metric/tiered.(_watermarkScanner).Operate(0xc211fba990, 0x88bfa0, 0xc21307f990, 0x847cc0, 0xc213081960, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/curator.go:408 +0xa3bgithub.com/prometheus/prometheus/storage/raw/leveldb.(_LevelDBPersistence).ForEach(0xc21005af60, 0x7f53480317b0, 0xc211fba990, 0x7f53480317e0, 0xc211fba990, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/raw/leveldb/leveldb.go:509 +0x332github.com/prometheus/prometheus/storage/metric/tiered.(_Curator).Run(0xc2134c24c0, 0x45d964b800, 0x537cada7, 0x7f5348031700, 0xc211b65ec0, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/curator.go:203 +0x748
main.(*prometheus).compact(0xc211965000, 0x45d964b800, 0x1f4, 0x0, 0x0)
/home/xxxxxxxx/workspace/prometheus/main.go:140 +0x37b
main.func·004()
/home/xxxxxxxx/workspace/prometheus/main.go:360 +0x163
created by main.main
/home/xxxxxxxx/workspace/prometheus/main.go:367 +0x100a

goroutine 1 [chan receive]:
main.main()
/home/xxxxxxxx/workspace/prometheus/main.go:389 +0x10d3

goroutine 3 [chan receive]:github.com/golang/glog.(*loggingT).flushDaemon(0x10e5360)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/golang/glog/glog.go:839 +0x50
created by github.com/golang/glog.init·1 http://github.com/golang/glog.init%C2%B71
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/golang/glog/glog.go:406 +0x276

goroutine 4 [syscall]:
runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1394

goroutine 7 [syscall]:
os/signal.loop()
/usr/local/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
/usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x31

goroutine 23 [chan receive]:github.com/prometheus/prometheus/notification.func·001() http://github.com/prometheus/prometheus/notification.func%C2%B7001()
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/notification/notification.go:171 +0x48
created by github.com/prometheus/prometheus/notification.(*NotificationHandler).Run
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/notification/notification.go:174 +0xa8

goroutine 22 [select]:github.com/prometheus/prometheus/storage/metric/tiered.(*TieredStorage).Serve(0xc210057480, 0xc211956360)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:271 +0x467
created by main.main
/home/xxxxxxxx/workspace/prometheus/main.go:352 +0xf83

goroutine 20 [select]:github.com/prometheus/prometheus/rules.(*ruleManager).Run(0xc21005acc0)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/rules/manager.go:89 +0x2e6
created by main.main
/home/xxxxxxxx/workspace/prometheus/main.go:285 +0x859

goroutine 16 [semacquire]:
sync.runtime_Semacquire(0xc211b7eb30)
/usr/local/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(_WaitGroup).Wait(0xc21217efe0)
/usr/local/go/src/pkg/sync/waitgroup.go:127 +0x14bgithub.com/prometheus/prometheus/retrieval.(_TargetPool).runIteration(0xc2101af690, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:164 +0x399github.com/prometheus/prometheus/retrieval.(_TargetPool).Run(0xc2101af690, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:61 +0xd0
created by github.com/prometheus/prometheus/retrieval.(_targetManager).TargetPoolForJob
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:73 +0x3f5

goroutine 21 [chan receive]:github.com/prometheus/prometheus/notification.(*NotificationHandler).Run(0xc2100e7120)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/notification/notification.go:177 +0xf5
created by main.main
/home/xxxxxxxx/workspace/prometheus/main.go:288 +0x89f

goroutine 19 [semacquire]:
sync.runtime_Semacquire(0xc211b7eb60)
/usr/local/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(_WaitGroup).Wait(0xc211c2d2c0)
/usr/local/go/src/pkg/sync/waitgroup.go:127 +0x14bgithub.com/prometheus/prometheus/retrieval.(_TargetPool).runIteration(0xc2101af930, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:164 +0x399github.com/prometheus/prometheus/retrieval.(_TargetPool).Run(0xc2101af930, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:61 +0xd0
created by github.com/prometheus/prometheus/retrieval.(_targetManager).TargetPoolForJob
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:73 +0x3f5

goroutine 17 [select]:github.com/prometheus/prometheus/retrieval.(_TargetPool).Run(0xc2101af770, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:59 +0x276
created by github.com/prometheus/prometheus/retrieval.(_targetManager).TargetPoolForJob
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:73 +0x3f5

goroutine 18 [semacquire]:
sync.runtime_Semacquire(0xc211b7eb50)
/usr/local/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(_WaitGroup).Wait(0xc211c2d1a0)
/usr/local/go/src/pkg/sync/waitgroup.go:127 +0x14bgithub.com/prometheus/prometheus/retrieval.(_TargetPool).runIteration(0xc2101af850, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:164 +0x399github.com/prometheus/prometheus/retrieval.(_TargetPool).Run(0xc2101af850, 0x7f534e38e3e8, 0xc2101b1870, 0xdf8475800)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:61 +0xd0
created by github.com/prometheus/prometheus/retrieval.(_targetManager).TargetPoolForJob
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:73 +0x3f5

goroutine 24 [chan receive]:github.com/prometheus/prometheus/storage/metric/tiered.func·035() http://github.com/prometheus/prometheus/storage/metric/tiered.func%C2%B7035()
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:264 +0x48
created by github.com/prometheus/prometheus/storage/metric/tiered.(*TieredStorage).Serve
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:267 +0x18e

goroutine 25 [chan receive]:
main.(*prometheus).interruptHandler(0xc211965000)
/home/xxxxxxxx/workspace/prometheus/main.go:100 +0x107
created by main.main
/home/xxxxxxxx/workspace/prometheus/main.go:355 +0xfba

goroutine 27 [chan receive]:
main.func·005()
/home/xxxxxxxx/workspace/prometheus/main.go:370 +0x50
created by main.main
/home/xxxxxxxx/workspace/prometheus/main.go:379 +0x105a

goroutine 28 [IO wait]:
net.runtime_pollWait(0x7f534e38fbf0, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc2101afed0, 0x72, 0x7f534e38c0e8, 0xb)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc2101afed0, 0xb, 0x7f534e38c0e8)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).accept(0xc2101afe70, 0x9dc1f8, 0x0, 0x7f534e38c0e8, 0xb)
/usr/local/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(_TCPListener).AcceptTCP(0xc210000f90, 0x541abb, 0x7f53485abd88, 0x541abb)
/usr/local/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(_TCPListener).Accept(0xc210000f90, 0x7f534e38fc98, 0xc21212f5b8, 0xc2128cfa80, 0x0)
/usr/local/go/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(_Server).Serve(0xc2101a4aa0, 0x7f534e38ec50, 0xc210000f90, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/server.go:1622 +0x91
net/http.(*Server).ListenAndServe(0xc2101a4aa0, 0xc2101a4aa0, 0x7f534e38ebd0)
/usr/local/go/src/pkg/net/http/server.go:1612 +0xa0
net/http.ListenAndServe(0x7fff924efdcf, 0x5, 0x7f534e38ebd0, 0xc210000488, 0xc211963160, ...)
/usr/local/go/src/pkg/net/http/server.go:1677 +0x6dgithub.com/prometheus/prometheus/web.WebService.ServeForever(0xc210057750, 0xc211956240, 0xc21018ca80, 0xc2101b06a0, 0xc211963060, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/web.go:89 +0x629
main.func·006()
/home/xxxxxxxx/workspace/prometheus/main.go:382 +0x3d
created by main.main
/home/xxxxxxxx/workspace/prometheus/main.go:386 +0x10aa

goroutine 9602 [IO wait]:
net.runtime_pollWait(0x7f534e38f8a8, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc21265ba70, 0x72, 0x7f534e38c0e8, 0xb)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc21265ba70, 0xb, 0x7f534e38c0e8)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).Read(0xc21265ba10, 0xc211e4a000, 0x1000, 0x1000, 0x0, ...)
/usr/local/go/src/pkg/net/fd_unix.go:204 +0x2a0
net.(_conn).Read(0xc21212f5b8, 0xc211e4a000, 0x1000, 0x1000, 0x7f534e383000, ...)
/usr/local/go/src/pkg/net/net.go:122 +0xc5
net/http.(_liveSwitchReader).Read(0xc2128cfaa8, 0xc211e4a000, 0x1000, 0x1000, 0x3, ...)
/usr/local/go/src/pkg/net/http/server.go:204 +0xa5
io.(_LimitedReader).Read(0xc21212d8a0, 0xc211e4a000, 0x1000, 0x1000, 0xc211f547c8, ...)
/usr/local/go/src/pkg/io/io.go:398 +0xbb
bufio.(_Reader).fill(0xc211cac960)
/usr/local/go/src/pkg/bufio/bufio.go:91 +0x110
bufio.(_Reader).ReadSlice(0xc211cac960, 0xc20000000a, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/pkg/bufio/bufio.go:274 +0x204
bufio.(_Reader).ReadLine(0xc211cac960, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/pkg/bufio/bufio.go:305 +0x63
net/textproto.(_Reader).readLineSlice(0xc211c17690, 0x7f534e383000, 0x766440, 0x7f5348573ce8, 0x428932, ...)
/usr/local/go/src/pkg/net/textproto/reader.go:55 +0x61
net/textproto.(_Reader).ReadLine(0xc211c17690, 0xc211f55b60, 0x0, 0xc211e4b000, 0x0)
/usr/local/go/src/pkg/net/textproto/reader.go:36 +0x27
net/http.ReadRequest(0xc211cac960, 0xc211f55b60, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/request.go:526 +0x88
net/http.(_conn).readRequest(0xc2128cfa80, 0x0, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/server.go:575 +0x1bb
net/http.(_conn).serve(0xc2128cfa80)
/usr/local/go/src/pkg/net/http/server.go:1123 +0x3b4
created by net/http.(_Server).Serve
/usr/local/go/src/pkg/net/http/server.go:1644 +0x28b

goroutine 9846 [runnable]:github.com/prometheus/prometheus/storage/metric/tiered.findUnindexed(0x7f534e38d818, 0xc210000b98, 0xc211eb2690, 0x7f5348591f38, 0x7f5348591ee0, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/index.go:566 +0x158github.com/prometheus/prometheus/storage/metric/tiered.(_TotalIndexer).IndexMetrics(0xc21018cd80, 0xc211eb2690, 0x86efcf16286c77f, 0x903870)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/index.go:663 +0x46github.com/prometheus/prometheus/storage/metric/tiered.func·015(0xc2120bba50) http://github.com/prometheus/prometheus/storage/metric/tiered.func%C2%B7015(0xc2120bba50)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/leveldb.go:334 +0x157
created by github.com/prometheus/prometheus/storage/metric/tiered.(_LevelDBPersistence).AppendSamples
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/leveldb.go:335 +0x1c0

goroutine 9603 [runnable]:
net.runtime_pollWait(0x7f534e38f758, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc211d8aca0, 0x72, 0x7f534e38c0e8, 0xb)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc211d8aca0, 0xb, 0x7f534e38c0e8)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).Read(0xc211d8ac40, 0xc212130000, 0x1000, 0x1000, 0x0, ...)
/usr/local/go/src/pkg/net/fd_unix.go:204 +0x2a0
net.(_conn).Read(0xc21212f5f8, 0xc212130000, 0x1000, 0x1000, 0x41b61d, ...)
/usr/local/go/src/pkg/net/net.go:122 +0xc5
bufio.(_Reader).fill(0xc211bc9f00)
/usr/local/go/src/pkg/bufio/bufio.go:91 +0x110
bufio.(_Reader).Peek(0xc211bc9f00, 0x1, 0x2, 0x2, 0x0, ...)
/usr/local/go/src/pkg/bufio/bufio.go:119 +0xcb
net/http.(_persistConn).readLoop(0xc2128cfb00)
/usr/local/go/src/pkg/net/http/transport.go:687 +0xb7
created by net/http.(_Transport).dialConn
/usr/local/go/src/pkg/net/http/transport.go:528 +0x607

goroutine 9604 [select]:
net/http.(_persistConn).writeLoop(0xc2128cfb00)
/usr/local/go/src/pkg/net/http/transport.go:791 +0x271
created by net/http.(_Transport).dialConn
/usr/local/go/src/pkg/net/http/transport.go:529 +0x61e

goroutine 9608 [select]:
net/http.(_persistConn).writeLoop(0xc2128cfc00)
/usr/local/go/src/pkg/net/http/transport.go:791 +0x271
created by net/http.(_Transport).dialConn
/usr/local/go/src/pkg/net/http/transport.go:529 +0x61e

goroutine 9586 [runnable]:
net.runtime_pollWait(0x7f534e38f6b0, 0x77, 0x0)
/usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc211a95df0, 0x77, 0x7f534e38c0e8, 0x73)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitWrite(0xc211a95df0, 0x73, 0x7f534e38c0e8)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:90 +0x30
net.(_netFD).connect(0xc211a95d90, 0x0, 0x0, 0x7f534e393dd0, 0xc211e3e660, ...)
/usr/local/go/src/pkg/net/fd_unix.go:86 +0x166
net.(_netFD).dial(0xc211a95d90, 0x7f534e38ec88, 0x0, 0x7f534e38ec88, 0xc211e38f00, ...)
/usr/local/go/src/pkg/net/sock_posix.go:121 +0x2fd
net.socket(0x8eeb00, 0x3, 0x2, 0x1, 0x0, ...)
/usr/local/go/src/pkg/net/sock_posix.go:91 +0x40b
net.internetSocket(0x8eeb00, 0x3, 0x7f534e38ec88, 0x0, 0x7f534e38ec88, ...)
/usr/local/go/src/pkg/net/ipsock_posix.go:136 +0x161
net.dialTCP(0x8eeb00, 0x3, 0x0, 0xc211e38f00, 0xecb0ea4e3, ...)
/usr/local/go/src/pkg/net/tcpsock_posix.go:155 +0xef
net.dialSingle(0x8eeb00, 0x3, 0xc2100b6bb5, 0x15, 0x0, ...)
/usr/local/go/src/pkg/net/dial.go:225 +0x3d8
net.func·015(0xecb0ea4e3, 0x20985ddc, 0x10e5140, 0xdf8475800, 0xecb0ea4e3, ...)
/usr/local/go/src/pkg/net/dial.go:158 +0xde
net.dial(0x8eeb00, 0x3, 0x7f534e38ec20, 0xc211e38f00, 0x7f534ac41c68, ...)
/usr/local/go/src/pkg/net/fd_unix.go:40 +0x45
net.(_Dialer).Dial(0xc2133ff600, 0x8eeb00, 0x3, 0xc2100b6bb5, 0x15, ...)
/usr/local/go/src/pkg/net/dial.go:165 +0x3e0
net.DialTimeout(0x8eeb00, 0x3, 0xc2100b6bb5, 0x15, 0xdf8475800, ...)
/usr/local/go/src/pkg/net/dial.go:145 +0xaagithub.com/prometheus/prometheus/utility.func·001(0x8eeb00 http://github.com/prometheus/prometheus/utility.func%C2%B7001(0x8eeb00, 0x3, 0xc2100b6bb5, 0x15, 0x0, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/utility/deadline_client.go:33 +0xb8
net/http.(_Transport).dial(0xc210061a80, 0x8eeb00, 0x3, 0xc2100b6bb5, 0x15, ...)
/usr/local/go/src/pkg/net/http/transport.go:399 +0x89
net/http.(_Transport).dialConn(0xc210061a80, 0xc2120f6c00, 0x41fc60, 0xc210057480, 0x7f534e395b30)
/usr/local/go/src/pkg/net/http/transport.go:444 +0x6e
net/http.func·014()
/usr/local/go/src/pkg/net/http/transport.go:419 +0x3e
created by net/http.(_Transport).getConn
/usr/local/go/src/pkg/net/http/transport.go:421 +0x11a

goroutine 9607 [chan receive]:
net/http.(_persistConn).readLoop(0xc2128cfc00)
/usr/local/go/src/pkg/net/http/transport.go:778 +0x68f
created by net/http.(_Transport).dialConn
/usr/local/go/src/pkg/net/http/transport.go:528 +0x607

goroutine 9567 [select]:
net/http.(_persistConn).roundTrip(0xc2128cfb00, 0xc212056f00, 0xc2128cfb00, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/transport.go:879 +0x6d6
net/http.(_Transport).RoundTrip(0xc210061d00, 0xc211c8a410, 0x1, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/transport.go:187 +0x391
net/http.send(0xc211c8a410, 0x7f534e38c2b8, 0xc210061d00, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/client.go:168 +0x37f
net/http.(_Client).send(0xc2101b1960, 0xc211c8a410, 0x1d, 0xc2120f65a0, 0x428a01)
/usr/local/go/src/pkg/net/http/client.go:100 +0xd9
net/http.(_Client).doFollowingRedirects(0xc2101b1960, 0xc211c8a410, 0x9dd198, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/client.go:294 +0x671
net/http.(_Client).Do(0xc2101b1960, 0xc211c8a410, 0x6, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/client.go:129 +0x8fgithub.com/prometheus/prometheus/retrieval.(_target).scrape(0xc2101a4780, 0x537cada7, 0x7f534e38e3e8, 0xc2101b1870, 0x0, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:223 +0x1bagithub.com/prometheus/prometheus/retrieval.(_target).Scrape(0xc2101a4780, 0xecb0ea4a7, 0x19ff1aa7, 0x10e5140, 0x7f534e38e3e8, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:187 +0x5agithub.com/prometheus/prometheus/retrieval.(_TargetPool).runSingle(0xc2101af690, 0xecb0ea4a7, 0x19ff1aa7, 0x10e5140, 0x7f534e38e3e8, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:124 +0xa8github.com/prometheus/prometheus/retrieval.func·003(0x7f534e38e4c8 http://github.com/prometheus/prometheus/retrieval.func%C2%B7003(0x7f534e38e4c8, 0xc2101a4780)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:159 +0x69
created by github.com/prometheus/prometheus/retrieval.(*TargetPool).runIteration
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:161 +0x53d

goroutine 9573 [runnable]:
net/http.(_Transport).getConn(0xc210061a00, 0xc2120f6b70, 0xc2120f6b70, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/transport.go:424 +0x24d
net/http.(_Transport).RoundTrip(0xc210061a00, 0xc211c8a8f0, 0x1, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/transport.go:182 +0x340
net/http.send(0xc211c8a8f0, 0x7f534e38c2b8, 0xc210061a00, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/client.go:168 +0x37f
net/http.(_Client).send(0xc2101b1ed0, 0xc211c8a8f0, 0x24, 0xc2120f6b10, 0x428a01)
/usr/local/go/src/pkg/net/http/client.go:100 +0xd9
net/http.(_Client).doFollowingRedirects(0xc2101b1ed0, 0xc211c8a8f0, 0x9dd198, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/client.go:294 +0x671
net/http.(_Client).Do(0xc2101b1ed0, 0xc211c8a8f0, 0x6, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/client.go:129 +0x8fgithub.com/prometheus/prometheus/retrieval.(_target).scrape(0xc2101a4960, 0x537cada7, 0x7f534e38e3e8, 0xc2101b1870, 0x0, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:223 +0x1bagithub.com/prometheus/prometheus/retrieval.(_target).Scrape(0xc2101a4960, 0xecb0ea4a7, 0x19ff6b38, 0x10e5140, 0x7f534e38e3e8, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:187 +0x5agithub.com/prometheus/prometheus/retrieval.(_TargetPool).runSingle(0xc2101af930, 0xecb0ea4a7, 0x19ff6b38, 0x10e5140, 0x7f534e38e3e8, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:124 +0xa8github.com/prometheus/prometheus/retrieval.func·003(0x7f534e38e4c8 http://github.com/prometheus/prometheus/retrieval.func%C2%B7003(0x7f534e38e4c8, 0xc2101a4960)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:159 +0x69
created by github.com/prometheus/prometheus/retrieval.(*TargetPool).runIteration
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:161 +0x53d

goroutine 9578 [chan receive]:github.com/prometheus/prometheus/storage/metric/tiered.(_LevelDBPersistence).AppendSamples(0xc210061b00, 0xc2134cf000, 0xdce, 0xea6, 0x0, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/leveldb.go:388 +0x46cgithub.com/prometheus/prometheus/storage/metric/tiered.(_TieredStorage).flushMemory(0xc210057480, 0x8bb2c97000)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:325 +0x3e4github.com/prometheus/prometheus/storage/metric/tiered.func·036() http://github.com/prometheus/prometheus/storage/metric/tiered.func%C2%B7036()
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:276 +0x3a
created by github.com/prometheus/prometheus/storage/metric/tiered.(*TieredStorage).Serve
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/metric/tiered/tiered.go:278 +0x24d

goroutine 9569 [runnable]:
net/http.(_persistConn).roundTrip(0xc2128cfc00, 0xc212056f40, 0xc2128cfc00, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/transport.go:879 +0x6d6
net/http.(_Transport).RoundTrip(0xc210061e00, 0xc211c8a5b0, 0x1, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/transport.go:187 +0x391
net/http.send(0xc211c8a5b0, 0x7f534e38c2b8, 0xc210061e00, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/client.go:168 +0x37f
net/http.(_Client).send(0xc2101b1ba0, 0xc211c8a5b0, 0x23, 0xc2120f66c0, 0x428a01)
/usr/local/go/src/pkg/net/http/client.go:100 +0xd9
net/http.(_Client).doFollowingRedirects(0xc2101b1ba0, 0xc211c8a5b0, 0x9dd198, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/client.go:294 +0x671
net/http.(_Client).Do(0xc2101b1ba0, 0xc211c8a5b0, 0x6, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/client.go:129 +0x8fgithub.com/prometheus/prometheus/retrieval.(_target).scrape(0xc2101a4820, 0x537cada7, 0x7f534e38e3e8, 0xc2101b1870, 0x0, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:223 +0x1bagithub.com/prometheus/prometheus/retrieval.(_target).Scrape(0xc2101a4820, 0xecb0ea4a7, 0x19ff48c4, 0x10e5140, 0x7f534e38e3e8, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:187 +0x5agithub.com/prometheus/prometheus/retrieval.(_TargetPool).runSingle(0xc2101af850, 0xecb0ea4a7, 0x19ff48c4, 0x10e5140, 0x7f534e38e3e8, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:124 +0xa8github.com/prometheus/prometheus/retrieval.func·003(0x7f534e38e4c8 http://github.com/prometheus/prometheus/retrieval.func%C2%B7003(0x7f534e38e4c8, 0xc2101a4820)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:159 +0x69
created by github.com/prometheus/prometheus/retrieval.(*TargetPool).runIteration
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:161 +0x53d

goroutine 9574 [select]:
net/http.(_Transport).getConn(0xc210061a80, 0xc2120f6c00, 0xc2120f6c00, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/transport.go:424 +0x24d
net/http.(_Transport).RoundTrip(0xc210061a80, 0xc211c8a9c0, 0x1, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/transport.go:182 +0x340
net/http.send(0xc211c8a9c0, 0x7f534e38c2b8, 0xc210061a80, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/client.go:168 +0x37f
net/http.(_Client).send(0xc2101b1f60, 0xc211c8a9c0, 0x24, 0xc2120f6ba0, 0x428a01)
/usr/local/go/src/pkg/net/http/client.go:100 +0xd9
net/http.(_Client).doFollowingRedirects(0xc2101b1f60, 0xc211c8a9c0, 0x9dd198, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/client.go:294 +0x671
net/http.(_Client).Do(0xc2101b1f60, 0xc211c8a9c0, 0x6, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/client.go:129 +0x8fgithub.com/prometheus/prometheus/retrieval.(_target).scrape(0xc2101a49b0, 0x537cada7, 0x7f534e38e3e8, 0xc2101b1870, 0x0, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:223 +0x1bagithub.com/prometheus/prometheus/retrieval.(_target).Scrape(0xc2101a49b0, 0xecb0ea4a7, 0x19ff7125, 0x10e5140, 0x7f534e38e3e8, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:187 +0x5agithub.com/prometheus/prometheus/retrieval.(_TargetPool).runSingle(0xc2101af930, 0xecb0ea4a7, 0x19ff7125, 0x10e5140, 0x7f534e38e3e8, ...)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:124 +0xa8github.com/prometheus/prometheus/retrieval.func·003(0x7f534e38e4c8 http://github.com/prometheus/prometheus/retrieval.func%C2%B7003(0x7f534e38e4c8, 0xc2101a49b0)
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:159 +0x69
created by github.com/prometheus/prometheus/retrieval.(*TargetPool).runIteration
/home/xxxxxxxx/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetpool.go:161 +0x53d

The line in the code is:

prometheus/storage/metric/tiered/processor.go

Line 236 in 01f652c

pendingBatch.PutRaw(k, b)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/392
.
@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member Author

juliusv commented Dec 10, 2014

The old storage is gone. Closing this.

@juliusv juliusv closed this Dec 10, 2014

simonpasquier pushed a commit to simonpasquier/prometheus that referenced this issue Oct 12, 2017

@grobie
Merge pull request prometheus#392 from syscrusher/patch-1 
Remove third-party blackbox exporter
8ef9d80
@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 24, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.