Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unclean shutdown on CTRL-C #1324

Closed
RichiH opened this Issue Jan 18, 2016 · 5 comments

Comments

Assignees

@fabxc fabxc

Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@RichiH @beorn7 @fabxc
@RichiH
Copy link
Member

RichiH commented Jan 18, 2016

After about a week of running prometheus, I stopped it with CTRL-C and got:

^CWARN[336720] Received SIGTERM, exiting gracefully...       source=main.go:178
INFO[336720] See you next time!                            source=main.go:185
INFO[336720] Stopping target manager...                    source=targetmanager.go:203
WARN[336724] Error while evaluating rule "job_service:rpc_durations_microseconds_count:avg_rate5m{} = avg(rate(rpc_durations_microseconds_count[5m])) BY (job, service)\n": rule result is not a vector or scalar  source=manager.go:220
WARN[336724] Error while evaluating rule "ALERT InstanceDown\n\tIF up == 0\n\tFOR 5m\n\tWITH {severity=\"page\"}\n\tANNOTATIONS {description=\"{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.\", summary=\"Instance {{ $labels.instance }} down\"}": query was canceled in query execution  source=manager.go:220

WARN[336740] Error while evaluating rule "job_service:rpc_durations_microseconds_count:avg_rate5m{} = avg(rate(rpc_durations_microseconds_count[5m])) BY (job, service)\n": rule result is not a vector or scalar  source=manager.go:220
WARN[336740] Error while evaluating rule "ALERT InstanceDown\n\tIF up == 0\n\tFOR 5m\n\tWITH {severity=\"page\"}\n\tANNOTATIONS {description=\"{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.\", summary=\"Instance {{ $labels.instance }} down\"}": query was canceled in query execution  source=manager.go:220
WARN[336755] Error while evaluating rule "ALERT InstanceDown\n\tIF up == 0\n\tFOR 5m\n\tWITH {severity=\"page\"}\n\tANNOTATIONS {description=\"{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.\", summary=\"Instance {{ $labels.instance }} down\"}": query was canceled in query queue  source=manager.go:220
WARN[336755] Error while evaluating rule "job_service:rpc_durations_microseconds_count:avg_rate5m{} = avg(rate(rpc_durations_microseconds_count[5m])) BY (job, service)\n": rule result is not a vector or scalar  source=manager.go:220
WARN[336770] Error while evaluating rule "ALERT InstanceDown\n\tIF up == 0\n\tFOR 5m\n\tWITH {severity=\"page\"}\n\tANNOTATIONS {description=\"{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.\", summary=\"Instance {{ $labels.instance }} down\"}": query was canceled in query execution  source=manager.go:220
WARN[336770] Error while evaluating rule "job_service:rpc_durations_microseconds_count:avg_rate5m{} = avg(rate(rpc_durations_microseconds_count[5m])) BY (job, service)\n": rule result is not a vector or scalar  source=manager.go:220
WARN[336785] Error while evaluating rule "job_service:rpc_durations_microseconds_count:avg_rate5m{} = avg(rate(rpc_durations_microseconds_count[5m])) BY (job, service)\n": rule result is not a vector or scalar  source=manager.go:220
WARN[336785] Error while evaluating rule "ALERT InstanceDown\n\tIF up == 0\n\tFOR 5m\n\tWITH {severity=\"page\"}\n\tANNOTATIONS {description=\"{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.\", summary=\"Instance {{ $labels.instance }} down\"}": query was canceled in query execution  source=manager.go:220
WARN[336800] Error while evaluating rule "ALERT InstanceDown\n\tIF up == 0\n\tFOR 5m\n\tWITH {severity=\"page\"}\n\tANNOTATIONS {description=\"{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.\", summary=\"Instance {{ $labels.instance }} down\"}": query was canceled in query queue  source=manager.go:220
WARN[336800] Error while evaluating rule "job_service:rpc_durations_microseconds_count:avg_rate5m{} = avg(rate(rpc_durations_microseconds_count[5m])) BY (job, service)\n": rule result is not a vector or scalar  source=manager.go:220
INFO[336800] Target manager stopped.                       source=targetmanager.go:216
INFO[336800] Stopping notification handler...              source=notification.go:274
INFO[336800] Stopping rule manager...                      source=manager.go:357
panic: send on closed channel

goroutine 63411387 [running]:
github.com/prometheus/prometheus/notification.(*Handler).setMore(0xc8200e8240)
        /home/gocode/src/github.com/prometheus/prometheus/notification/notification.go:235 +0x3d
github.com/prometheus/prometheus/notification.(*Handler).Send(0xc8200e8240, 0xc84ac34000, 0x24c, 0x400)
        /home/gocode/src/github.com/prometheus/prometheus/notification/notification.go:227 +0x524
github.com/prometheus/prometheus/rules.(*Group).sendAlerts(0xc8c55f65f0, 0xc8c55f66e0, 0x152548dc551, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:317 +0x15c
github.com/prometheus/prometheus/rules.(*Group).eval.func1(0xc87e6e2a40, 0x152548dc551, 0xc8c55f65f0, 0x7fb600ffc708, 0xc8c55f66e0)
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:227 +0x34e
created by github.com/prometheus/prometheus/rules.(*Group).eval
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:243 +0x128

goroutine 1 [runnable]:
github.com/prometheus/prometheus/rules.(*Manager).Stop(0xc820139ef0)
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:360 +0x180
main.Main(0x0)
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:186 +0x1700
main.main()
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:46 +0x18

goroutine 17 [syscall, 5613 minutes, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 34 [syscall]:
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 106 [select]:
main.Main.func2(0xc820077200, 0xc8200771a0, 0xc8200a8c00, 0xc820460ff0, 0x5, 0x5)
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:127 +0x134
created by main.Main
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:133 +0xe3f

goroutine 133 [select, 5613 minutes, locked to thread]:
runtime.gopark(0xfb5f68, 0xc820519728, 0xde9e18, 0x6, 0x434318, 0x2)
        /usr/local/go/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc820519728, 0x0, 0x18)
        /usr/local/go/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc820519728)
        /usr/local/go/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 107 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8200d8000)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 21 [select, 372 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc820154000)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 22 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc820154000)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 23 [select, 372 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc820154000)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 24 [select, 374 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc820154000)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 25 [select, 5613 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc820154000)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 26 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc82019a1c0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 108 [select, 456 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc8200969a0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 109 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc8200969a0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 110 [select, 456 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc8200969a0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB

        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 111 [select, 466 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc8200969a0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 112 [select, 5613 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc8200969a0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 113 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc82019a540)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 118 [select, 3 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc820096c60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 119 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc820096c60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 120 [select, 3 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc820096c60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 121 [select, 3 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc820096c60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 122 [select, 3 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc820096c60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 123 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc82019a620)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 146 [select, 13 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc820096dc0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 147 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc820096dc0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 148 [select, 13 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc820096dc0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 149 [select, 13 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc820096dc0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 150 [select, 3 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc820096dc0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 151 [select]:
github.com/prometheus/prometheus/storage/local.(*persistence).processIndexingQueue(0xc82027f700)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/persistence.go:1341 +0xcaf
github.com/prometheus/prometheus/storage/local.(*persistence).run(0xc82027f700)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/persistence.go:280 +0x21
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:252 +0x137

goroutine 152 [select]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).handleEvictList(0xc8200ec3c0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:658 +0x52d
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:275 +0x499

goroutine 153 [select]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loop(0xc8200ec3c0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:875 +0x59b
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:276 +0x4be

goroutine 161 [IO wait]:
net.runtime_pollWait(0x7fb6002949d8, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8414ba140, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8414ba140, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8414ba0e0, 0x0, 0x7fb600294ad0, 0xc8b3b12460)
        /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc841520000, 0x459060, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc841520000, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:2135 +0x41
net/http.(*Server).Serve(0xc8414f6000, 0x7fb600294a98, 0xc841520000, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1887 +0xb3
net/http.(*Server).ListenAndServe(0xc8414f6000, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1877 +0x136
net/http.ListenAndServe(0xde1028, 0x5, 0x7fb600293a00, 0xc8201de140, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1967 +0x8f
github.com/prometheus/prometheus/web.(*Handler).Run(0xc8200a8c00)
        /home/gocode/src/github.com/prometheus/prometheus/web/web.go:240 +0x144
created by main.Main
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:169 +0x1338

goroutine 27 [runnable]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc8200ec3c0, 0x95c9d, 0x3fe98e9a98e9a98e, 0x1)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:762 +0x19a
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughMemoryFingerprints.func1(0xc84146e060, 0xc8200ec3c0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:802 +0x252
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughMemoryFingerprints
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:812 +0x5d

goroutine 28 [runnable]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc8200ec3c0, 0x16444, 0x3ff0000000000000, 0x16401)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:762 +0x19a
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughArchivedFingerprints.func1(0xc84146e0c0, 0xc8200ec3c0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:846 +0x406
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughArchivedFingerprints
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:855 +0x5d

goroutine 163 [chan send, 5602 minutes]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func2(0xc84142b1a0, 0xc84142b140, 0xc820190700, 0xc84142b020)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:143 +0x1bb
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:148 +0x4f2

goroutine 185 [chan send, 5602 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8201ace80, 0xc84142b680, 0xc84142b020)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 187 [chan send, 5602 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8201acd80, 0xc84142b500, 0xc84142b020)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 188 [chan send, 5602 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8201acc80, 0xc84142b440, 0xc84142b020)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 193 [chan send, 5602 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8201acd00, 0xc84142b080, 0xc84142b020)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 60989356 [chan send, 7 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8b4462c80, 0xc853132600, 0xc853132360)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 173118 [chan send, 5591 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8412ec3c0, 0xc855fe3920, 0xc855fe3740)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 173117 [chan send, 5591 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8412ec800, 0xc855fe39e0, 0xc855fe3740)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 60127063 [chan send]:
github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter.func1(0xc821c89fe0, 0xc84498aba0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/series.go:129 +0xdc
created by github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/series.go:134 +0x5d

goroutine 173041 [chan send, 5591 minutes]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func2(0xc855fe3800, 0xc855fe37a0, 0xc82ce8e700, 0xc855fe3740)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:143 +0x1bb
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:148 +0x4f2

goroutine 173111 [chan send, 5591 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8412f4600, 0xc855fe3e60, 0xc855fe3740)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 711171 [chan send, 65 minutes]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func2(0xc8221af740, 0xc8221af6e0, 0xc82ce8fc00, 0xc8221af680)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:143 +0x1bb
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:148 +0x4f2

goroutine 60989353 [chan send, 7 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8b4462b80, 0xc853132840, 0xc853132360)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 711264 [chan send, 65 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8258125c0, 0xc8221af860, 0xc8221af680)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 60989355 [chan send, 7 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8b4462a80, 0xc8531326c0, 0xc853132360)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 63381319 [chan send]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8d4624740, 0xc8cd723380, 0xc8cd723260)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 711257 [chan send, 65 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc825804a40, 0xc8221afda0, 0xc8221af680)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 711178 [chan send, 65 minutes]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func2(0xc8221afc80, 0xc8221afc20, 0xc82aee01c0, 0xc8221af680)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:143 +0x1bb
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:148 +0x4f2

goroutine 60988841 [chan send, 7 minutes]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func2(0xc853132420, 0xc8531323c0, 0xc8b0b621c0, 0xc853132360)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:143 +0x1bb
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:148 +0x4f2

goroutine 63381217 [chan send]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func2(0xc8cd7236e0, 0xc8cd723680, 0xc85d9e4e00, 0xc8cd723260)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:143 +0x1bb
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:148 +0x4f2

goroutine 62173719 [chan send, 38 minutes]:
github.com/prometheus/prometheus/promql.lexKeywordOrIdentifier(0xc85528f140, 0xfb4768)
        /home/gocode/src/github.com/prometheus/prometheus/promql/lex.go:811 +0x31a
github.com/prometheus/prometheus/promql.(*lexer).run(0xc85528f140)
        /home/gocode/src/github.com/prometheus/prometheus/promql/lex.go:401 +0x52
created by github.com/prometheus/prometheus/promql.lex
        /home/gocode/src/github.com/prometheus/prometheus/promql/lex.go:394 +0xc0

goroutine 63381313 [chan send]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8450739c0, 0xc8cd723800, 0xc8cd723260)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 63256738 [chan send]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func2(0xc8ca6ae0c0, 0xc8ca6ae060, 0xc85f42e700, 0xc8f0c89c20)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:143 +0x1bb
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:148 +0x4f2

goroutine 63256759 [chan send]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc85397f980, 0xc8f0c89e00, 0xc8f0c89c20)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 63256760 [chan send]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc85397fdc0, 0xc8f0c89d40, 0xc8f0c89c20)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 63123389 [IO wait]:
net.runtime_pollWait(0x7fb5d40d8110, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8c7ca1d40, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8c7ca1d40, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8c7ca1ce0, 0xc869ef4000, 0x1000, 0x1000, 0x0, 0x7fb604ec4050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8f1b14670, 0xc869ef4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.(*liveSwitchReader).Read(0xc848049178, 0xc869ef4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:219 +0xa4
io.(*LimitedReader).Read(0xc8caa70d60, 0xc869ef4000, 0x1000, 0x1000, 0x11, 0x0, 0x0)
        /usr/local/go/src/io/io.go:427 +0xbd
bufio.(*Reader).fill(0xc8eb74b800)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadSlice(0xc8eb74b800, 0xc81a3c270a, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:328 +0x21a
bufio.(*Reader).ReadLine(0xc8eb74b800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:357 +0x53
net/textproto.(*Reader).readLineSlice(0xc8f5638960, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/textproto/reader.go:55 +0x81
net/textproto.(*Reader).ReadLine(0xc8f5638960, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/textproto/reader.go:36 +0x40
net/http.ReadRequest(0xc8eb74b800, 0xc8d87b08c0, 0x0, 0x0)
        /usr/local/go/src/net/http/request.go:653 +0xb6
net/http.(*conn).readRequest(0xc848049130, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:633 +0x32f
net/http.(*conn).serve(0xc848049130)
        /usr/local/go/src/net/http/server.go:1319 +0x727
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:1910 +0x3f6

goroutine 62680103 [IO wait, 23 minutes]:
net.runtime_pollWait(0x7fb5d40931d8, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc869359e20, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc869359e20, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc869359dc0, 0xc8cf8bf000, 0x1000, 0x1000, 0x0, 0x7fb604ec4050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8ae78ec10, 0xc8cf8bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.(*liveSwitchReader).Read(0xc92dc63b18, 0xc8cf8bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:219 +0xa4
io.(*LimitedReader).Read(0xc870d98b80, 0xc8cf8bf000, 0x1000, 0x1000, 0x11, 0x0, 0x0)
        /usr/local/go/src/io/io.go:427 +0xbd
bufio.(*Reader).fill(0xc8e41d98c0)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadSlice(0xc8e41d98c0, 0xc81fbec70a, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:328 +0x21a
bufio.(*Reader).ReadLine(0xc8e41d98c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:357 +0x53
net/textproto.(*Reader).readLineSlice(0xc8dd104930, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/textproto/reader.go:55 +0x81
net/textproto.(*Reader).ReadLine(0xc8dd104930, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/textproto/reader.go:36 +0x40
net/http.ReadRequest(0xc8e41d98c0, 0xc828270700, 0x0, 0x0)
        /usr/local/go/src/net/http/request.go:653 +0xb6
net/http.(*conn).readRequest(0xc92dc63ad0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:633 +0x32f
net/http.(*conn).serve(0xc92dc63ad0)
        /usr/local/go/src/net/http/server.go:1319 +0x727
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:1910 +0x3f6

goroutine 62685058 [IO wait, 23 minutes]:
net.runtime_pollWait(0x7fb5d41f4710, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8e20ba0d0, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8e20ba0d0, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8e20ba070, 0xc8d88b2000, 0x1000, 0x1000, 0x0, 0x7fb604ec4050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8e8406028, 0xc8d88b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.(*liveSwitchReader).Read(0xc8b728d598, 0xc8d88b2000, 0x1000, 0x1000, 0xc84d044760, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:219 +0xa4
io.(*LimitedReader).Read(0xc8dc9cc0a0, 0xc8d88b2000, 0x1000, 0x1000, 0xe00000000, 0x0, 0x0)
        /usr/local/go/src/io/io.go:427 +0xbd
bufio.(*Reader).fill(0xc8a8260000)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadSlice(0xc8a8260000, 0xc81c41fe0a, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:328 +0x21a
bufio.(*Reader).ReadLine(0xc8a8260000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:357 +0x53
net/textproto.(*Reader).readLineSlice(0xc88497dda0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/textproto/reader.go:55 +0x81
net/textproto.(*Reader).ReadLine(0xc88497dda0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/textproto/reader.go:36 +0x40
net/http.ReadRequest(0xc8a8260000, 0xc897c028c0, 0x0, 0x0)
        /usr/local/go/src/net/http/request.go:653 +0xb6
net/http.(*conn).readRequest(0xc8b728d550, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:633 +0x32f
net/http.(*conn).serve(0xc8b728d550)
        /usr/local/go/src/net/http/server.go:1319 +0x727
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:1910 +0x3f6

goroutine 63381318 [chan send]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8d4624780, 0xc8cd723440, 0xc8cd723260)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 63256754 [chan send]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc85397fac0, 0xc8ca6ae1e0, 0xc8f0c89c20)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:391 +0x2a2
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

Exit code was 2. If you need any more info, please let me know.
@RichiH

This comment has been minimized.

Sign in to view
Copy link
Member Author

RichiH commented Jan 18, 2016

FWIW, after restarting, I am getting a ton of "Recovered metric" messages.

@beorn7 beorn7 added the bug label Jan 18, 2016

@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jan 18, 2016

Which release (or at which git commit) is that?
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jan 18, 2016

Relatively sure I know where to look here. But as @beorn7 said, commit ID is always helpful.
@RichiH

This comment has been minimized.

Sign in to view
Copy link
Member Author

RichiH commented Jan 18, 2016

37d80c4

@beorn7 beorn7 assigned fabxc Jan 18, 2016

@fabxc fabxc referenced this issue Jan 18, 2016

Merged

Fix rule manager shutdown issues #1325

@fabxc fabxc closed this in #1325 Jan 20, 2016

@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 24, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.