Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash on startup due to unguarded concurrent map writes in discovery.go #2444

Closed
gottwald opened this Issue Feb 23, 2017 · 1 comment

Comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@gottwald
@gottwald
Copy link
Contributor

gottwald commented Feb 23, 2017

What did you do?
Start prometheus

What did you expect to see?
A normal start

What did you see instead? Under which circumstances?
A crash

Environment
GCE instance with prometheus running inside of a container

  • System information:
    Linux 4.4.0-63-generic x86_64

  • Prometheus version:
    prometheus, version 1.5.2 (branch: master, revision: bd1182d)
    build user: root@1a01c5f68840
    build date: 20170210-16:23:28
    go version: go1.7.5

  • Prometheus configuration file:
    Sorry. Not sure if I can disclose it here. But won't be needed for debugging.

  • Logs:

time="2017-02-23T11:15:23Z" level=info msg="Starting prometheus (version=1.5.2, branch=master, revision=bd1182d29f462c39544f94cc822830e1c64cf55b)" source="main.go:75"
time="2017-02-23T11:15:23Z" level=info msg="Build context (go=go1.7.5, user=root@1a01c5f68840, date=20170210-16:23:28)" source="main.go:76"
time="2017-02-23T11:15:23Z" level=info msg="Loading configuration file /etc/prometheus/config.yml" source="main.go:248"
time="2017-02-23T11:15:32Z" level=info msg="Loading series map and head chunks..." source="storage.go:373"
time="2017-02-23T11:15:54Z" level=info msg="1133679 series loaded." source="storage.go:378"
time="2017-02-23T11:15:54Z" level=info msg="Starting target manager..." source="targetmanager.go:61"
time="2017-02-23T11:15:54Z" level=info msg="Listening on :9090" source="web.go:259"
fatal error: concurrent map writes
goroutine 837 [running]:
runtime.throw(0x1b05b22, 0x15)
	/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc50ff77c68 sp=0xc50ff77c48
runtime.mapassign1(0x189d2a0, 0xc50ff84030, 0xc50ff77de8, 0xc50ff77de0)
	/usr/local/go/src/runtime/hashmap.go:458 +0x8ef fp=0xc50ff77d50 sp=0xc50ff77c68
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50fda0570, 0xc50eaf3bc0, 0xc50fe00a00, 0xc50fda0590, 0xc50fda0508, 0x5, 0x267d740, 0xc50ea74a80)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:255 +0x424 fp=0xc50ff77f70 sp=0xc50ff77d50
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc50ff77f78 sp=0xc50ff77f70
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 1 [select]:
main.Main(0x0)
	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:228 +0x1b46
main.main()
	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:43 +0x22
goroutine 77 [chan receive]:
github.com/prometheus/prometheus/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x2730ca0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:879 +0x7a
created by github.com/prometheus/prometheus/vendor/github.com/golang/glog.init.1
	/go/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:410 +0x21d
goroutine 278 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc421010300)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x5e1
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x592
goroutine 126 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41
goroutine 128 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc421010000)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x5e1
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x592
goroutine 245 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc420c788c0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x247
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x1bc
goroutine 279 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc421010300)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:96 +0x249
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x5b4
goroutine 244 [select]:
main.Main.func2(0xc420c32de0, 0xc420c32d80, 0xc4206a3810, 0x5, 0x5, 0xc420436380)
	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:163 +0x358
created by main.Main
	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:177 +0x1135
goroutine 274 [select]:
github.com/prometheus/prometheus/rules.(*Group).run(0xc420ca99a0)
	/go/src/github.com/prometheus/prometheus/rules/manager.go:144 +0x31a
github.com/prometheus/prometheus/rules.(*Manager).ApplyConfig.func1.1(0xc420446f00, 0xc420ca99a0)
	/go/src/github.com/prometheus/prometheus/rules/manager.go:426 +0x52
created by github.com/prometheus/prometheus/rules.(*Manager).ApplyConfig.func1
	/go/src/github.com/prometheus/prometheus/rules/manager.go:427 +0xa7
goroutine 275 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc420ca9a40, 0x7f05b79b2028, 0xc420d29180)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:192 +0x2e4
created by github.com/prometheus/prometheus/notifier.(*Notifier).ApplyConfig
	/go/src/github.com/prometheus/prometheus/notifier/notifier.go:163 +0x384
goroutine 259 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc421010180)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x5e1
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x592
goroutine 243 [select, locked to thread]:
runtime.gopark(0x1be2d58, 0x0, 0x1aed2a1, 0x6, 0x18, 0x2)
	/usr/local/go/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc4205d9730, 0x0, 0x18)
	/usr/local/go/src/runtime/select.go:423 +0x1235
runtime.selectgo(0xc4205d9730)
	/usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
goroutine 260 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc421010180)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:96 +0x249
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x5b4
goroutine 289 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc421010000)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:96 +0x249
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x5b4
goroutine 290 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc421010000)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x83c
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x73a
goroutine 291 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc421010000)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x263
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x75c
goroutine 292 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc420436540)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x247
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x1bc
goroutine 261 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc421010180)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x83c
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x73a
goroutine 262 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc421010180)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x263
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x75c
goroutine 263 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc420436a80)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x247
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x1bc
goroutine 280 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc421010300)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x83c
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x73a
goroutine 281 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc421010300)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x263
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x75c
goroutine 282 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc420436c40)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x247
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x1bc
goroutine 305 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc421010480)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x5e1
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x592
goroutine 306 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc421010480)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:96 +0x249
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x5b4
goroutine 307 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc421010480)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x83c
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x73a
goroutine 308 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc421010480)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x263
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x75c
goroutine 309 [select]:
github.com/prometheus/prometheus/storage/local.(*persistence).processIndexingQueue(0xc42009d1e0)
	/go/src/github.com/prometheus/prometheus/storage/local/persistence.go:1386 +0xb96
github.com/prometheus/prometheus/storage/local.(*persistence).run(0xc42009d1e0)
	/go/src/github.com/prometheus/prometheus/storage/local/persistence.go:344 +0x2b
created by github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).Start
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:363 +0x12f
goroutine 264 [select]:
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).handleEvictList(0xc420170580)
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1051 +0x465
created by github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).Start
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:386 +0x3da
goroutine 265 [select]:
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).handleQuarantine(0xc420170580)
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1692 +0x45f
created by github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).Start
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:387 +0x3ff
goroutine 266 [select]:
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).logThrottling(0xc420170580)
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:919 +0x838
created by github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).Start
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:388 +0x424
goroutine 267 [syscall]:
syscall.Syscall(0x0, 0x22, 0xc51a40f600, 0x11, 0x20, 0x20, 0x435ab9)
	/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x22, 0xc51a40f600, 0x11, 0x11, 0x7f05b7a04001, 0x10000000187e680, 0x0)
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:783 +0x55
syscall.Read(0x22, 0xc51a40f600, 0x11, 0x11, 0xc420eafae8, 0x43a3f1, 0xc42029d520)
	/usr/local/go/src/syscall/syscall_unix.go:161 +0x49
os.(*File).read(0xc50ff28108, 0xc51a40f600, 0x11, 0x11, 0x8, 0x22, 0x0)
	/usr/local/go/src/os/file_unix.go:228 +0x4f
os.(*File).Read(0xc50ff28108, 0xc51a40f600, 0x11, 0x11, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/file.go:101 +0x59
io.ReadAtLeast(0x2683740, 0xc50ff28108, 0xc51a40f600, 0x11, 0x11, 0x11, 0x0, 0x0, 0xc420eafc10)
	/usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x2683740, 0xc50ff28108, 0xc51a40f600, 0x11, 0x11, 0x0, 0xc420eafc90, 0xc420eafc90)
	/usr/local/go/src/io/io.go:325 +0x58
github.com/prometheus/prometheus/storage/local.(*persistence).dropAndPersistChunks(0xc42009d1e0, 0xc1f9721ba6783fdf, 0x15a1d6fddab, 0x27514d8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/prometheus/prometheus/storage/local/persistence.go:931 +0x306
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).writeMemorySeries(0xc420170580, 0xc1f9721ba6783fdf, 0xc4373e35e0, 0x15a1d6fddab, 0x2698400)
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1494 +0x25a
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).maintainMemorySeries(0xc420170580, 0xc1f9721ba6783fdf, 0x15a1d6fddab, 0xc50ea8e100)
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1390 +0x250
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).loop(0xc420170580)
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1302 +0x268
created by github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).Start
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:389 +0x449
goroutine 272 [select]:
github.com/prometheus/prometheus/notifier.(*Notifier).Run(0xc42012a0c0)
	/go/src/github.com/prometheus/prometheus/notifier/notifier.go:205 +0x1b8
created by main.Main
	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:208 +0x14fd
goroutine 21 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50eaac140, 0x7f05b79b2028, 0xc50eaa40c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:184 +0x339
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50eaa40c0, 0xc420690310, 0xc50eabc0f0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 322 [semacquire]:
sync.runtime_Semacquire(0xc420690374)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc420690368)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run(0xc420690310)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:70 +0x15e
created by main.Main
	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:214 +0x15a1
goroutine 323 [IO wait]:
net.runtime_pollWait(0x7f05b7895f00, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc50eace140, 0x72, 0xc42273ace8, 0xc420010150)
	/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc50eace140, 0x2686580, 0xc420010150)
	/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc50eace0e0, 0x0, 0x2683840, 0xc50f5bc240)
	/usr/local/go/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc50eaec000, 0xc42273adc0, 0x40a89d, 0x17b3620)
	/usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).Accept(0xc50eaec000, 0xc50eaf4000, 0xc42273adf8, 0x43aacb, 0xc42273ae00)
	/usr/local/go/src/net/tcpsock.go:222 +0x49
github.com/prometheus/prometheus/vendor/golang.org/x/net/netutil.(*limitListener).Accept(0xc50eaca0c0, 0x1be26e8, 0xc50fad0380, 0x2693440, 0xc50eab0150)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/netutil/listen.go:30 +0x5f
net/http.(*Server).Serve(0xc50eada000, 0x268c8c0, 0xc50eaca0c0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2273 +0x1ce
github.com/prometheus/prometheus/web.(*Handler).Run(0xc420436380)
	/go/src/github.com/prometheus/prometheus/web/web.go:271 +0x309
created by main.Main
	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:221 +0x1614
goroutine 337 [select]:
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).waitForNextFP(0xc420170580, 0x114c64, 0x3fe4e61e00000000, 0xc50ea8e001)
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1155 +0x198
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).cycleThroughMemoryFingerprints.func1(0xc50ea8e060, 0xc420170580)
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1195 +0x245
created by github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).cycleThroughMemoryFingerprints
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1205 +0x67
goroutine 338 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.internalKey.ukey(0xc51a4e2470, 0x10, 0x10, 0x28, 0x10, 0x714b1a)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/key.go:115
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*iComparer).Compare(0xc420d97bb0, 0xc51a4e2470, 0x10, 0x10, 0xc51a4e24d0, 0x10, 0x10, 0xc51aae9d10)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/comparer.go:38 +0x4c
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/iterator.(*mergedIterator).next(0xc502863e80, 0x10)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:134 +0x110
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/iterator.(*mergedIterator).Next(0xc502863e80, 0xc50ea5f090)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:176 +0x192
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*dbIter).Next(0xc50e8eecf0, 0xc50ea5f0f0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_iter.go:240 +0xa8
github.com/prometheus/prometheus/storage/local/index.(*LevelDB).ForEach(0xc420bacac0, 0xc50ea63650, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/storage/local/index/leveldb.go:153 +0x175
github.com/prometheus/prometheus/storage/local.(*persistence).fingerprintsModifiedBefore(0xc42009d1e0, 0x15a1d6fdaa0, 0xc50ea8e0c0, 0xc4205dbe70, 0x441d9e, 0xc4205dbf90, 0x0)
	/go/src/github.com/prometheus/prometheus/storage/local/persistence.go:1184 +0x15b
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).cycleThroughArchivedFingerprints.func1(0xc50ea8e0c0, 0xc420170580)
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1221 +0xca
created by github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).cycleThroughArchivedFingerprints
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1248 +0x67
goroutine 339 [select]:
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).loop.func2(0xc420170580, 0xc50ea8e120, 0xc50ea98000, 0xc50ea9e000)
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1271 +0x29a
created by github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).loop
	/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1294 +0x149
goroutine 293 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50eade000, 0x7f05b79b2028, 0xc50eac20c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:184 +0x339
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50eac20c0, 0xc420690310, 0xc50eaaa180)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 297 [semacquire]:
sync.runtime_Semacquire(0xc50eaa22fc)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50eaa22f0)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50eadef00, 0x7f05b79b2028, 0xc50eac2880, 0xc50eaaadb0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50eadef00, 0x7f05b79b2028, 0xc50eac2880)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50eac2880, 0xc420690310, 0xc50eaaad80)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 296 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc50eac6230, 0x2540be400, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:378 +0x795
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:247 +0x3a4
goroutine 194 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50ead0050, 0x7f05b79b2028, 0xc50eaee000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:184 +0x339
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50eaee000, 0xc420690310, 0xc50eaf80c0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 198 [semacquire]:
sync.runtime_Semacquire(0xc50eaf657c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50eaf6570)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50ead07d0, 0x7f05b79b2028, 0xc50eaee400, 0xc50eaf88a0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50ead07d0, 0x7f05b79b2028, 0xc50eaee400)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50eaee400, 0xc420690310, 0xc50eaf8870)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 197 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc50eb24150, 0x2540be400, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:378 +0x795
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:247 +0x3a4
goroutine 246 [semacquire]:
sync.runtime_Semacquire(0xc50eaa218c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50eaa2180)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50ead60a0, 0x7f05b79b2028, 0xc50eac8180, 0xc50eab0270)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50ead60a0, 0x7f05b79b2028, 0xc50eac8180)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50eac8180, 0xc420690310, 0xc50eab0240)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 247 [select]:
github.com/prometheus/prometheus/discovery/marathon.(*Discovery).Run(0xc50eac81c0, 0x7f05b79b2028, 0xc50eac8240, 0xc50eaf42a0)
	/go/src/github.com/prometheus/prometheus/discovery/marathon/marathon.go:107 +0x1d8
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 248 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50eaa2160, 0xc50eaf42a0, 0xc50ead60a0, 0xc50eaa2180, 0xc50eaa2150, 0xa, 0x267d7c0, 0xc50eac81c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 340 [chan send]:
github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter.func1(0xc420e80240, 0xc50ea8e180)
	/go/src/github.com/prometheus/prometheus/storage/local/series.go:126 +0xe0
created by github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter
	/go/src/github.com/prometheus/prometheus/storage/local/series.go:131 +0x67
goroutine 341 [chan receive]:
github.com/prometheus/prometheus/discovery/kubernetes.(*Pod).Run(0xc50ebee2d0, 0x7f05b79b2028, 0xc50ea98280, 0xc50ea8e2a0)
	/go/src/github.com/prometheus/prometheus/discovery/kubernetes/pod.go:106 +0x74c
github.com/prometheus/prometheus/discovery/kubernetes.(*Kubernetes).Run(0xc50eaaac90, 0x7f05b79b2028, 0xc50ea98280, 0xc50ea8e2a0)
	/go/src/github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:189 +0xda5
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 342 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9e1b0, 0xc50ea8e2a0, 0xc50eaac140, 0xc50ea9e1d0, 0xc50ea9aaf0, 0xc, 0x267d780, 0xc50eaaac90)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 343 [semacquire]:
sync.runtime_notifyListWait(0xc420362188, 0x1)
	/usr/local/go/src/runtime/sema.go:267 +0x122
sync.(*Cond).Wait(0xc420362178)
	/usr/local/go/src/sync/cond.go:57 +0x80
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*DeltaFIFO).Pop(0xc420362160, 0xc50ea9e200, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/delta_fifo.go:407 +0xc5
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Controller).processLoop(0xc50ebea1c0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/controller.go:126 +0x3c
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Controller).(github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.processLoop)-fm()
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/controller.go:102 +0x2a
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil.func1(0xc50f1dbee0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:84 +0x19
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil(0xc50f1dbee0, 0x3b9aca00, 0x0, 0x17bdd01, 0xc50ea8e240)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:85 +0xad
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.Until(0xc50f1dbee0, 0x3b9aca00, 0xc50ea8e240)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:47 +0x4d
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Controller).Run(0xc50ebea1c0, 0xc50ea8e240)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/controller.go:102 +0x1bc
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*sharedIndexInformer).Run(0xc50ebea150, 0xc50ea8e240)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:184 +0x322
created by github.com/prometheus/prometheus/discovery/kubernetes.(*Kubernetes).Run
	/go/src/github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:184 +0xd3c
goroutine 344 [select]:
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).watchHandler(0xc50f1de000, 0x2688e80, 0xc511dac420, 0xc51023bc60, 0xc510e76a20, 0xc50ea8e240, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:357 +0xdf6
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).ListAndWatch(0xc50f1de000, 0xc50ea8e240, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:328 +0xa6f
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).RunUntil.func1()
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:213 +0x33
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil.func1(0xc50ea90600)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:84 +0x19
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.JitterUntil(0xc50ea90600, 0x3b9aca00, 0x0, 0xc4201f0501, 0xc50ea8e240)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:85 +0xad
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait.Until(0xc50ea90600, 0x3b9aca00, 0xc50ea8e240)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/wait/wait.go:47 +0x4d
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).RunUntil
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:216 +0x1c4
goroutine 302 [runnable]:
github.com/prometheus/prometheus/retrieval.populateLabels(0xc51a9b7830, 0xc420526d80, 0xc510381b48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:321 +0xb14
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50f4a0000, 0xc420526d80, 0x27309c0, 0xc420060000, 0x0, 0x7f05b79fb840, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50ead4900, 0xc519c43d80, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50eadf1d0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50eadf1d0, 0x7f05b79b2028, 0xc50eac2c80)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50eac2c80, 0xc420690310, 0xc50eaab410)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 249 [runnable]:
reflect.Value.IsNil(0x18797c0, 0xc51afa0160, 0x196, 0x0)
	/usr/local/go/src/reflect/value.go:960
encoding/json.(*decodeState).indirect(0xc51069f3a8, 0x18797c0, 0xc51afa0160, 0x196, 0x1878c00, 0xc51ad511c0, 0x196, 0x0, 0x0, 0x1794720, ...)
	/usr/local/go/src/encoding/json/decode.go:430 +0x10c
encoding/json.(*decodeState).object(0xc51069f3a8, 0x18797c0, 0xc51afa0160, 0x196)
	/usr/local/go/src/encoding/json/decode.go:559 +0x82
encoding/json.(*decodeState).value(0xc51069f3a8, 0x18797c0, 0xc51afa0160, 0x196)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc
encoding/json.(*decodeState).array(0xc51069f3a8, 0x1794b20, 0xc51ad15880, 0x197)
	/usr/local/go/src/encoding/json/decode.go:519 +0x1e1
encoding/json.(*decodeState).value(0xc51069f3a8, 0x1794b20, 0xc51ad15880, 0x197)
	/usr/local/go/src/encoding/json/decode.go:365 +0x25f
encoding/json.(*decodeState).object(0xc51069f3a8, 0x18779c0, 0xc519ac3988, 0x196)
	/usr/local/go/src/encoding/json/decode.go:696 +0x1080
encoding/json.(*decodeState).value(0xc51069f3a8, 0x18779c0, 0xc519ac3988, 0x196)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc
encoding/json.(*decodeState).array(0xc51069f3a8, 0x17943a0, 0xc510af7510, 0x197)
	/usr/local/go/src/encoding/json/decode.go:519 +0x1e1
encoding/json.(*decodeState).value(0xc51069f3a8, 0x17943a0, 0xc510af7510, 0x197)
	/usr/local/go/src/encoding/json/decode.go:365 +0x25f
encoding/json.(*decodeState).object(0xc51069f3a8, 0x174f540, 0xc50f5064e0, 0x16)
	/usr/local/go/src/encoding/json/decode.go:696 +0x1080
encoding/json.(*decodeState).value(0xc51069f3a8, 0x174f540, 0xc50f5064e0, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc
encoding/json.(*decodeState).unmarshal(0xc51069f3a8, 0x174f540, 0xc50f5064e0, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa
encoding/json.(*Decoder).Decode(0xc51069f380, 0x174f540, 0xc50f5064e0, 0x7f05b7499848, 0xc510bc77e0)
	/usr/local/go/src/encoding/json/stream.go:67 +0x15a
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Do(0xc5109a9cd8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26210 +0x377
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc5109a9cd8, 0x0, 0x0, 0xc5109a9c88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26277 +0x104
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50eac68c0, 0xc50f46e000, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eac68c0, 0x7f05b79b2028, 0xc50eac8340, 0xc50eaf44e0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 303 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eac6b60, 0x7f05b79b2028, 0xc50eac2f40, 0xc50ead4ae0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 250 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50eaa22d0, 0xc50eaf44e0, 0xc50eadef00, 0xc50eaa22f0, 0xc50ea9ad00, 0x5, 0x267d740, 0xc50eac68c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 251 [select]:
net/http.(*http2ClientConn).RoundTrip(0xc510b2a340, 0xc50fad22d0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:5560 +0xa29
net/http.(*http2Transport).RoundTripOpt(0xc50f53c820, 0xc50fad22d0, 0xc420262000, 0xc50fad22d0, 0x1000200000000, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:5160 +0x18e
net/http.(*http2Transport).RoundTrip(0xc50f53c820, 0xc50fad22d0, 0x0, 0xc50f552180, 0x5)
	/usr/local/go/src/net/http/h2_bundle.go:5130 +0x40
net/http.(*Transport).RoundTrip(0xc420262000, 0xc50fad22d0, 0xc50fad22d0, 0x0, 0xc50fb5f6b0)
	/usr/local/go/src/net/http/transport.go:378 +0x39c
github.com/prometheus/prometheus/vendor/golang.org/x/oauth2.(*Transport).RoundTrip(0xc50eaab230, 0xc50fa5c000, 0xc50eaab230, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/oauth2/transport.go:48 +0xda
net/http.send(0xc50fa5c000, 0x267ea80, 0xc50eaab230, 0x0, 0x0, 0x0, 0x8, 0xc50fccc010, 0x416bb8)
	/usr/local/go/src/net/http/client.go:256 +0x15f
net/http.(*Client).send(0xc50eaab260, 0xc50fa5c000, 0x0, 0x0, 0x0, 0xc50fccc010, 0x0, 0x1)
	/usr/local/go/src/net/http/client.go:146 +0x102
net/http.(*Client).doFollowingRedirects(0xc50eaab260, 0xc50fa5c000, 0x1be2898, 0x3, 0x1, 0xc50f5ba2c0)
	/usr/local/go/src/net/http/client.go:528 +0x5e5
net/http.(*Client).Do(0xc50eaab260, 0xc50fa5c000, 0xc420ebbaa8, 0xc420ebba98, 0xc50f596150)
	/usr/local/go/src/net/http/client.go:184 +0x1ea
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).doRequest(0xc510a4fcd8, 0x1aea84c, 0x4, 0x0, 0xb554b0, 0xc420ebbb68)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26174 +0x43f
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Do(0xc510a4fcd8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26186 +0xa1
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc510a4fcd8, 0x0, 0x0, 0xc510a4fc88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26277 +0x104
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50eac6a80, 0xc50f46a000, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eac6a80, 0x7f05b79b2028, 0xc50eac8340, 0xc50eaf4540)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 252 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50eaa22d0, 0xc50eaf4540, 0xc50eadef00, 0xc50eaa22f0, 0xc50ea9ad60, 0x5, 0x267d740, 0xc50eac6a80)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 253 [runnable]:
sync.(*Mutex).Unlock(0xc51027de28)
	/usr/local/go/src/sync/mutex.go:102
net/http.(*http2pipe).Read(0xc51027de28, 0xc51aeb5e00, 0x10000, 0x10000, 0x10000, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:2727 +0x1aa
net/http.http2transportResponseBody.Read(0xc51027de00, 0xc51aeb5e00, 0x10000, 0x10000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:6241 +0xd9
github.com/prometheus/prometheus/vendor/golang.org/x/oauth2.(*onEOFReader).Read(0xc51a4f36c0, 0xc51aeb5e00, 0x10000, 0x10000, 0xfe00, 0x1fe00, 0xc50fb739f8)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/oauth2/transport.go:114 +0x55
encoding/json.(*Decoder).refill(0xc515f9aea0, 0x61, 0x0)
	/usr/local/go/src/encoding/json/stream.go:152 +0xfa
encoding/json.(*Decoder).readValue(0xc515f9aea0, 0x0, 0x0, 0x1a0)
	/usr/local/go/src/encoding/json/stream.go:128 +0x198
encoding/json.(*Decoder).Decode(0xc515f9aea0, 0x174f540, 0xc50eaec100, 0x7f05b7499848, 0xc51a4f36c0)
	/usr/local/go/src/encoding/json/stream.go:57 +0x8e
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Do(0xc50fb73cd8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26210 +0x377
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc50fb73cd8, 0x0, 0x0, 0xc50fb73c88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26277 +0x104
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50eac6af0, 0xc50eab0630, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eac6af0, 0x7f05b79b2028, 0xc50eac8340, 0xc50eaf45a0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 254 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50eaa22d0, 0xc50eaf45a0, 0xc50eadef00, 0xc50eaa22f0, 0xc50ea9adc0, 0x5, 0x267d740, 0xc50eac6af0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 304 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9af10, 0xc50ead4ae0, 0xc50eadf1d0, 0xc50ea9af30, 0xc50ea9ae40, 0x5, 0x267d740, 0xc50eac6b60)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 353 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eac6bd0, 0x7f05b79b2028, 0xc50eac2f40, 0xc50ead4b40)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 354 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9af10, 0xc50ead4b40, 0xc50eadf1d0, 0xc50ea9af30, 0xc50ea9aea0, 0x5, 0x267d740, 0xc50eac6bd0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 355 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eac6c40, 0x7f05b79b2028, 0xc50eac2f40, 0xc50ead4ba0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 356 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9af10, 0xc50ead4ba0, 0xc50eadf1d0, 0xc50ea9af30, 0xc50ea9af00, 0x5, 0x267d740, 0xc50eac6c40)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 357 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50eadf400, 0x7f05b79b2028, 0xc50eac2fc0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:184 +0x339
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50eac2fc0, 0xc420690310, 0xc50eaab7d0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 386 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc50f4c63f0, 0x2540be400, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:378 +0x795
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:247 +0x3a4
goroutine 385 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc50f4c6230, 0x2540be400, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:378 +0x795
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:247 +0x3a4
goroutine 360 [semacquire]:
sync.runtime_Semacquire(0xc50ea9b12c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50ea9b120)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50eadf590, 0x7f05b79b2028, 0xc50eac30c0, 0xc50eaab980)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50eadf590, 0x7f05b79b2028, 0xc50eac30c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50eac30c0, 0xc420690310, 0xc50eaab950)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 361 [runnable]:
encoding/json.equalFoldRight(0xc511d89ab0, 0xb, 0x10, 0xc519f69ef1, 0x4, 0x27f0f, 0x0)
	/usr/local/go/src/encoding/json/fold.go:61
encoding/json.(*decodeState).object(0xc50ff67548, 0x18779c0, 0xc51ab88780, 0x196)
	/usr/local/go/src/encoding/json/decode.go:658 +0x153e
encoding/json.(*decodeState).value(0xc50ff67548, 0x18779c0, 0xc51ab88780, 0x196)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc
encoding/json.(*decodeState).array(0xc50ff67548, 0x17943a0, 0xc50f1de810, 0x197)
	/usr/local/go/src/encoding/json/decode.go:519 +0x1e1
encoding/json.(*decodeState).value(0xc50ff67548, 0x17943a0, 0xc50f1de810, 0x197)
	/usr/local/go/src/encoding/json/decode.go:365 +0x25f
encoding/json.(*decodeState).object(0xc50ff67548, 0x174f540, 0xc50ebec430, 0x16)
	/usr/local/go/src/encoding/json/decode.go:696 +0x1080
encoding/json.(*decodeState).value(0xc50ff67548, 0x174f540, 0xc50ebec430, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc
encoding/json.(*decodeState).unmarshal(0xc50ff67548, 0x174f540, 0xc50ebec430, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa
encoding/json.(*Decoder).Decode(0xc50ff67520, 0x174f540, 0xc50ebec430, 0x7f05b7499848, 0xc50ea91000)
	/usr/local/go/src/encoding/json/stream.go:67 +0x15a
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Do(0xc510e73cd8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26210 +0x377
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc510e73cd8, 0x0, 0x0, 0xc510e73c88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26277 +0x104
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50eac6cb0, 0xc50f4ce000, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eac6cb0, 0x7f05b79b2028, 0xc50eac3380, 0xc50ead51a0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 362 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9b100, 0xc50ead51a0, 0xc50eadf590, 0xc50ea9b120, 0xc50ea9b038, 0x5, 0x267d740, 0xc50eac6cb0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 363 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eac6d20, 0x7f05b79b2028, 0xc50eac3380, 0xc50ead5200)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 364 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9b100, 0xc50ead5200, 0xc50eadf590, 0xc50ea9b120, 0xc50ea9b098, 0x5, 0x267d740, 0xc50eac6d20)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 365 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eac6d90, 0x7f05b79b2028, 0xc50eac3380, 0xc50ead5260)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 366 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9b100, 0xc50ead5260, 0xc50eadf590, 0xc50ea9b120, 0xc50ea9b0f8, 0x5, 0x267d740, 0xc50eac6d90)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 369 [semacquire]:
sync.runtime_Semacquire(0xc50f4da17c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50f4da170)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50f4e8000, 0x7f05b79b2028, 0xc50f4d0000, 0xc50f4e40f0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50f4e8000, 0x7f05b79b2028, 0xc50f4d0000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f4d0000, 0xc420690310, 0xc50f4e40c0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 370 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f4fa070, 0x7f05b79b2028, 0xc50f4d0300, 0xc50f4d2240)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 371 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f4da150, 0xc50f4d2240, 0xc50f4e8000, 0xc50f4da170, 0xc50f4da0e0, 0x5, 0x267d740, 0xc50f4fa070)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 372 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f4fa0e0, 0x7f05b79b2028, 0xc50f4d0300, 0xc50f4d22a0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 373 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f4da150, 0xc50f4d22a0, 0xc50f4e8000, 0xc50f4da170, 0xc50f4da140, 0x5, 0x267d740, 0xc50f4fa0e0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 374 [runnable]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh.func2(0xc5148cb480, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:192 +0x83f
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc510849cd8, 0x0, 0x0, 0xc510849c88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26281 +0x132
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50f4fa000, 0xc50f53e090, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f4fa000, 0x7f05b79b2028, 0xc50f4d0300, 0xc50f4d2300)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 375 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f4da150, 0xc50f4d2300, 0xc50f4e8000, 0xc50f4da170, 0xc50f4da080, 0x5, 0x267d740, 0xc50f4fa000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 401 [runnable]:
github.com/prometheus/prometheus/retrieval.populateLabels(0xc51ae565a0, 0xc420527d40, 0xc50f1e9b48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:321 +0xb14
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50f5367b0, 0xc420527d40, 0x0, 0x0, 0xc517450600, 0x0, 0x5f)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50f52c060, 0xc51956ecc0, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50f53c000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50f53c000, 0x7f05b79b2028, 0xc50f52a000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f52a000, 0xc420690310, 0xc50f5360c0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 402 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f558000, 0x7f05b79b2028, 0xc50f52a2c0, 0xc50f52c240)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 403 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f532150, 0xc50f52c240, 0xc50f53c000, 0xc50f532170, 0xc50f532080, 0x5, 0x267d740, 0xc50f558000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 404 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f558070, 0x7f05b79b2028, 0xc50f52a2c0, 0xc50f52c2a0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 405 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f532150, 0xc50f52c2a0, 0xc50f53c000, 0xc50f532170, 0xc50f5320e0, 0x5, 0x267d740, 0xc50f558070)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 406 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f5580e0, 0x7f05b79b2028, 0xc50f52a2c0, 0xc50f52c300)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 407 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f532150, 0xc50f52c300, 0xc50f53c000, 0xc50f532170, 0xc50f532140, 0x5, 0x267d740, 0xc50f5580e0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 408 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.LabelSet.Clone(0xc51aa0d920, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/labelset.go:111 +0x127
github.com/prometheus/prometheus/retrieval.populateLabels(0xc51aa0d920, 0xc420554240, 0xc50fe39b48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:331 +0x393
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50f596000, 0xc420554240, 0x27309c0, 0xc42058c400, 0x0, 0x7f05b7a04590, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50f52c420, 0xc51a86efc0, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50f53c280)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50f53c280, 0x7f05b79b2028, 0xc50f52a340)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f52a340, 0xc420690310, 0xc50f5364b0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 409 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f558150, 0x7f05b79b2028, 0xc50f52a640, 0xc50f52c600)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 410 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f5322c0, 0xc50f52c600, 0xc50f53c280, 0xc50f5322e0, 0xc50f5321f0, 0x5, 0x267d740, 0xc50f558150)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 411 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f5581c0, 0x7f05b79b2028, 0xc50f52a640, 0xc50f52c660)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 412 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f5322c0, 0xc50f52c660, 0xc50f53c280, 0xc50f5322e0, 0xc50f532250, 0x5, 0x267d740, 0xc50f5581c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 413 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f558230, 0x7f05b79b2028, 0xc50f52a640, 0xc50f52c6c0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 414 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f5322c0, 0xc50f52c6c0, 0xc50f53c280, 0xc50f5322e0, 0xc50f5322b0, 0x5, 0x267d740, 0xc50f558230)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 417 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.LabelSet.Clone(0xc51af660c0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/labelset.go:111 +0x127
github.com/prometheus/prometheus/retrieval.populateLabels(0xc51af660c0, 0xc4201f4240, 0xc5100bdb48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:331 +0x393
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50f618000, 0xc4201f4240, 0x27309c0, 0xc42048a000, 0x0, 0x7f05b7a01b60, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50f588060, 0xc51a27dbc0, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50f59c000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50f59c000, 0x7f05b79b2028, 0xc50f586000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f586000, 0xc420690310, 0xc50f5940c0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 418 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f5ae000, 0x7f05b79b2028, 0xc50f5862c0, 0xc50f588240)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 419 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f58e150, 0xc50f588240, 0xc50f59c000, 0xc50f58e170, 0xc50f58e080, 0x5, 0x267d740, 0xc50f5ae000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 420 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f5ae070, 0x7f05b79b2028, 0xc50f5862c0, 0xc50f5882a0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 421 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f58e150, 0xc50f5882a0, 0xc50f59c000, 0xc50f58e170, 0xc50f58e0e0, 0x5, 0x267d740, 0xc50f5ae070)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 422 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f5ae0e0, 0x7f05b79b2028, 0xc50f5862c0, 0xc50f588300)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 423 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f58e150, 0xc50f588300, 0xc50f59c000, 0xc50f58e170, 0xc50f58e140, 0x5, 0x267d740, 0xc50f5ae0e0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 433 [semacquire]:
sync.runtime_Semacquire(0xc50f65002c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50f650020)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50f634000, 0x7f05b79b2028, 0xc50f60e000, 0xc50f62c0f0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50f634000, 0x7f05b79b2028, 0xc50f60e000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f60e000, 0xc420690310, 0xc50f62c0c0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 434 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.LabelSet.Clone(0xc51ac7aba0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/labelset.go:111 +0x127
github.com/prometheus/prometheus/retrieval.populateLabels(0xc51ac7aba0, 0xc4201f4b40, 0xc5108f7b48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:331 +0x393
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50f62c750, 0xc4201f4b40, 0x27309c0, 0xc42058c000, 0x0, 0x7f05b7a01200, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50f6101e0, 0xc51a1534a0, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50f634140)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50f634140, 0x7f05b79b2028, 0xc50f60e2c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f60e2c0, 0xc420690310, 0xc50f62c450)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 435 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f63a1c0, 0x7f05b79b2028, 0xc50f60e580, 0xc50f6103c0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 436 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f624290, 0xc50f6103c0, 0xc50f634140, 0xc50f6242b0, 0xc50f624220, 0x5, 0x267d740, 0xc50f63a1c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 437 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f63a230, 0x7f05b79b2028, 0xc50f60e580, 0xc50f610420)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 438 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f624290, 0xc50f610420, 0xc50f634140, 0xc50f6242b0, 0xc50f624280, 0x5, 0x267d740, 0xc50f63a230)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 439 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f63a150, 0x7f05b79b2028, 0xc50f60e580, 0xc50f610480)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 440 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f624290, 0xc50f610480, 0xc50f634140, 0xc50f6242b0, 0xc50f6241c0, 0x5, 0x267d740, 0xc50f63a150)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 449 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f63a070, 0x7f05b79b2028, 0xc50f612100, 0xc50f606180)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 450 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f650000, 0xc50f606180, 0xc50f634000, 0xc50f650020, 0xc50f6240e0, 0x5, 0x267d740, 0xc50f63a070)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 451 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f63a0e0, 0x7f05b79b2028, 0xc50f612100, 0xc50f6061e0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 452 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f650000, 0xc50f6061e0, 0xc50f634000, 0xc50f650020, 0xc50f624140, 0x5, 0x267d740, 0xc50f63a0e0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 453 [runnable]:
reflect.Value.SetString(0x17bdd20, 0xc51ad23de0, 0x198, 0xc51afad900, 0x1813)
	/usr/local/go/src/reflect/value.go:1510
encoding/json.(*decodeState).literalStore(0xc50ff676e8, 0xc519fefba6, 0x1821, 0x2e25a, 0x1788280, 0xc51ae80be0, 0x196, 0x628600)
	/usr/local/go/src/encoding/json/decode.go:875 +0x1e58
encoding/json.(*decodeState).literal(0xc50ff676e8, 0x1788280, 0xc51ae80be0, 0x196)
	/usr/local/go/src/encoding/json/decode.go:759 +0xd3
encoding/json.(*decodeState).value(0xc50ff676e8, 0x1788280, 0xc51ae80be0, 0x196)
	/usr/local/go/src/encoding/json/decode.go:371 +0x306
encoding/json.(*decodeState).object(0xc50ff676e8, 0x1878ce0, 0xc51aecb220, 0x196)
	/usr/local/go/src/encoding/json/decode.go:696 +0x1080
encoding/json.(*decodeState).value(0xc50ff676e8, 0x1878ce0, 0xc51aecb220, 0x196)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc
encoding/json.(*decodeState).array(0xc50ff676e8, 0x1794720, 0xc51ac93910, 0x197)
	/usr/local/go/src/encoding/json/decode.go:519 +0x1e1
encoding/json.(*decodeState).value(0xc50ff676e8, 0x1794720, 0xc51ac93910, 0x197)
	/usr/local/go/src/encoding/json/decode.go:365 +0x25f
encoding/json.(*decodeState).object(0xc50ff676e8, 0x1878c80, 0xc51ade90b8, 0x196)
	/usr/local/go/src/encoding/json/decode.go:696 +0x1080
encoding/json.(*decodeState).value(0xc50ff676e8, 0x1878c80, 0xc51ade90b8, 0x196)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc
encoding/json.(*decodeState).object(0xc50ff676e8, 0x18779c0, 0xc519446bd0, 0x196)
	/usr/local/go/src/encoding/json/decode.go:696 +0x1080
encoding/json.(*decodeState).value(0xc50ff676e8, 0x18779c0, 0xc519446bd0, 0x196)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc
encoding/json.(*decodeState).array(0xc50ff676e8, 0x17943a0, 0xc50f1de890, 0x197)
	/usr/local/go/src/encoding/json/decode.go:519 +0x1e1
encoding/json.(*decodeState).value(0xc50ff676e8, 0x17943a0, 0xc50f1de890, 0x197)
	/usr/local/go/src/encoding/json/decode.go:365 +0x25f
encoding/json.(*decodeState).object(0xc50ff676e8, 0x174f540, 0xc50ebec438, 0x16)
	/usr/local/go/src/encoding/json/decode.go:696 +0x1080
encoding/json.(*decodeState).value(0xc50ff676e8, 0x174f540, 0xc50ebec438, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc
encoding/json.(*decodeState).unmarshal(0xc50ff676e8, 0x174f540, 0xc50ebec438, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa
encoding/json.(*Decoder).Decode(0xc50ff676c0, 0x174f540, 0xc50ebec438, 0x7f05b7499848, 0xc50ea91060)
	/usr/local/go/src/encoding/json/stream.go:67 +0x15a
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Do(0xc510e97cd8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26210 +0x377
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc510e97cd8, 0x0, 0x0, 0xc510e97c88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26277 +0x104
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50f63a000, 0xc50f6c2000, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f63a000, 0x7f05b79b2028, 0xc50f612100, 0xc50f606240)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 454 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f650000, 0xc50f606240, 0xc50f634000, 0xc50f650020, 0xc50f624080, 0x5, 0x267d740, 0xc50f63a000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 465 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.LabelSet.Clone(0xc51ae283f0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/labelset.go:111 +0x127
github.com/prometheus/prometheus/retrieval.populateLabels(0xc51ae283f0, 0xc4201f4fc0, 0xc50fb1fb48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:331 +0x393
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50eaf8de0, 0xc4201f4fc0, 0x27309c0, 0xc420276000, 0x0, 0x7f05b7a064c0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50f5d60c0, 0xc51a5798e0, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50f60a0f0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50f60a0f0, 0x7f05b79b2028, 0xc50f5dc040)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f5dc040, 0xc420690310, 0xc50f65e0c0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 466 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f680000, 0x7f05b79b2028, 0xc50f5dc340, 0xc50f5d62a0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 467 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f658150, 0xc50f5d62a0, 0xc50f60a0f0, 0xc50f658170, 0xc50f658080, 0x5, 0x267d740, 0xc50f680000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 468 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f680070, 0x7f05b79b2028, 0xc50f5dc340, 0xc50f5d6300)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 469 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f658150, 0xc50f5d6300, 0xc50f60a0f0, 0xc50f658170, 0xc50f6580e0, 0x5, 0x267d740, 0xc50f680070)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 470 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f6800e0, 0x7f05b79b2028, 0xc50f5dc340, 0xc50f5d6360)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 471 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f658150, 0xc50f5d6360, 0xc50f60a0f0, 0xc50f658170, 0xc50f658140, 0x5, 0x267d740, 0xc50f6800e0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 472 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.LabelSet.Clone(0xc514cde5d0, 0xc420ba3c00)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/labelset.go:111 +0x127
github.com/prometheus/prometheus/retrieval.populateLabels(0xc514cde5d0, 0xc4201f5200, 0xc5100e5b48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:308 +0x43
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50f65e810, 0xc4201f5200, 0x27309c0, 0xc4204e2400, 0x0, 0x7f05b7a03c30, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50f5d6480, 0xc5195f9ee0, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50f60a370)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50f60a370, 0x7f05b79b2028, 0xc50f5dc3c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f5dc3c0, 0xc420690310, 0xc50f65e4b0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 473 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f680150, 0x7f05b79b2028, 0xc50f5dc6c0, 0xc50f5d6660)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 474 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f6582c0, 0xc50f5d6660, 0xc50f60a370, 0xc50f6582e0, 0xc50f6581f0, 0x5, 0x267d740, 0xc50f680150)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 475 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f6801c0, 0x7f05b79b2028, 0xc50f5dc6c0, 0xc50f5d66c0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 476 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f6582c0, 0xc50f5d66c0, 0xc50f60a370, 0xc50f6582e0, 0xc50f658250, 0x5, 0x267d740, 0xc50f6801c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 477 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50f680230, 0x7f05b79b2028, 0xc50f5dc6c0, 0xc50f5d6720)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 478 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f6582c0, 0xc50f5d6720, 0xc50f60a370, 0xc50f6582e0, 0xc50f6582b0, 0x5, 0x267d740, 0xc50f680230)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 199 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eb243f0, 0x7f05b79b2028, 0xc50eaee6c0, 0xc50eaf07e0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 200 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50eaf6550, 0xc50eaf07e0, 0xc50ead07d0, 0xc50eaf6570, 0xc50eaf6488, 0x5, 0x267d740, 0xc50eb243f0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 201 [select]:
net/http.(*http2ClientConn).RoundTrip(0xc510b2a340, 0xc50fa925a0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:5560 +0xa29
net/http.(*http2Transport).RoundTripOpt(0xc50f53c820, 0xc50fa925a0, 0xc420262000, 0xc50fa925a0, 0x100020000, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:5160 +0x18e
net/http.(*http2Transport).RoundTrip(0xc50f53c820, 0xc50fa925a0, 0x0, 0xc50f4eb440, 0x5)
	/usr/local/go/src/net/http/h2_bundle.go:5130 +0x40
net/http.(*Transport).RoundTrip(0xc420262000, 0xc50fa925a0, 0xc50fa925a0, 0x0, 0xc50f8696c0)
	/usr/local/go/src/net/http/transport.go:378 +0x39c
github.com/prometheus/prometheus/vendor/golang.org/x/oauth2.(*Transport).RoundTrip(0xc50eaf8a20, 0xc50fa06780, 0xc50eaf8a20, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/oauth2/transport.go:48 +0xda
net/http.send(0xc50fa06780, 0x267ea80, 0xc50eaf8a20, 0x0, 0x0, 0x0, 0x8, 0xc50f49e0d0, 0x416bb8)
	/usr/local/go/src/net/http/client.go:256 +0x15f
net/http.(*Client).send(0xc50eaf8a50, 0xc50fa06780, 0x0, 0x0, 0x0, 0xc50f49e0d0, 0x0, 0x1)
	/usr/local/go/src/net/http/client.go:146 +0x102
net/http.(*Client).doFollowingRedirects(0xc50eaf8a50, 0xc50fa06780, 0x1be2898, 0x3, 0x1, 0xc50f4c2b00)
	/usr/local/go/src/net/http/client.go:528 +0x5e5
net/http.(*Client).Do(0xc50eaf8a50, 0xc50fa06780, 0xc50f869aa8, 0xc50f869a98, 0xc50f4a2ea0)
	/usr/local/go/src/net/http/client.go:184 +0x1ea
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).doRequest(0xc51084dcd8, 0x1aea84c, 0x4, 0x0, 0xb554b0, 0xc50f869b68)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26174 +0x43f
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Do(0xc51084dcd8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26186 +0xa1
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc51084dcd8, 0x0, 0x0, 0xc51084dc88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26277 +0x104
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50eb24460, 0xc50f4a2d80, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eb24460, 0x7f05b79b2028, 0xc50eaee6c0, 0xc50eaf0840)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 202 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50eaf6550, 0xc50eaf0840, 0xc50ead07d0, 0xc50eaf6570, 0xc50eaf64e8, 0x5, 0x267d740, 0xc50eb24460)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 203 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50eb244d0, 0x7f05b79b2028, 0xc50eaee6c0, 0xc50eaf08a0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 204 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50eaf6550, 0xc50eaf08a0, 0xc50ead07d0, 0xc50eaf6570, 0xc50eaf6548, 0x5, 0x267d740, 0xc50eb244d0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 332 [semacquire]:
sync.runtime_Semacquire(0xc50fda042c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50fda0420)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50fe00780, 0x7f05b79b2028, 0xc50f971ec0, 0xc50faed9b0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50fe00780, 0x7f05b79b2028, 0xc50f971ec0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f971ec0, 0xc420690310, 0xc50faed980)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 216 [semacquire]:
sync.runtime_Semacquire(0xc50f4d655c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50f4d6550)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50f4907d0, 0x7f05b79b2028, 0xc50f48a940, 0xc50f4e27b0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50f4907d0, 0x7f05b79b2028, 0xc50f48a940)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f48a940, 0xc420690310, 0xc50f4e2780)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 1061 [IO wait]:
net.runtime_pollWait(0x7f05b7895d80, 0x72, 0x1c)
	/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc50f4c67d0, 0x72, 0xc50f493808, 0xc420010150)
	/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc50f4c67d0, 0x2686580, 0xc420010150)
	/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc50f4c6770, 0xc510fe6000, 0x4000, 0x4000, 0x0, 0x2686580, 0xc420010150)
	/usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc50fc06000, 0xc510fe6000, 0x4000, 0x4000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:173 +0x70
crypto/tls.(*block).readFromUntil(0xc50f4b8cf0, 0x7f05b77d1880, 0xc50fc06000, 0x5, 0xc50fc06000, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:476 +0x91
crypto/tls.(*Conn).readRecord(0xc420441880, 0x1be2e17, 0xc420441988, 0x196d1a0)
	/usr/local/go/src/crypto/tls/conn.go:578 +0xc4
crypto/tls.(*Conn).Read(0xc420441880, 0xc510f6c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1113 +0x116
bufio.(*Reader).fill(0xc510b80ae0)
	/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc510b80ae0, 0xc50f5b6578, 0x9, 0x9, 0x10, 0x1ae9a3c, 0xc50eadcab0)
	/usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x267cdc0, 0xc510b80ae0, 0xc50f5b6578, 0x9, 0x9, 0x9, 0x4, 0xc50f493e28, 0xc420278d00)
	/usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x267cdc0, 0xc510b80ae0, 0xc50f5b6578, 0x9, 0x9, 0x4, 0x0, 0x8)
	/usr/local/go/src/io/io.go:325 +0x58
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.readFrameHeader(0xc50f5b6578, 0x9, 0x9, 0x267cdc0, 0xc510b80ae0, 0x0, 0x0, 0x17b30e0, 0xc510e76c00)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc50f5b6540, 0xc511dac330, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/frame.go:464 +0xa4
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run(0xc50f493f80, 0x1bde930, 0xc50ff6ff90)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/transport.go:1233 +0x97
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0xc50fd4dd40)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/transport.go:1170 +0xa6
created by github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.(*Transport).NewClientConn
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/transport.go:492 +0x89b
goroutine 316 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.LabelSet.Clone(0xc51ad78db0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/labelset.go:111 +0x127
github.com/prometheus/prometheus/retrieval.populateLabels(0xc51ad78db0, 0xc42052e240, 0xc510b1db48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:331 +0x393
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50eab0ab0, 0xc42052e240, 0x27309c0, 0xc42042dc00, 0x0, 0x7f05b79faa30, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50fb16f60, 0xc51a972ec0, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50fa8e550)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50fa8e550, 0x7f05b79b2028, 0xc50fae8580)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50fae8580, 0xc420690310, 0xc50f4a0690)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 1082 [select]:
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).ListAndWatch.func1(0xc50ebec018, 0xc50ea8e240, 0xc50f1de000, 0xc510e76a20, 0xc50ebec020)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:279 +0x2ae
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*Reflector).ListAndWatch
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:292 +0x4c0
goroutine 619 [semacquire]:
sync.runtime_notifyListWait(0xc510e62680, 0x0)
	/usr/local/go/src/runtime/sema.go:267 +0x122
sync.(*Cond).Wait(0xc510e62670)
	/usr/local/go/src/sync/cond.go:57 +0x80
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.(*pipe).Read(0xc510e62668, 0xc420cb5c00, 0x200, 0x200, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/pipe.go:60 +0xd9
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.transportResponseBody.Read(0xc510e62640, 0xc420cb5c00, 0x200, 0x200, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/transport.go:1465 +0xd9
encoding/json.(*Decoder).refill(0xc510edc4e0, 0x18, 0x17ea780)
	/usr/local/go/src/encoding/json/stream.go:152 +0xfa
encoding/json.(*Decoder).readValue(0xc510edc4e0, 0x0, 0x0, 0x20)
	/usr/local/go/src/encoding/json/stream.go:128 +0x198
encoding/json.(*Decoder).Decode(0xc510edc4e0, 0x18b76e0, 0xc510e75980, 0x7, 0xc50eac3980)
	/usr/local/go/src/encoding/json/stream.go:57 +0x8e
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/framer.(*jsonFrameReader).Read(0xc511dac3f0, 0xc50ee44800, 0x400, 0x400, 0x40, 0x38, 0x192fb80)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/util/framer/framer.go:150 +0x2b3
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc510eb49b0, 0x0, 0x26826c0, 0xc50ffed3c0, 0xc50eb336f8, 0x430972, 0xc50eb336c8, 0x12a9792, 0xc50eb33708)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/runtime/serializer/streaming/streaming.go:77 +0x85
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/watch/versioned.(*Decoder).Decode(0xc510e75960, 0x1be2148, 0x0, 0x0, 0x0, 0x0, 0xc50ff8e000)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/watch/versioned/decoder.go:48 +0xa1
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/watch.(*StreamWatcher).receive(0xc511dac420)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/watch/streamwatcher.go:93 +0xf4
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/watch.NewStreamWatcher
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/pkg/watch/streamwatcher.go:60 +0xd5
goroutine 214 [IO wait]:
net.runtime_pollWait(0x7f05b78950c0, 0x72, 0x2c)
	/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc50faa8220, 0x72, 0xc50f45d9d0, 0xc420010150)
	/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc50faa8220, 0x2686580, 0xc420010150)
	/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc50faa81c0, 0xc50fe08000, 0x1000, 0x1000, 0x0, 0x2686580, 0xc420010150)
	/usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc50fdac030, 0xc50fe08000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc50fd9c000, 0xc50fe08000, 0x1000, 0x1000, 0x6edfa0, 0xc50f45db58, 0x40a89d)
	/usr/local/go/src/net/http/transport.go:1261 +0x154
bufio.(*Reader).fill(0xc50f47e7e0)
	/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc50f47e7e0, 0x1, 0xc50f45dbbd, 0x1, 0x0, 0xc50f47e840, 0x0)
	/usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc50fd9c000)
	/usr/local/go/src/net/http/transport.go:1418 +0x1a1
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1062 +0x4e9
goroutine 785 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fafe380, 0x7f05b79b2028, 0xc50ea98ac0, 0xc50ea8ed20)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 695 [semacquire]:
sync.runtime_Semacquire(0xc50f58c3fc)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50f58c3f0)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50f5604b0, 0x7f05b79b2028, 0xc50f55a6c0, 0xc50f5965a0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50f5604b0, 0x7f05b79b2028, 0xc50f55a6c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f55a6c0, 0xc420690310, 0xc50f596570)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 215 [select]:
net/http.(*persistConn).writeLoop(0xc50fd9c000)
	/usr/local/go/src/net/http/transport.go:1646 +0x3bd
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1063 +0x50e
goroutine 285 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.LabelSet.Clone(0xc51ae781e0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/labelset.go:111 +0x127
github.com/prometheus/prometheus/retrieval.populateLabels(0xc51ae781e0, 0xc42040d440, 0xc51008bb48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:331 +0x393
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50f6b44b0, 0xc42040d440, 0x1, 0x1, 0xc51092ef00, 0x1, 0x5f)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50ea8a900, 0xc519be6320, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50eaa0640)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50eaa0640, 0x7f05b79b2028, 0xc50ea94400)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50ea94400, 0xc420690310, 0xc50f486420)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 286 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa38310, 0x7f05b79b2028, 0xc50ea94700, 0xc50ea8aae0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 287 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f470330, 0xc50ea8aae0, 0xc50eaa0640, 0xc50f470350, 0xc50f470328, 0x5, 0x267d740, 0xc50fa38310)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 288 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa38230, 0x7f05b79b2028, 0xc50ea94700, 0xc50ea8ab40)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 753 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f470330, 0xc50ea8ab40, 0xc50eaa0640, 0xc50f470350, 0xc50f470268, 0x5, 0x267d740, 0xc50fa38230)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 754 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa382a0, 0x7f05b79b2028, 0xc50ea94700, 0xc50ea8aba0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 755 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f470330, 0xc50ea8aba0, 0xc50eaa0640, 0xc50f470350, 0xc50f4702c8, 0x5, 0x267d740, 0xc50fa382a0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 333 [runnable]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh.func2(0xc5147c4900, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:192 +0x8b0
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc5110afcd8, 0x0, 0x0, 0xc5110afc88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26281 +0x132
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50ea748c0, 0xc50f486a50, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50ea748c0, 0x7f05b79b2028, 0xc50ff701c0, 0xc50eaf3800)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 620 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc50eac7dc0, 0x2540be400, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:378 +0x795
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:247 +0x3a4
goroutine 317 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.LabelSet.Clone(0xc51aaeecf0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/labelset.go:111 +0x127
github.com/prometheus/prometheus/retrieval.populateLabels(0xc51aaeecf0, 0xc42052f680, 0xc5100e9b48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:331 +0x393
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50eaf9470, 0xc42052f680, 0x27309c0, 0x2731c20, 0x0, 0x7f05b7a00d50, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50fb170e0, 0xc519edfa00, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50fa8e690)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50fa8e690, 0x7f05b79b2028, 0xc50fae8840)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50fae8840, 0xc420690310, 0xc50f4a09f0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 786 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9e420, 0xc50ea8ed20, 0xc50fa8e550, 0xc50ea9e440, 0xc50f48e3e0, 0x5, 0x267d740, 0xc50fafe380)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 787 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fafe3f0, 0x7f05b79b2028, 0xc50ea98ac0, 0xc50ea8ed80)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 788 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9e420, 0xc50ea8ed80, 0xc50fa8e550, 0xc50ea9e440, 0xc50f48e440, 0x5, 0x267d740, 0xc50fafe3f0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 789 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fafe460, 0x7f05b79b2028, 0xc50ea98ac0, 0xc50ea8ede0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 790 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9e420, 0xc50ea8ede0, 0xc50fa8e550, 0xc50ea9e440, 0xc50f48e4a0, 0x5, 0x267d740, 0xc50fafe460)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 318 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fafe4d0, 0x7f05b79b2028, 0xc50fae8b00, 0xc50fb172c0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 319 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f48e5f0, 0xc50fb172c0, 0xc50fa8e690, 0xc50f48e610, 0xc50f48e520, 0x5, 0x267d740, 0xc50fafe4d0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 320 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fafe540, 0x7f05b79b2028, 0xc50fae8b00, 0xc50fb17320)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 801 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f48e5f0, 0xc50fb17320, 0xc50fa8e690, 0xc50f48e610, 0xc50f48e580, 0x5, 0x267d740, 0xc50fafe540)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 802 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fafe5b0, 0x7f05b79b2028, 0xc50fae8b00, 0xc50fb17380)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 803 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f48e5f0, 0xc50fb17380, 0xc50fa8e690, 0xc50f48e610, 0xc50f48e5e0, 0x5, 0x267d740, 0xc50fafe5b0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 217 [runnable]:
strings.Join(0xc51a3eec00, 0x3, 0x4, 0x1ae957b, 0x1, 0xc51ae18c70, 0x10)
	/usr/local/go/src/strings/strings.go:362 +0x286
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh.func2(0xc510c17300, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:180 +0x8f4
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc510f67cd8, 0x0, 0x0, 0xc510f67c88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26281 +0x132
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50fa80460, 0xc50f4e2a80, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa80460, 0x7f05b79b2028, 0xc50f48ac40, 0xc50f47f080)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 218 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f4d6530, 0xc50f47f080, 0xc50f4907d0, 0xc50f4d6550, 0xc50f4d6468, 0x5, 0x267d740, 0xc50fa80460)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 219 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa804d0, 0x7f05b79b2028, 0xc50f48ac40, 0xc50f47f0e0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 220 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f4d6530, 0xc50f47f0e0, 0xc50f4907d0, 0xc50f4d6550, 0xc50f4d64c8, 0x5, 0x267d740, 0xc50fa804d0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 221 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa80540, 0x7f05b79b2028, 0xc50f48ac40, 0xc50f47f140)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 222 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f4d6530, 0xc50f47f140, 0xc50f4907d0, 0xc50f4d6550, 0xc50f4d6528, 0x5, 0x267d740, 0xc50fa80540)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 791 [semacquire]:
sync.runtime_Semacquire(0xc50ea9e75c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50ea9e750)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50ea889b0, 0x7f05b79b2028, 0xc50ea98ec0, 0xc50ebeeb10)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50ea889b0, 0x7f05b79b2028, 0xc50ea98ec0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50ea98ec0, 0xc420690310, 0xc50ebeeae0)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 792 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50ebea540, 0x7f05b79b2028, 0xc50ea991c0, 0xc50ea8f4a0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 793 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9e730, 0xc50ea8f4a0, 0xc50ea889b0, 0xc50ea9e750, 0xc50ea9e668, 0x5, 0x267d740, 0xc50ebea540)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 794 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50ebea5b0, 0x7f05b79b2028, 0xc50ea991c0, 0xc50ea8f500)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 795 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9e730, 0xc50ea8f500, 0xc50ea889b0, 0xc50ea9e750, 0xc50ea9e6c8, 0x5, 0x267d740, 0xc50ebea5b0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 796 [semacquire]:
sync.runtime_Semacquire(0xc420c6416c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*Mutex).Lock(0xc420c64168)
	/usr/local/go/src/sync/mutex.go:85 +0xd0
net/http.(*http2pipe).Read(0xc420c64168, 0xc5180fa800, 0x200, 0x200, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:2717 +0x55
net/http.http2transportResponseBody.Read(0xc420c64140, 0xc5180fa800, 0x200, 0x200, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:6241 +0xd9
github.com/prometheus/prometheus/vendor/golang.org/x/oauth2.(*onEOFReader).Read(0xc51a4f3700, 0xc5180fa800, 0x200, 0x200, 0x0, 0x200, 0xc418503270)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/oauth2/transport.go:114 +0x55
encoding/json.(*Decoder).refill(0xc515f9b040, 0x1a0, 0x1a4b480)
	/usr/local/go/src/encoding/json/stream.go:152 +0xfa
encoding/json.(*Decoder).readValue(0xc515f9b040, 0x0, 0x0, 0x1a0)
	/usr/local/go/src/encoding/json/stream.go:128 +0x198
encoding/json.(*Decoder).Decode(0xc515f9b040, 0x174f540, 0xc50eaec108, 0x7f05b7499848, 0xc51a4f3700)
	/usr/local/go/src/encoding/json/stream.go:57 +0x8e
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Do(0xc510219cd8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26210 +0x377
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc510219cd8, 0x0, 0x0, 0xc510219c88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26277 +0x104
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50ebea620, 0xc50ebeede0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50ebea620, 0x7f05b79b2028, 0xc50ea991c0, 0xc50ea8f560)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 797 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50ea9e730, 0xc50ea8f560, 0xc50ea889b0, 0xc50ea9e750, 0xc50ea9e728, 0x5, 0x267d740, 0xc50ebea620)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 334 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50fda0400, 0xc50eaf3800, 0xc50fe00780, 0xc50fda0420, 0xc50fda0338, 0x5, 0x267d740, 0xc50ea748c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 335 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50ea74930, 0x7f05b79b2028, 0xc50ff701c0, 0xc50eaf3860)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 336 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50fda0400, 0xc50eaf3860, 0xc50fe00780, 0xc50fda0420, 0xc50fda0398, 0x5, 0x267d740, 0xc50ea74930)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 833 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50ea749a0, 0x7f05b79b2028, 0xc50ff701c0, 0xc50eaf38c0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 834 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50fda0400, 0xc50eaf38c0, 0xc50fe00780, 0xc50fda0420, 0xc50fda03f8, 0x5, 0x267d740, 0xc50ea749a0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 835 [semacquire]:
sync.runtime_Semacquire(0xc50fda059c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50fda0590)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50fe00a00, 0x7f05b79b2028, 0xc50ff70240, 0xc50faedd70)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50fe00a00, 0x7f05b79b2028, 0xc50ff70240)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50ff70240, 0xc420690310, 0xc50faedd40)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 836 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50ea74a80, 0x7f05b79b2028, 0xc50ff70540, 0xc50eaf3bc0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 838 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50ea74af0, 0x7f05b79b2028, 0xc50ff70540, 0xc50eaf3c20)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 839 [runnable]:
sync.(*WaitGroup).Done(0xc50fda0590)
	/usr/local/go/src/sync/waitgroup.go:99
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50fda0570, 0xc50eaf3c20, 0xc50fe00a00, 0xc50fda0590, 0xc50fda0568, 0x5, 0x267d740, 0xc50ea74af0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:261 +0x133
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 840 [runnable]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh.func2(0xc514c7e980, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:192 +0x8b0
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc510dfdcd8, 0x0, 0x0, 0xc510dfdc88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26281 +0x132
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50ea74a10, 0xc50eabc450, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50ea74a10, 0x7f05b79b2028, 0xc50ff70540, 0xc50eaf3c80)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 841 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50fda0570, 0xc50eaf3c80, 0xc50fe00a00, 0xc50fda0590, 0xc50fda04a8, 0x5, 0x267d740, 0xc50ea74a10)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 696 [runnable]:
github.com/prometheus/prometheus/vendor/golang.org/x/oauth2.(*onEOFReader).Read(0xc519a2d860, 0xc51af0be00, 0x40000, 0x40000, 0x3fe00, 0x7fe00, 0xc51107b9f8)
	/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/oauth2/transport.go:113
encoding/json.(*Decoder).refill(0xc519258680, 0x72, 0x0)
	/usr/local/go/src/encoding/json/stream.go:152 +0xfa
encoding/json.(*Decoder).readValue(0xc519258680, 0x0, 0x0, 0x1a0)
	/usr/local/go/src/encoding/json/stream.go:128 +0x198
encoding/json.(*Decoder).Decode(0xc519258680, 0x174f540, 0xc50ff280f8, 0x7f05b7499848, 0xc519a2d860)
	/usr/local/go/src/encoding/json/stream.go:57 +0x8e
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Do(0xc51107bcd8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26210 +0x377
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc51107bcd8, 0x0, 0x0, 0xc51107bc88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26277 +0x104
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50fa4e2a0, 0xc50f6a07b0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa4e2a0, 0x7f05b79b2028, 0xc50f55a9c0, 0xc50f552de0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 697 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f58c3d0, 0xc50f552de0, 0xc50f5604b0, 0xc50f58c3f0, 0xc50f58c308, 0x5, 0x267d740, 0xc50fa4e2a0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 698 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa4e310, 0x7f05b79b2028, 0xc50f55a9c0, 0xc50f552e40)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 699 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f58c3d0, 0xc50f552e40, 0xc50f5604b0, 0xc50f58c3f0, 0xc50f58c368, 0x5, 0x267d740, 0xc50fa4e310)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 700 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa4e380, 0x7f05b79b2028, 0xc50f55a9c0, 0xc50f552ea0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 701 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f58c3d0, 0xc50f552ea0, 0xc50f5604b0, 0xc50f58c3f0, 0xc50f58c3c8, 0x5, 0x267d740, 0xc50fa4e380)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 521 [runnable]:
github.com/prometheus/prometheus/retrieval.populateLabels(0xc51ab23020, 0xc420523d40, 0xc50ffc9b48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:321 +0xb14
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50fd52420, 0xc420523d40, 0x27309c0, 0xc4204c2000, 0x0, 0x7f05b7a056b0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50facd200, 0xc51a4ba8c0, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50fa12640)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50fa12640, 0x7f05b79b2028, 0xc50fab0880)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50fab0880, 0xc420690310, 0xc50f6b4870)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 1062 [select]:
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*processorListener).run(0xc510cf4e00, 0xc50ea8e240)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:391 +0x3f1
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*sharedIndexInformer).AddEventHandler
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:257 +0x24b
goroutine 522 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fabc460, 0x7f05b79b2028, 0xc50fab0b80, 0xc50facd3e0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 523 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f69e520, 0xc50facd3e0, 0xc50fa12640, 0xc50f69e540, 0xc50f69e458, 0x5, 0x267d740, 0xc50fabc460)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 524 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fabc4d0, 0x7f05b79b2028, 0xc50fab0b80, 0xc50facd440)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 525 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f69e520, 0xc50facd440, 0xc50fa12640, 0xc50f69e540, 0xc50f69e4b8, 0x5, 0x267d740, 0xc50fabc4d0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 526 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fabc540, 0x7f05b79b2028, 0xc50fab0b80, 0xc50facd4a0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 527 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f69e520, 0xc50facd4a0, 0xc50fa12640, 0xc50f69e540, 0xc50f69e518, 0x5, 0x267d740, 0xc50fabc540)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 528 [runnable]:
regexp.(*inputString).context(0xc51ac65f38, 0x0, 0x0)
	/usr/local/go/src/regexp/regexp.go:314
regexp.(*machine).tryBacktrack(0xc51ac65e60, 0xc51a949500, 0x2696920, 0xc51ac65f38, 0x1, 0x0, 0x416bb8)
	/usr/local/go/src/regexp/backtrack.go:258 +0x491
regexp.(*machine).backtrack(0xc51ac65e60, 0x2696920, 0xc51ac65f38, 0x0, 0x87, 0x0, 0x0)
	/usr/local/go/src/regexp/backtrack.go:333 +0x2b8
regexp.(*Regexp).doExecute(0xc420223400, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc51a6e0000, 0x87, 0x0, 0x0, ...)
	/usr/local/go/src/regexp/exec.go:437 +0x2e5
regexp.(*Regexp).MatchString(0xc420223400, 0xc51a6e0000, 0x87, 0x4)
	/usr/local/go/src/regexp/regexp.go:416 +0x88
github.com/prometheus/prometheus/relabel.relabel(0xc51ad8d3b0, 0xc4201b1180, 0xc51ad8d3b0)
	/go/src/github.com/prometheus/prometheus/relabel/relabel.go:53 +0xfcb
github.com/prometheus/prometheus/relabel.Process(0xc51ad8d3b0, 0xc420bbade0, 0x3, 0x3, 0x2752520)
	/go/src/github.com/prometheus/prometheus/relabel/relabel.go:32 +0x57
github.com/prometheus/prometheus/retrieval.populateLabels(0xc51ad8d3b0, 0xc420519200, 0xc50fb4fb48, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:332 +0x3da
github.com/prometheus/prometheus/retrieval.targetsFromGroup(0xc50f486d80, 0xc420519200, 0x27309c0, 0xc4204e2000, 0x0, 0x7f05b7a03780, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:394 +0x1b1
github.com/prometheus/prometheus/retrieval.(*scrapePool).Sync(0xc50facd5c0, 0xc51ab86fa0, 0x3, 0x4)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:208 +0xb1
github.com/prometheus/prometheus/discovery.(*TargetSet).sync(0xc50fa128c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:211 +0x1b5
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50fa128c0, 0x7f05b79b2028, 0xc50fab0c00)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:196 +0x26c
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50fab0c00, 0xc420690310, 0xc50f6b4c30)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 849 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fabc5b0, 0x7f05b79b2028, 0xc50fab0f00, 0xc50facd7a0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 850 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f69e690, 0xc50facd7a0, 0xc50fa128c0, 0xc50f69e6b0, 0xc50f69e5c8, 0x5, 0x267d740, 0xc50fabc5b0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 851 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fabc620, 0x7f05b79b2028, 0xc50fab0f00, 0xc50facd800)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 852 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f69e690, 0xc50facd800, 0xc50fa128c0, 0xc50f69e6b0, 0xc50f69e628, 0x5, 0x267d740, 0xc50fabc620)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 853 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fabc690, 0x7f05b79b2028, 0xc50fab0f00, 0xc50facd860)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 854 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f69e690, 0xc50facd860, 0xc50fa128c0, 0xc50f69e6b0, 0xc50f69e688, 0x5, 0x267d740, 0xc50fabc690)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 1063 [semacquire]:
sync.runtime_notifyListWait(0xc510cf4e28, 0x0)
	/usr/local/go/src/runtime/sema.go:267 +0x122
sync.(*Cond).Wait(0xc510cf4e18)
	/usr/local/go/src/sync/cond.go:57 +0x80
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*processorListener).pop.func1(0x0, 0x0, 0xc50ffd7700)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:365 +0xd3
github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*processorListener).pop(0xc510cf4e00, 0xc50ea8e240)
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:373 +0xcf
created by github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache.(*sharedIndexInformer).AddEventHandler
	/go/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/1.5/tools/cache/shared_informer.go:258 +0x27e
goroutine 866 [semacquire]:
sync.runtime_Semacquire(0xc50f4d686c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc50f4d6860)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders(0xc50f490e60, 0x7f05b79b2028, 0xc50f48b000, 0xc50f4e2e70)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:284 +0x358
github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc50f490e60, 0x7f05b79b2028, 0xc50f48b000)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:188 +0x175
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0x7f05b79b2028, 0xc50f48b000, 0xc420690310, 0xc50f4e2e40)
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:114 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
	/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:117 +0x63c
goroutine 867 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa80930, 0x7f05b79b2028, 0xc50f48b2c0, 0xc50f47fbc0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 868 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f4d6840, 0xc50f47fbc0, 0xc50f490e60, 0xc50f4d6860, 0xc50f4d6838, 0x5, 0x267d740, 0xc50fa80930)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 869 [runnable]:
encoding/json.(*decodeState).scanWhile(0xc50fdf7208, 0x9, 0xc51afc4058)
	/usr/local/go/src/encoding/json/decode.go:309
encoding/json.(*decodeState).object(0xc50fdf7208, 0x18779c0, 0xc519b8cc10, 0x196)
	/usr/local/go/src/encoding/json/decode.go:737 +0x48b
encoding/json.(*decodeState).value(0xc50fdf7208, 0x18779c0, 0xc519b8cc10, 0x196)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc
encoding/json.(*decodeState).array(0xc50fdf7208, 0x17943a0, 0xc5161b9990, 0x197)
	/usr/local/go/src/encoding/json/decode.go:519 +0x1e1
encoding/json.(*decodeState).value(0xc50fdf7208, 0x17943a0, 0xc5161b9990, 0x197)
	/usr/local/go/src/encoding/json/decode.go:365 +0x25f
encoding/json.(*decodeState).object(0xc50fdf7208, 0x174f540, 0xc50fdd20d8, 0x16)
	/usr/local/go/src/encoding/json/decode.go:696 +0x1080
encoding/json.(*decodeState).value(0xc50fdf7208, 0x174f540, 0xc50fdd20d8, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc
encoding/json.(*decodeState).unmarshal(0xc50fdf7208, 0x174f540, 0xc50fdd20d8, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa
encoding/json.(*Decoder).Decode(0xc50fdf71e0, 0x174f540, 0xc50fdd20d8, 0x7f05b7499848, 0xc5164b3900)
	/usr/local/go/src/encoding/json/stream.go:67 +0x15a
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Do(0xc5110e1cd8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26210 +0x377
github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1.(*InstancesListCall).Pages(0xc5110e1cd8, 0x0, 0x0, 0xc5110e1c88, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/google.golang.org/api/compute/v1/compute-gen.go:26277 +0x104
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).refresh(0xc50fa80850, 0xc50f4e49f0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:205 +0x395
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa80850, 0x7f05b79b2028, 0xc50f48b2c0, 0xc50f47fc20)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:110 +0x50
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 879 [runnable]:
sync.(*Cond).Signal(0xc420c64170)
	/usr/local/go/src/sync/cond.go:65
net/http.(*http2pipe).Write(0xc420c64168, 0xc5125d8000, 0x2004, 0x4000, 0x2004, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:2754 +0x148
net/http.(*http2clientConnReadLoop).processData(0xc50ff4ef80, 0xc51ab85c80, 0x0, 0xc50ff4eeb0)
	/usr/local/go/src/net/http/h2_bundle.go:6377 +0x2e0
net/http.(*http2clientConnReadLoop).run(0xc50ff4ef80, 0x1be2718, 0xc50ffd3790)
	/usr/local/go/src/net/http/h2_bundle.go:6038 +0x510
net/http.(*http2ClientConn).readLoop(0xc510b2a340)
	/usr/local/go/src/net/http/h2_bundle.go:5937 +0xa6
created by net/http.(*http2Transport).newClientConn
	/usr/local/go/src/net/http/h2_bundle.go:5314 +0x709
goroutine 750 [select]:
net/http.(*persistConn).writeLoop(0xc420a98e00)
	/usr/local/go/src/net/http/transport.go:1646 +0x3bd
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1063 +0x50e
goroutine 749 [IO wait]:
net.runtime_pollWait(0x7f05b06ec9b8, 0x72, 0x62)
	/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc50eac7560, 0x72, 0xc50ff0b9d0, 0xc420010150)
	/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc50eac7560, 0x2686580, 0xc420010150)
	/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc50eac7500, 0xc510104000, 0x1000, 0x1000, 0x0, 0x2686580, 0xc420010150)
	/usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc50fdd20a0, 0xc510104000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc420a98e00, 0xc510104000, 0x1000, 0x1000, 0x6e5db3, 0xc50ff6cb78, 0xc50ff6cb88)
	/usr/local/go/src/net/http/transport.go:1261 +0x154
bufio.(*Reader).fill(0xc50fa579e0)
	/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc50fa579e0, 0x1, 0xc50fa57a40, 0xc50ff6cc60, 0x434293, 0x1a596e0, 0x414c7c)
	/usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc420a98e00)
	/usr/local/go/src/net/http/transport.go:1418 +0x1a1
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1062 +0x4e9
goroutine 870 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f4d6840, 0xc50f47fc20, 0xc50f490e60, 0xc50f4d6860, 0xc50f4d6778, 0x5, 0x267d740, 0xc50fa80850)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:245 +0x47e
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325
goroutine 871 [select]:
github.com/prometheus/prometheus/discovery/gce.(*GCEDiscovery).Run(0xc50fa808c0, 0x7f05b79b2028, 0xc50f48b2c0, 0xc50f47fc80)
	/go/src/github.com/prometheus/prometheus/discovery/gce/gce.go:124 +0x3c9
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4
goroutine 872 [select]:
github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc50f4d6840, 0xc50f47fc80, 0xc50f490e60, 0xc50f4d6860, 0xc50f4d67d8, 0x5, 0x267d740, 0xc50fa808c0)
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:265 +0x2d3
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:279 +0x325

AFAICS in the current release-1.5 branch, this is caused by an unguarded write to the target groups map invoked here:

prometheus/discovery/discovery.go

Line 255 in bd1182d

ts.setTargetGroup(name, tgroup)

And finally done here:

prometheus/discovery/discovery.go

Line 310 in bd1182d

ts.tgroups[name+"/"+tg.Source] = tg

Sadly the comments in the TargetSet struct definition don't state what the mutex guards (only the map vs. the whole struct).
If it only guards the map, it would be possible to move the locking closer to the actual map write inside of TargetSet.setTargetGroup instead of doing it before all invocations of TargetSet.setTargetGroup. Probably would make it easier for the caller.

The bug also exists in the master branch.

@gouthamve gouthamve referenced this issue Mar 6, 2017

Merged

Fix Map Race by Moving Locking closer to the Write #2476

@fabxc fabxc closed this in #2476 Apr 7, 2017

@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 23, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 23, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.