Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash log #1367

Closed
RichiH opened this Issue Feb 3, 2016 · 4 comments

Comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@RichiH @fabxc
@RichiH
Copy link
Member

RichiH commented Feb 3, 2016

And another one, this one at 308d447

net/http.noteEOFReader.Read(0x7feb4efd4030, 0xc820035b70, 0xc870f76e18, 0xc91fe1c000, 0x1000, 0x1000, 0xc8f2606d80, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc89a58b140, 0xc91fe1c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8f2606c60)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8f2606c60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc870f76dc0)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 1313371 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc835fe0480, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1313176 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8363b27e0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:302 +0x780

goroutine 1370752 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc82c3fc870, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39120761 [IO wait]:
net.runtime_pollWait(0x7feb4ef8c3e0, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc87f904a70, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc87f904a70, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc87f904a10, 0xc8d762f000, 0x1000, 0x1000, 0x0, 0x7feb6e9f7050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8ee1b37e0, 0xc8d762f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7feb4efd4030, 0xc8ee1b37e0, 0xc8237a35a8, 0xc8d762f000, 0x1000, 0x1000, 0xc8e4122480, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc8fb383600, 0xc8d762f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8e4122360)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8e4122360, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8237a3550)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 1371686 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc82c03e2d0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285957 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8aad59440, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39121327 [select]:
net/http.(*persistConn).writeLoop(0xc8cf51b810)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 1332324 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc834dacea0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1332032 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc835620f30, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1408356 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8ed653200, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285527 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8f172b170, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1313379 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc835fe1170, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39121824 [select]:
net/http.(*persistConn).writeLoop(0xc8dd1b1e40)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 1332033 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8356210e0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39117116 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc8e5ad7ad0)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 39117218 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc8e0dc22c0)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 1332332 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc834dadc20, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285645 [select]:
net/http.(*persistConn).roundTrip(0xc8bd3e74a0, 0xc8df510210, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc88a7e3290, 0xc8bb19e540, 0xc800000000, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:235 +0x530
net/http.send(0xc8bb19e540, 0x7feb6e9fb560, 0xc88a7e3290, 0xc8391f6ac0, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xc881a89da0, 0xc8bb19e540, 0x3e, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xc881a89da0, 0xc8bb19e540, 0xfb9e88, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0xc881a89da0, 0xc8bb19e540, 0x6, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:175 +0x9c
github.com/prometheus/prometheus/retrieval.(*Target).scrape(0xc88a7e3200, 0x7feb4efd3f80, 0xc8c57125a0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:444 +0x428
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc88a7e3200, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:364 +0x623
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460345 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc896e6ddd0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285656 [select]:
net/http.(*persistConn).roundTrip(0xc8b296c370, 0xc8bdd384a0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc88a7e3ef0, 0xc87bba81c0, 0xc800000000, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:235 +0x530
net/http.send(0xc87bba81c0, 0x7feb6e9fb560, 0xc88a7e3ef0, 0xc8396bfcc0, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xc8c5d66ff0, 0xc87bba81c0, 0x3f, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xc8c5d66ff0, 0xc87bba81c0, 0xfb9e88, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0xc8c5d66ff0, 0xc87bba81c0, 0x6, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:175 +0x9c
github.com/prometheus/prometheus/retrieval.(*Target).scrape(0xc88a7e3e60, 0x7feb4efd3f80, 0xc896cd66e0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:444 +0x428
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc88a7e3e60, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:364 +0x623
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39120287 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc8dcac2bb0)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 39121766 [select]:
net/http.(*persistConn).writeLoop(0xc87dc1bef0)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 1285529 [select, 1 minutes]:
net/http.(*persistConn).roundTrip(0xc8e5ad7ad0, 0xc8a0a52a80, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc8f172b4d0, 0xc8a52e8460, 0xc800000000, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:235 +0x530
net/http.send(0xc8a52e8460, 0x7feb6e9fb560, 0xc8f172b4d0, 0xc830459600, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xc8b60bb320, 0xc8a52e8460, 0x38, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xc8b60bb320, 0xc8a52e8460, 0xfb9e88, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0xc8b60bb320, 0xc8a52e8460, 0x6, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:175 +0x9c
github.com/prometheus/prometheus/retrieval.(*Target).scrape(0xc8f172b440, 0x7feb4efd3f80, 0xc87ab03200, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:444 +0x428
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8f172b440, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:364 +0x623
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285524 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8f172ad80, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285818 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8890b30e0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1271250 [select, 3053 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8662c7900, 0xc8fa5729c0, 0xc8fa572660)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:386 +0x2b8
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 1271245 [select, 3052 minutes]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).handleUpdates(0xc820140640, 0xc8fa572d80, 0xc8fa572660)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:174 +0x36f
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:154 +0x5ac

goroutine 1332309 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8356215f0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1332346 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc834c2b200, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285423 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8f172a7e0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39117115 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7feb4db1ad68, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc89cb3bb10, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc89cb3bb10, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc89cb3bab0, 0xc8e80a7000, 0x1000, 0x1000, 0x0, 0x7feb6e9f7050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8d8f38ff0, 0xc8e80a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7feb4efd4030, 0xc8d8f38ff0, 0xc8e5ad7b28, 0xc8e80a7000, 0x1000, 0x1000, 0xc8f1b1ae40, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc87ab032a0, 0xc8e80a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc87ab04420)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc87ab04420, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8e5ad7ad0)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 1283792 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8a70acbd0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460258 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8dc3afa70, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285420 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8f172a480, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460319 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc896e6c090, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460335 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc896e6d290, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460287 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc87f87db90, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460352 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc87c7c0630, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460291 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc892efc090, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460384 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8df982ab0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285658 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc89f03c120, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285981 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8b0e15440, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39120289 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7feb4c94be40, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8ccef28b0, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8ccef28b0, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8ccef2850, 0xc8b28c5000, 0x1000, 0x1000, 0x0, 0x7feb6e9f7050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8edebee50, 0xc8b28c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7feb4efd4030, 0xc8edebee50, 0xc8a2e33c88, 0xc8b28c5000, 0x1000, 0x1000, 0xc8d39b0d80, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc8d82bb3e0, 0xc8b28c5000, 0x1000, 0x1000, 0xc82006a900, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8d39b0c60)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8d39b0c60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8a2e33c30)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 1332313 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc835621b00, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285947 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc886dba900, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285948 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc886dbaab0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39116140 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc8e6d634a0)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 1271235 [chan receive, 3056 minutes]:
github.com/prometheus/prometheus/retrieval.merge.func1(0xc8fa572720)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:83 +0xc6
created by github.com/prometheus/prometheus/retrieval.merge
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:94 +0x151

goroutine 1460267 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc87f87c510, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460426 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8ce36da70, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39121470 [IO wait]:
net.runtime_pollWait(0x7feb4ef8d2e0, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc88e264990, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc88e264990, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc88e264930, 0xc882a7c000, 0x1000, 0x1000, 0x0, 0x7feb6e9f7050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc881e0ab20, 0xc882a7c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7feb4efd4030, 0xc881e0ab20, 0xc89797ed68, 0xc882a7c000, 0x1000, 0x1000, 0xc8c547b740, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc87b6f2ba0, 0xc882a7c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8715d68a0)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8715d68a0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc89797ed10)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 1460262 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8dc3afef0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1313145 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc836c20ea0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460306 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc892efd170, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460279 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc87f87d290, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39118352 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7feb4da4e538, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8e244af40, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8e244af40, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8e244aee0, 0xc8bdeb8000, 0x1000, 0x1000, 0x0, 0x7feb6e9f7050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8ee1b3238, 0xc8bdeb8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7feb4efd4030, 0xc8ee1b3238, 0xc8e2af0c08, 0xc8bdeb8000, 0x1000, 0x1000, 0xc894fbecc0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc8dc752080, 0xc8bdeb8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc929174840)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc929174840, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8e2af0bb0)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 1440445 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc88c06d320, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1313202 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8363b3680, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1418501 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8c6f8ba70, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1271117 [select, 3052 minutes]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func2(0xc8fa572960, 0xc8fa572900, 0xc86f8d6f00, 0xc8fa572660)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:141 +0x210
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:148 +0x4f2

goroutine 39122250 [select]:
net/http.(*persistConn).writeLoop(0xc8bd3e74a0)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 1460273 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc87f87cbd0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1271135 [chan receive, 3056 minutes]:
github.com/prometheus/prometheus/retrieval.(*StaticProvider).Run(0xc8cbe7cb20, 0xc8ffb4b7a0, 0xc8fa572660)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:552 +0x1dc
created by github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:383 +0xcc

goroutine 1332336 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc834c2a360, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1332317 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc834dac240, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1313368 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc835fe0090, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1440423 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8a209f170, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 20842191 [IO wait, 1498 minutes]:
net.runtime_pollWait(0x7feb4da4ed78, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8bdb84370, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8bdb84370, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8bdb84310, 0xc8f63dd000, 0x1000, 0x1000, 0x0, 0x7feb6e9f7050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc88bfb8d10, 0xc8f63dd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.(*liveSwitchReader).Read(0xc89797e728, 0xc8f63dd000, 0x1000, 0x1000, 0xc86ebdde20, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:219 +0xa4
io.(*LimitedReader).Read(0xc8d3863840, 0xc8f63dd000, 0x1000, 0x1000, 0xe00000000, 0x0, 0x0)
        /usr/local/go/src/io/io.go:427 +0xbd
bufio.(*Reader).fill(0xc89ac31c20)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadSlice(0xc89ac31c20, 0xc81bf5de0a, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:328 +0x21a
bufio.(*Reader).ReadLine(0xc89ac31c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:357 +0x53
net/textproto.(*Reader).readLineSlice(0xc8c9212ff0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/textproto/reader.go:55 +0x81
net/textproto.(*Reader).ReadLine(0xc8c9212ff0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/textproto/reader.go:36 +0x40
net/http.ReadRequest(0xc89ac31c20, 0xc8a1443180, 0x0, 0x0)
        /usr/local/go/src/net/http/request.go:653 +0xb6
net/http.(*conn).readRequest(0xc89797e6e0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:633 +0x32f
net/http.(*conn).serve(0xc89797e6e0)
        /usr/local/go/src/net/http/server.go:1319 +0x727
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:1910 +0x3f6

goroutine 1460436 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc89f48e630, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285654 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc88a7e3c20, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1371630 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc82c1f34d0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285978 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8b0e14cf0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460359 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc87c7c0e10, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460249 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8dc3af050, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460284 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc87f87d830, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1408486 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8f66fa090, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285650 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc88a7e37a0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460420 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8ce36d3b0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1440442 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc88c06cfc0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460295 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc892efc510, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1313179 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8363b2cf0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:302 +0x780

goroutine 1313370 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc835fe0360, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285640 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc88a7e2c60, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460257 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8dc3af950, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1271252 [select, 3056 minutes]:
github.com/prometheus/prometheus/retrieval.(*prefixedTargetProvider).Run(0xc8662deac0, 0xc8fa572840, 0xc8fa572660)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:386 +0x2b8
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func1
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:132 +0x51

goroutine 1285556 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8b8cd1b00, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1313377 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc835fe0ea0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285993 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc87462a510, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1332306 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc835621200, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39122392 [select]:
net/http.(*persistConn).writeLoop(0xc8a9eca8f0)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 1460376 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8df9821b0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460374 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc87c7c1ef0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460372 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc87c7c1cb0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1313184 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8363b3290, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:302 +0x780

goroutine 1285544 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc877b28d80, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39116374 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc8e7059130)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 39122296 [select]:
net/http.(*persistConn).writeLoop(0xc87a3b6370)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 1284975 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc877b282d0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39120710 [select]:
net/http.(*persistConn).writeLoop(0xc882fa9290)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 1285817 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8890b2fc0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285970 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8890b3b90, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285984 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8b0e15830, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1313400 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc836a6b050, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1285425 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8f172aa20, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1271115 [select, 3056 minutes]:
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run.func2(0xc8fa5727e0, 0xc8fa572780, 0xc86f8d6000, 0xc8fa572660)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:141 +0x210
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).Run
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:148 +0x4f2

goroutine 1460321 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc896e6c2d0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460323 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc896e6c510, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460325 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc896e6c750, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460327 [select, 1 minutes]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc896e6c990, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460315 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc892efdb90, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39122320 [IO wait]:
net.runtime_pollWait(0x7feb4ef8c4a0, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8edd71560, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8edd71560, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8edd71500, 0xc8b4dcd000, 0x1000, 0x1000, 0x0, 0x7feb6e9f7050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc87975b240, 0xc8b4dcd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7feb4efd4030, 0xc87975b240, 0xc8b29edd38, 0xc8b4dcd000, 0x1000, 0x1000, 0xc896993d40, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc87cff6de0, 0xc8b4dcd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc884e753e0)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc884e753e0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8b29edce0)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 39122391 [IO wait]:
net.runtime_pollWait(0x7feb36bcba80, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc897a19f70, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc897a19f70, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc897a19f10, 0xc91ad3e000, 0x1000, 0x1000, 0x0, 0x7feb6e9f7050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc88b713d48, 0xc91ad3e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7feb4efd4030, 0xc88b713d48, 0xc8a9eca948, 0xc91ad3e000, 0x1000, 0x1000, 0xc8dedc8600, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc89ea36d40, 0xc91ad3e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8dedc8480)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8dedc8480, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8a9eca8f0)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 39120098 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7feb4da50c48, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8787ad5d0, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8787ad5d0, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8787ad570, 0xc901389000, 0x1000, 0x1000, 0x0, 0x7feb6e9f7050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8702a1d38, 0xc901389000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7feb4efd4030, 0xc8702a1d38, 0xc8dd1b03c8, 0xc901389000, 0x1000, 0x1000, 0xc889395380, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc8cd23d860, 0xc901389000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc889395260)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc889395260, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8dd1b0370)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 39121169 [IO wait]:
net.runtime_pollWait(0x7feb4db1ba28, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8bef441b0, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8bef441b0, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8bef44150, 0xc90a0b5000, 0x1000, 0x1000, 0x0, 0x7feb6e9f7050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8c87154a0, 0xc90a0b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7feb4efd4030, 0xc8c87154a0, 0xc8dcfcf9c8, 0xc90a0b5000, 0x1000, 0x1000, 0xc8b7c70c60, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc929788c60, 0xc90a0b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8b7c70b40)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8b7c70b40, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8dcfcf970)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 1460404 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8ce36c1b0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460402 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8df983ef0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 1460406 [select]:
github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc8ce36c3f0, 0x7feb6c3b2a38, 0xc820146b80)
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/target.go:342 +0x673
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
        /home/gocode/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:285 +0x5d6

goroutine 39118423 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7feb4da4f1f8, 0x72, 0xc8200101e0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8ac547410, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8ac547410, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8ac5473b0, 0xc87c2c0000, 0x1000, 0x1000, 0x0, 0x7feb6e9f7050, 0xc8200101e0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc896e39828, 0xc87c2c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7feb4efd4030, 0xc896e39828, 0xc871d04aa8, 0xc87c2c0000, 0x1000, 0x1000, 0xc8fa2c8480, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc926f35780, 0xc87c2c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8fa2c8360)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8fa2c8360, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc871d04a50)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 39120030 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc8e60c53f0)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 39118424 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc871d04a50)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 39117401 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc882fa8000)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 39120636 [select]:
net/http.(*persistConn).writeLoop(0xc882fa8fd0)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Feb 3, 2016

This doesn't seem complete. I see all goroutines but there should be a panic message in the beginning.
@RichiH

This comment has been minimized.

Sign in to view
Copy link
Member Author

RichiH commented Feb 3, 2016

That's what my coworker had in his screen session when prometheus died. I didn't collect that info myself, and there's nothing more I could offer.

I really need to migrate this instance to systemd so I can collect full info.

If you don't have enough info to go on in this report feel free to close.
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Feb 3, 2016

Sorry, without the top lines, this unfortunately doesn't tell us anything about what happened.

@fabxc fabxc closed this Feb 3, 2016

@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 24, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.