Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data races in scrape package. #3994

Closed
beorn7 opened this Issue Mar 21, 2018 · 11 comments

Comments

Assignees
No one assigned
Labels
component/scraping kind/bug priority/P1
Projects
None yet
Milestone
No milestone
3 participants
@beorn7 @krasi-georgiev @juliusv
@beorn7
Copy link
Member

beorn7 commented Mar 21, 2018

What did you do?

Ran Prometheus with race detection enabled.

What did you expect to see?

No data races.

What did you see instead? Under which circumstances?

Several data races in the scrape package. Most likely when hitting the web UI (targets page, SD page). See logs below.

Environment

  • System information:

    Linux 4.15.3+soundcloud2 x86_64

  • Prometheus version:

    v2.2.1

  • Logs:

2018-03-20_19:08:10.24243 ==================
2018-03-20_19:08:10.24249 WARNING: DATA RACE
2018-03-20_19:08:10.24250 Write at 0x00c426fcf950 by goroutine 109:
2018-03-20_19:08:10.24250   github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
2018-03-20_19:08:10.24251       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/target.go:120 +0x3ae
2018-03-20_19:08:10.24251   github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
2018-03-20_19:08:10.24252       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/scrape.go:270 +0x80a
2018-03-20_19:08:10.24252   github.com/prometheus/prometheus/scrape.(*Manager).reload()
2018-03-20_19:08:10.24252       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/manager.go:164 +0x241
2018-03-20_19:08:10.24253   github.com/prometheus/prometheus/scrape.(*Manager).Run()
2018-03-20_19:08:10.24253       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/manager.go:62 +0x5c
2018-03-20_19:08:10.24254   main.main.func12()
2018-03-20_19:08:10.24254       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:425 +0xc7
2018-03-20_19:08:10.24254   github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1()
2018-03-20_19:08:10.24255       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x34
2018-03-20_19:08:10.24256 
2018-03-20_19:08:10.24257 Previous read at 0x00c426fcf950 by goroutine 296:
2018-03-20_19:08:10.24257   [failed to restore the stack]
2018-03-20_19:08:10.24257 
2018-03-20_19:08:10.24258 Goroutine 109 (running) created at:
2018-03-20_19:08:10.24258   github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run()
2018-03-20_19:08:10.24258       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0x10b
2018-03-20_19:08:10.24259   main.main()
2018-03-20_19:08:10.24259       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:581 +0x95b2
2018-03-20_19:08:10.24260 
2018-03-20_19:08:10.24260 Goroutine 296 (running) created at:
2018-03-20_19:08:10.24260   net/http.(*Server).Serve()
2018-03-20_19:08:10.24261       /usr/local/go/src/net/http/server.go:2795 +0x364
2018-03-20_19:08:10.24263   github.com/prometheus/prometheus/web.(*Handler).Run.func5()
2018-03-20_19:08:10.24263       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:455 +0x53
2018-03-20_19:08:10.24264 ==================

2018-03-20_18:53:58.62334 ==================
2018-03-20_18:53:58.62336 WARNING: DATA RACE
2018-03-20_18:53:58.62337 Read at 0x00c421f3d0e0 by goroutine 199:
2018-03-20_18:53:58.62337   github.com/prometheus/prometheus/scrape.(*Target).DiscoveredLabels()
2018-03-20_18:53:58.62338       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/target.go:113 +0x3c
2018-03-20_18:53:58.62338   runtime.call32()
2018-03-20_18:53:58.62338       /usr/local/go/src/runtime/asm_amd64.s:573 +0x3a
2018-03-20_18:53:58.62340   reflect.Value.Call()
2018-03-20_18:53:58.62340       /usr/local/go/src/reflect/value.go:308 +0xc0
2018-03-20_18:53:58.62341   text/template.(*state).evalCall()
2018-03-20_18:53:58.62341       /usr/local/go/src/text/template/exec.go:667 +0x667
2018-03-20_18:53:58.62341   text/template.(*state).evalField()
2018-03-20_18:53:58.62342       /usr/local/go/src/text/template/exec.go:557 +0xe4a
2018-03-20_18:53:58.62342   text/template.(*state).evalFieldChain()
2018-03-20_18:53:58.62342       /usr/local/go/src/text/template/exec.go:522 +0x174
2018-03-20_18:53:58.62345   text/template.(*state).evalFieldNode()
2018-03-20_18:53:58.62345       /usr/local/go/src/text/template/exec.go:489 +0x161
2018-03-20_18:53:58.62345   text/template.(*state).evalCommand()
2018-03-20_18:53:58.62346       /usr/local/go/src/text/template/exec.go:427 +0x903
2018-03-20_18:53:58.62346   text/template.(*state).evalPipeline()
2018-03-20_18:53:58.62346       /usr/local/go/src/text/template/exec.go:405 +0x1a8
2018-03-20_18:53:58.62347   text/template.(*state).walkRange()
2018-03-20_18:53:58.62347       /usr/local/go/src/text/template/exec.go:314 +0x126
2018-03-20_18:53:58.62347   text/template.(*state).walk()
2018-03-20_18:53:58.62348       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-03-20_18:53:58.62348   text/template.(*state).walk()
2018-03-20_18:53:58.62349       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-03-20_18:53:58.62349   text/template.(*state).walkRange.func1()
2018-03-20_18:53:58.62349       /usr/local/go/src/text/template/exec.go:326 +0x139
2018-03-20_18:53:58.62350   text/template.(*state).walkRange()
2018-03-20_18:53:58.62351       /usr/local/go/src/text/template/exec.go:335 +0x424
2018-03-20_18:53:58.62351   text/template.(*state).walk()
2018-03-20_18:53:58.62351       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-03-20_18:53:58.62352   text/template.(*state).walk()
2018-03-20_18:53:58.62352       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-03-20_18:53:58.62352   text/template.(*state).walkRange.func1()
2018-03-20_18:53:58.62353       /usr/local/go/src/text/template/exec.go:326 +0x139
2018-03-20_18:53:58.62353   text/template.(*state).walkRange()
2018-03-20_18:53:58.62354       /usr/local/go/src/text/template/exec.go:343 +0x6ed
2018-03-20_18:53:58.62355   text/template.(*state).walk()
2018-03-20_18:53:58.62355       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-03-20_18:53:58.62356   text/template.(*state).walk()
2018-03-20_18:53:58.62356       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-03-20_18:53:58.62357   text/template.(*state).walkTemplate()
2018-03-20_18:53:58.62357       /usr/local/go/src/text/template/exec.go:388 +0x398
2018-03-20_18:53:58.62358   text/template.(*state).walk()
2018-03-20_18:53:58.62358       /usr/local/go/src/text/template/exec.go:244 +0x2ef
2018-03-20_18:53:58.62359   text/template.(*state).walk()
2018-03-20_18:53:58.62360       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-03-20_18:53:58.62360   text/template.(*Template).execute()
2018-03-20_18:53:58.62361       /usr/local/go/src/text/template/exec.go:194 +0x3cf
2018-03-20_18:53:58.62361   text/template.(*Template).Execute()
2018-03-20_18:53:58.62363       /usr/local/go/src/text/template/exec.go:177 +0x64
2018-03-20_18:53:58.62364   html/template.(*Template).Execute()
2018-03-20_18:53:58.62364       /usr/local/go/src/html/template/template.go:122 +0xd0
2018-03-20_18:53:58.62365   github.com/prometheus/prometheus/template.Expander.ExpandHTML()
2018-03-20_18:53:58.62366       /home/bjoern/gocode/src/github.com/prometheus/prometheus/template/template.go:312 +0x5f3
2018-03-20_18:53:58.62366   github.com/prometheus/prometheus/web.(*Handler).executeTemplate()
2018-03-20_18:53:58.62367       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:779 +0x387
2018-03-20_18:53:58.62367   github.com/prometheus/prometheus/web.(*Handler).targets()
2018-03-20_18:53:58.62368       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:624 +0x4c7
2018-03-20_18:53:58.62369   github.com/prometheus/prometheus/web.(*Handler).(github.com/prometheus/prometheus/web.targets)-fm()
2018-03-20_18:53:58.62369       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:219 +0x5f
2018-03-20_18:53:58.62370   github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1()
2018-03-20_18:53:58.62371       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/http.go:309 +0x2a2
2018-03-20_18:53:58.62373   github.com/prometheus/prometheus/web.(*Handler).testReady.func1()
2018-03-20_18:53:58.62374       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:357 +0x81
2018-03-20_18:53:58.62375   github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).handle.func1()
2018-03-20_18:53:58.62376       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:50 +0x2db
2018-03-20_18:53:58.62376   github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP()
2018-03-20_18:53:58.62377       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter/router.go:299 +0xbad
2018-03-20_18:53:58.62378   github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP()
2018-03-20_18:53:58.62379       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:88 +0x67
2018-03-20_18:53:58.62379   net/http.(*ServeMux).ServeHTTP()
2018-03-20_18:53:58.62380       /usr/local/go/src/net/http/server.go:2337 +0x9f
2018-03-20_18:53:58.62381   github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2()
2018-03-20_18:53:58.62382       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x543
2018-03-20_18:53:58.62383   net/http.HandlerFunc.ServeHTTP()
2018-03-20_18:53:58.62384       /usr/local/go/src/net/http/server.go:1947 +0x51
2018-03-20_18:53:58.62384   net/http.serverHandler.ServeHTTP()
2018-03-20_18:53:58.62385       /usr/local/go/src/net/http/server.go:2694 +0xb9
2018-03-20_18:53:58.62386   net/http.(*conn).serve()
2018-03-20_18:53:58.62386       /usr/local/go/src/net/http/server.go:1830 +0x7dc
2018-03-20_18:53:58.62387 
2018-03-20_18:53:58.62387 Previous write at 0x00c421f3d0e0 by goroutine 109:
2018-03-20_18:53:58.62388   github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
2018-03-20_18:53:58.62389       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/target.go:120 +0x3ae
2018-03-20_18:53:58.62389   github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
2018-03-20_18:53:58.62390       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/scrape.go:270 +0x80a
2018-03-20_18:53:58.62391   github.com/prometheus/prometheus/scrape.(*Manager).reload()
2018-03-20_18:53:58.62391       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/manager.go:164 +0x241
2018-03-20_18:53:58.62392   github.com/prometheus/prometheus/scrape.(*Manager).Run()
2018-03-20_18:53:58.62394       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/manager.go:62 +0x5c
2018-03-20_18:53:58.62394   main.main.func12()
2018-03-20_18:53:58.62395       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:425 +0xc7
2018-03-20_18:53:58.62396   github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1()
2018-03-20_18:53:58.62397       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x34
2018-03-20_18:53:58.62397 
2018-03-20_18:53:58.62398 Goroutine 199 (running) created at:
2018-03-20_18:53:58.62398   net/http.(*Server).Serve()
2018-03-20_18:53:58.62399       /usr/local/go/src/net/http/server.go:2795 +0x364
2018-03-20_18:53:58.62400   github.com/prometheus/prometheus/web.(*Handler).Run.func5()
2018-03-20_18:53:58.62400       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:455 +0x53
2018-03-20_18:53:58.62401 
2018-03-20_18:53:58.62402 Goroutine 109 (running) created at:
2018-03-20_18:53:58.62402   github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run()
2018-03-20_18:53:58.62403       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0x10b
2018-03-20_18:53:58.62404   main.main()
2018-03-20_18:53:58.62404       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:581 +0x95b2
2018-03-20_18:53:58.62407 ==================

2018-03-20_18:53:58.62408 ==================
2018-03-20_18:53:58.62408 WARNING: DATA RACE
2018-03-20_18:53:58.62409 Read at 0x00c42d25bc00 by goroutine 199:
2018-03-20_18:53:58.62409   runtime.slicecopy()
2018-03-20_18:53:58.62410       /usr/local/go/src/runtime/slice.go:192 +0x0
2018-03-20_18:53:58.62411   github.com/prometheus/prometheus/scrape.(*Target).DiscoveredLabels()
2018-03-20_18:53:58.62411       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/target.go:114 +0xd8
2018-03-20_18:53:58.62412   runtime.call32()
2018-03-20_18:53:58.62412       /usr/local/go/src/runtime/asm_amd64.s:573 +0x3a
2018-03-20_18:53:58.62413   reflect.Value.Call()
2018-03-20_18:53:58.62413       /usr/local/go/src/reflect/value.go:308 +0xc0
2018-03-20_18:53:58.62414   text/template.(*state).evalCall()
2018-03-20_18:53:58.62415       /usr/local/go/src/text/template/exec.go:667 +0x667
2018-03-20_18:53:58.62417   text/template.(*state).evalField()
2018-03-20_18:53:58.62418       /usr/local/go/src/text/template/exec.go:557 +0xe4a
2018-03-20_18:53:58.62418   text/template.(*state).evalFieldChain()
2018-03-20_18:53:58.62419       /usr/local/go/src/text/template/exec.go:522 +0x174
2018-03-20_18:53:58.62419   text/template.(*state).evalFieldNode()
2018-03-20_18:53:58.62420       /usr/local/go/src/text/template/exec.go:489 +0x161
2018-03-20_18:53:58.62421   text/template.(*state).evalCommand()
2018-03-20_18:53:58.62421       /usr/local/go/src/text/template/exec.go:427 +0x903
2018-03-20_18:53:58.62422   text/template.(*state).evalPipeline()
2018-03-20_18:53:58.62424       /usr/local/go/src/text/template/exec.go:405 +0x1a8
2018-03-20_18:53:58.62424   text/template.(*state).walkRange()
2018-03-20_18:53:58.62425       /usr/local/go/src/text/template/exec.go:314 +0x126
2018-03-20_18:53:58.62425   text/template.(*state).walk()
2018-03-20_18:53:58.62426       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-03-20_18:53:58.62426   text/template.(*state).walk()
2018-03-20_18:53:58.62427       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-03-20_18:53:58.62427   text/template.(*state).walkRange.func1()
2018-03-20_18:53:58.62428       /usr/local/go/src/text/template/exec.go:326 +0x139
2018-03-20_18:53:58.62429   text/template.(*state).walkRange()
2018-03-20_18:53:58.62429       /usr/local/go/src/text/template/exec.go:335 +0x424
2018-03-20_18:53:58.62430   text/template.(*state).walk()
2018-03-20_18:53:58.62430       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-03-20_18:53:58.62431   text/template.(*state).walk()
2018-03-20_18:53:58.62432       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-03-20_18:53:58.62432   text/template.(*state).walkRange.func1()
2018-03-20_18:53:58.62433       /usr/local/go/src/text/template/exec.go:326 +0x139
2018-03-20_18:53:58.62434   text/template.(*state).walkRange()
2018-03-20_18:53:58.62434       /usr/local/go/src/text/template/exec.go:343 +0x6ed
2018-03-20_18:53:58.62435   text/template.(*state).walk()
2018-03-20_18:53:58.62435       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-03-20_18:53:58.62436   text/template.(*state).walk()
2018-03-20_18:53:58.62438       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-03-20_18:53:58.62439   text/template.(*state).walkTemplate()
2018-03-20_18:53:58.62439       /usr/local/go/src/text/template/exec.go:388 +0x398
2018-03-20_18:53:58.62440   text/template.(*state).walk()
2018-03-20_18:53:58.62441       /usr/local/go/src/text/template/exec.go:244 +0x2ef
2018-03-20_18:53:58.62441   text/template.(*state).walk()
2018-03-20_18:53:58.62442       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-03-20_18:53:58.62443   text/template.(*Template).execute()
2018-03-20_18:53:58.62443       /usr/local/go/src/text/template/exec.go:194 +0x3cf
2018-03-20_18:53:58.62444   text/template.(*Template).Execute()
2018-03-20_18:53:58.62444       /usr/local/go/src/text/template/exec.go:177 +0x64
2018-03-20_18:53:58.62445   html/template.(*Template).Execute()
2018-03-20_18:53:58.62447       /usr/local/go/src/html/template/template.go:122 +0xd0
2018-03-20_18:53:58.62448   github.com/prometheus/prometheus/template.Expander.ExpandHTML()
2018-03-20_18:53:58.62448       /home/bjoern/gocode/src/github.com/prometheus/prometheus/template/template.go:312 +0x5f3
2018-03-20_18:53:58.62449   github.com/prometheus/prometheus/web.(*Handler).executeTemplate()
2018-03-20_18:53:58.62450       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:779 +0x387
2018-03-20_18:53:58.62450   github.com/prometheus/prometheus/web.(*Handler).targets()
2018-03-20_18:53:58.62451       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:624 +0x4c7
2018-03-20_18:53:58.62453   github.com/prometheus/prometheus/web.(*Handler).(github.com/prometheus/prometheus/web.targets)-fm()
2018-03-20_18:53:58.62454       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:219 +0x5f
2018-03-20_18:53:58.62454   github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1()
2018-03-20_18:53:58.62455       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/http.go:309 +0x2a2
2018-03-20_18:53:58.62456   github.com/prometheus/prometheus/web.(*Handler).testReady.func1()
2018-03-20_18:53:58.62456       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:357 +0x81
2018-03-20_18:53:58.62457   github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).handle.func1()
2018-03-20_18:53:58.62458       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:50 +0x2db
2018-03-20_18:53:58.62458   github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP()
2018-03-20_18:53:58.62459       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter/router.go:299 +0xbad
2018-03-20_18:53:58.62461   github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP()
2018-03-20_18:53:58.62462       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:88 +0x67
2018-03-20_18:53:58.62462   net/http.(*ServeMux).ServeHTTP()
2018-03-20_18:53:58.62463       /usr/local/go/src/net/http/server.go:2337 +0x9f
2018-03-20_18:53:58.62464   github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2()
2018-03-20_18:53:58.62464       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x543
2018-03-20_18:53:58.62465   net/http.HandlerFunc.ServeHTTP()
2018-03-20_18:53:58.62466       /usr/local/go/src/net/http/server.go:1947 +0x51
2018-03-20_18:53:58.62466   net/http.serverHandler.ServeHTTP()
2018-03-20_18:53:58.62467       /usr/local/go/src/net/http/server.go:2694 +0xb9
2018-03-20_18:53:58.62467   net/http.(*conn).serve()
2018-03-20_18:53:58.62468       /usr/local/go/src/net/http/server.go:1830 +0x7dc
2018-03-20_18:53:58.62469 
2018-03-20_18:53:58.62469 Previous write at 0x00c42d25bc00 by goroutine 109:
2018-03-20_18:53:58.62470   runtime.slicecopy()
2018-03-20_18:53:58.62471       /usr/local/go/src/runtime/slice.go:192 +0x0
2018-03-20_18:53:58.62472   github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
2018-03-20_18:53:58.62473       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/target.go:114 +0x39d
2018-03-20_18:53:58.62474   github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
2018-03-20_18:53:58.62474       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/scrape.go:270 +0x80a
2018-03-20_18:53:58.62475   github.com/prometheus/prometheus/scrape.(*Manager).reload()
2018-03-20_18:53:58.62475       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/manager.go:164 +0x241
2018-03-20_18:53:58.62476   github.com/prometheus/prometheus/scrape.(*Manager).Run()
2018-03-20_18:53:58.62478       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/manager.go:62 +0x5c
2018-03-20_18:53:58.62478   main.main.func12()
2018-03-20_18:53:58.62479       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:425 +0xc7
2018-03-20_18:53:58.62480   github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1()
2018-03-20_18:53:58.62480       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x34
2018-03-20_18:53:58.62481 
2018-03-20_18:53:58.62482 Goroutine 199 (running) created at:
2018-03-20_18:53:58.62482   net/http.(*Server).Serve()
2018-03-20_18:53:58.62484       /usr/local/go/src/net/http/server.go:2795 +0x364
2018-03-20_18:53:58.62485   github.com/prometheus/prometheus/web.(*Handler).Run.func5()
2018-03-20_18:53:58.62486       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:455 +0x53
2018-03-20_18:53:58.62486 
2018-03-20_18:53:58.62487 Goroutine 109 (running) created at:
2018-03-20_18:53:58.62487   github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run()
2018-03-20_18:53:58.62488       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0x10b
2018-03-20_18:53:58.62489   main.main()
2018-03-20_18:53:58.62489       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:581 +0x95b2
2018-03-20_18:53:58.62490 ==================

@beorn7 beorn7 added kind/bug priority/P1 component/scraping labels Mar 21, 2018

@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member Author

beorn7 commented Mar 21, 2018

@krasi-georgiev is this something that rings a bell for you?
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member Author

beorn7 commented Mar 22, 2018

I think data races are rather critical. Could anybody familiar with this part of the codebase look into it?

It's kind of bad to have no sane 2.x version available.
@krasi-georgiev

This comment has been minimized.

Sign in to view
Copy link
Member

krasi-georgiev commented Mar 22, 2018

checking it now! looks rather simple so should be able to open a PR soon.

@krasi-georgiev krasi-georgiev referenced this issue Mar 22, 2018

Closed

fix races - discoveredLabels, scrapePools #4000

@krasi-georgiev

This comment has been minimized.

Sign in to view
Copy link
Member

krasi-georgiev commented Mar 22, 2018

one more poped up in the common package and will add a PR for that one as well.

==================
WARNING: DATA RACE
Read at 0x00c42000e2a8 by goroutine 57:
  github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).handle.func1()
      /home/krasi/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:57 +0x366
  github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP()
      /home/krasi/src/github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter/router.go:299 +0xbad
  github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP()
      /home/krasi/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:97 +0x67
  net/http.(*ServeMux).ServeHTTP()
      /usr/local/go/src/net/http/server.go:2337 +0x9f
  github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2()
      /home/krasi/src/github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x4fb
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:1947 +0x51
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2694 +0xb9
  net/http.(*conn).serve()
      /usr/local/go/src/net/http/server.go:1830 +0x7dc

Previous write at 0x00c42000e2a8 by goroutine 256:
  github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).handle.func1()
      /home/krasi/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:57 +0x3b2
  github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP()
      /home/krasi/src/github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter/router.go:299 +0xbad
  github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP()
      /home/krasi/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:97 +0x67
  net/http.(*ServeMux).ServeHTTP()
      /usr/local/go/src/net/http/server.go:2337 +0x9f
  github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2()
      /home/krasi/src/github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x4fb
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:1947 +0x51
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2694 +0xb9
  net/http.(*conn).serve()
      /usr/local/go/src/net/http/server.go:1830 +0x7dc

Goroutine 57 (running) created at:
  net/http.(*Server).Serve()
      /usr/local/go/src/net/http/server.go:2795 +0x364
  github.com/prometheus/prometheus/web.(*Handler).Run.func5()
      /home/krasi/src/github.com/prometheus/prometheus/web/web.go:488 +0x53

Goroutine 256 (running) created at:
  net/http.(*Server).Serve()
      /usr/local/go/src/net/http/server.go:2795 +0x364
  github.com/prometheus/prometheus/web.(*Handler).Run.func5()
      /home/krasi/src/github.com/prometheus/prometheus/web/web.go:488 +0x53
==================

@krasi-georgiev krasi-georgiev referenced this issue Mar 24, 2018

Merged

Races/3994 #4005

@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Apr 1, 2018

@krasi-georgiev can this be closed now? Somehow it didn't autoclose.
@krasi-georgiev

This comment has been minimized.

Sign in to view
Copy link
Member

krasi-georgiev commented Apr 1, 2018

not yet , I opened the PR against master when it should have been against release-2.2.
I didn't think I can update the PR against a different branch so I opened a new one.
#4005
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member Author

beorn7 commented Apr 9, 2018

I have built a binary with race detection at bd44e7f . Sadly, there are still races:

2018-04-09_15:52:13.05318 ==================
2018-04-09_15:52:13.05320 WARNING: DATA RACE
2018-04-09_15:52:13.05320 Read at 0x00c42fa8c2d0 by goroutine 314:
2018-04-09_15:52:13.05321   github.com/prometheus/prometheus/scrape.(*Target).DiscoveredLabels()
2018-04-09_15:52:13.05321       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/target.go:113 +0x3c
2018-04-09_15:52:13.05321   runtime.call32()
2018-04-09_15:52:13.05322       /usr/local/go/src/runtime/asm_amd64.s:573 +0x3a
2018-04-09_15:52:13.05322   reflect.Value.Call()
2018-04-09_15:52:13.05322       /usr/local/go/src/reflect/value.go:308 +0xc0
2018-04-09_15:52:13.05323   text/template.(*state).evalCall()
2018-04-09_15:52:13.05323       /usr/local/go/src/text/template/exec.go:667 +0x667
2018-04-09_15:52:13.05323   text/template.(*state).evalField()
2018-04-09_15:52:13.05324       /usr/local/go/src/text/template/exec.go:557 +0xe4a
2018-04-09_15:52:13.05324   text/template.(*state).evalFieldChain()
2018-04-09_15:52:13.05325       /usr/local/go/src/text/template/exec.go:522 +0x174
2018-04-09_15:52:13.05325   text/template.(*state).evalFieldNode()
2018-04-09_15:52:13.05325       /usr/local/go/src/text/template/exec.go:489 +0x161
2018-04-09_15:52:13.05326   text/template.(*state).evalCommand()
2018-04-09_15:52:13.05326       /usr/local/go/src/text/template/exec.go:427 +0x903
2018-04-09_15:52:13.05326   text/template.(*state).evalPipeline()
2018-04-09_15:52:13.05327       /usr/local/go/src/text/template/exec.go:405 +0x1a8
2018-04-09_15:52:13.05327   text/template.(*state).walkRange()
2018-04-09_15:52:13.05327       /usr/local/go/src/text/template/exec.go:314 +0x126
2018-04-09_15:52:13.05329   text/template.(*state).walk()
2018-04-09_15:52:13.05329       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-04-09_15:52:13.05329   text/template.(*state).walk()
2018-04-09_15:52:13.05330       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-04-09_15:52:13.05330   text/template.(*state).walkRange.func1()
2018-04-09_15:52:13.05330       /usr/local/go/src/text/template/exec.go:326 +0x139
2018-04-09_15:52:13.05331   text/template.(*state).walkRange()
2018-04-09_15:52:13.05331       /usr/local/go/src/text/template/exec.go:335 +0x424
2018-04-09_15:52:13.05332   text/template.(*state).walk()
2018-04-09_15:52:13.05332       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-04-09_15:52:13.05332   text/template.(*state).walk()
2018-04-09_15:52:13.05333       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-04-09_15:52:13.05334   text/template.(*state).walkRange.func1()
2018-04-09_15:52:13.05335       /usr/local/go/src/text/template/exec.go:326 +0x139
2018-04-09_15:52:13.05335   text/template.(*state).walkRange()
2018-04-09_15:52:13.05335       /usr/local/go/src/text/template/exec.go:343 +0x6ed
2018-04-09_15:52:13.05336   text/template.(*state).walk()
2018-04-09_15:52:13.05336       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-04-09_15:52:13.05337   text/template.(*state).walk()
2018-04-09_15:52:13.05337       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-04-09_15:52:13.05337   text/template.(*state).walkTemplate()
2018-04-09_15:52:13.05338       /usr/local/go/src/text/template/exec.go:388 +0x398
2018-04-09_15:52:13.05339   text/template.(*state).walk()
2018-04-09_15:52:13.05339       /usr/local/go/src/text/template/exec.go:244 +0x2ef
2018-04-09_15:52:13.05340   text/template.(*state).walk()
2018-04-09_15:52:13.05340       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-04-09_15:52:13.05341   text/template.(*Template).execute()
2018-04-09_15:52:13.05341       /usr/local/go/src/text/template/exec.go:194 +0x3cf
2018-04-09_15:52:13.05342   text/template.(*Template).Execute()
2018-04-09_15:52:13.05342       /usr/local/go/src/text/template/exec.go:177 +0x64
2018-04-09_15:52:13.05342   html/template.(*Template).Execute()
2018-04-09_15:52:13.05343       /usr/local/go/src/html/template/template.go:122 +0xd0
2018-04-09_15:52:13.05343   github.com/prometheus/prometheus/template.Expander.ExpandHTML()
2018-04-09_15:52:13.05344       /home/bjoern/gocode/src/github.com/prometheus/prometheus/template/template.go:312 +0x5f3
2018-04-09_15:52:13.05344   github.com/prometheus/prometheus/web.(*Handler).executeTemplate()
2018-04-09_15:52:13.05345       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:785 +0x387
2018-04-09_15:52:13.05345   github.com/prometheus/prometheus/web.(*Handler).targets()
2018-04-09_15:52:13.05345       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:624 +0x4c7
2018-04-09_15:52:13.05346   github.com/prometheus/prometheus/web.(*Handler).(github.com/prometheus/prometheus/web.targets)-fm()
2018-04-09_15:52:13.05349       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:219 +0x5f
2018-04-09_15:52:13.05350   github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1()
2018-04-09_15:52:13.05350       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/http.go:309 +0x2a2
2018-04-09_15:52:13.05350   github.com/prometheus/prometheus/web.(*Handler).testReady.func1()
2018-04-09_15:52:13.05351       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:357 +0x81
2018-04-09_15:52:13.05351   github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).handle.func1()
2018-04-09_15:52:13.05351       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:60 +0x2db
2018-04-09_15:52:13.05352   github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP()
2018-04-09_15:52:13.05352       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter/router.go:299 +0xbad
2018-04-09_15:52:13.05353   github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP()
2018-04-09_15:52:13.05354       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:98 +0x67
2018-04-09_15:52:13.05354   net/http.(*ServeMux).ServeHTTP()
2018-04-09_15:52:13.05354       /usr/local/go/src/net/http/server.go:2337 +0x9f
2018-04-09_15:52:13.05355   github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2()
2018-04-09_15:52:13.05355       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x543
2018-04-09_15:52:13.05356   net/http.HandlerFunc.ServeHTTP()
2018-04-09_15:52:13.05356       /usr/local/go/src/net/http/server.go:1947 +0x51
2018-04-09_15:52:13.05357   net/http.serverHandler.ServeHTTP()
2018-04-09_15:52:13.05357       /usr/local/go/src/net/http/server.go:2694 +0xb9
2018-04-09_15:52:13.05357   net/http.(*conn).serve()
2018-04-09_15:52:13.05358       /usr/local/go/src/net/http/server.go:1830 +0x7dc
2018-04-09_15:52:13.05358 
2018-04-09_15:52:13.05358 Previous write at 0x00c42fa8c2d0 by goroutine 109:
2018-04-09_15:52:13.05359   [failed to restore the stack]
2018-04-09_15:52:13.05359 
2018-04-09_15:52:13.05359 Goroutine 314 (running) created at:
2018-04-09_15:52:13.05360   net/http.(*Server).Serve()
2018-04-09_15:52:13.05360       /usr/local/go/src/net/http/server.go:2795 +0x364
2018-04-09_15:52:13.05360   github.com/prometheus/prometheus/web.(*Handler).Run.func5()
2018-04-09_15:52:13.05362       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:455 +0x53
2018-04-09_15:52:13.05362 
2018-04-09_15:52:13.05363 Goroutine 109 (running) created at:
2018-04-09_15:52:13.05363   github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run()
2018-04-09_15:52:13.05363       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0x10b
2018-04-09_15:52:13.05364   main.main()
2018-04-09_15:52:13.05364       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:581 +0x95f9
2018-04-09_15:52:13.05365 ==================
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member Author

beorn7 commented Apr 9, 2018

This seems to be the same, but this time with both stack traces:

2018-04-09_15:52:13.02258 ==================
2018-04-09_15:52:13.02260 WARNING: DATA RACE
2018-04-09_15:52:13.02261 Read at 0x00c43aeddb80 by goroutine 314:
2018-04-09_15:52:13.02261   runtime.slicecopy()
2018-04-09_15:52:13.02261       /usr/local/go/src/runtime/slice.go:192 +0x0
2018-04-09_15:52:13.02262   github.com/prometheus/prometheus/scrape.(*Target).DiscoveredLabels()
2018-04-09_15:52:13.02262       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/target.go:114 +0xd8
2018-04-09_15:52:13.02262   runtime.call32()
2018-04-09_15:52:13.02263       /usr/local/go/src/runtime/asm_amd64.s:573 +0x3a
2018-04-09_15:52:13.02263   reflect.Value.Call()
2018-04-09_15:52:13.02263       /usr/local/go/src/reflect/value.go:308 +0xc0
2018-04-09_15:52:13.02264   text/template.(*state).evalCall()
2018-04-09_15:52:13.02264       /usr/local/go/src/text/template/exec.go:667 +0x667
2018-04-09_15:52:13.02265   text/template.(*state).evalField()
2018-04-09_15:52:13.02265       /usr/local/go/src/text/template/exec.go:557 +0xe4a
2018-04-09_15:52:13.02266   text/template.(*state).evalFieldChain()
2018-04-09_15:52:13.02266       /usr/local/go/src/text/template/exec.go:522 +0x174
2018-04-09_15:52:13.02266   text/template.(*state).evalFieldNode()
2018-04-09_15:52:13.02267       /usr/local/go/src/text/template/exec.go:489 +0x161
2018-04-09_15:52:13.02267   text/template.(*state).evalCommand()
2018-04-09_15:52:13.02267       /usr/local/go/src/text/template/exec.go:427 +0x903
2018-04-09_15:52:13.02268   text/template.(*state).evalPipeline()
2018-04-09_15:52:13.02268       /usr/local/go/src/text/template/exec.go:405 +0x1a8
2018-04-09_15:52:13.02269   text/template.(*state).walkRange()
2018-04-09_15:52:13.02270       /usr/local/go/src/text/template/exec.go:314 +0x126
2018-04-09_15:52:13.02270   text/template.(*state).walk()
2018-04-09_15:52:13.02270       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-04-09_15:52:13.02271   text/template.(*state).walk()
2018-04-09_15:52:13.02271       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-04-09_15:52:13.02271   text/template.(*state).walkRange.func1()
2018-04-09_15:52:13.02272       /usr/local/go/src/text/template/exec.go:326 +0x139
2018-04-09_15:52:13.02272   text/template.(*state).walkRange()
2018-04-09_15:52:13.02272       /usr/local/go/src/text/template/exec.go:335 +0x424
2018-04-09_15:52:13.02273   text/template.(*state).walk()
2018-04-09_15:52:13.02273       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-04-09_15:52:13.02273   text/template.(*state).walk()
2018-04-09_15:52:13.02274       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-04-09_15:52:13.02274   text/template.(*state).walkRange.func1()
2018-04-09_15:52:13.02275       /usr/local/go/src/text/template/exec.go:326 +0x139
2018-04-09_15:52:13.02275   text/template.(*state).walkRange()
2018-04-09_15:52:13.02275       /usr/local/go/src/text/template/exec.go:343 +0x6ed
2018-04-09_15:52:13.02276   text/template.(*state).walk()
2018-04-09_15:52:13.02276       /usr/local/go/src/text/template/exec.go:242 +0x61b
2018-04-09_15:52:13.02276   text/template.(*state).walk()
2018-04-09_15:52:13.02277       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-04-09_15:52:13.02278   text/template.(*state).walkTemplate()
2018-04-09_15:52:13.02278       /usr/local/go/src/text/template/exec.go:388 +0x398
2018-04-09_15:52:13.02279   text/template.(*state).walk()
2018-04-09_15:52:13.02279       /usr/local/go/src/text/template/exec.go:244 +0x2ef
2018-04-09_15:52:13.02280   text/template.(*state).walk()
2018-04-09_15:52:13.02280       /usr/local/go/src/text/template/exec.go:239 +0x263
2018-04-09_15:52:13.02280   text/template.(*Template).execute()
2018-04-09_15:52:13.02281       /usr/local/go/src/text/template/exec.go:194 +0x3cf
2018-04-09_15:52:13.02281   text/template.(*Template).Execute()
2018-04-09_15:52:13.02282       /usr/local/go/src/text/template/exec.go:177 +0x64
2018-04-09_15:52:13.02283   html/template.(*Template).Execute()
2018-04-09_15:52:13.02284       /usr/local/go/src/html/template/template.go:122 +0xd0
2018-04-09_15:52:13.02284   github.com/prometheus/prometheus/template.Expander.ExpandHTML()
2018-04-09_15:52:13.02285       /home/bjoern/gocode/src/github.com/prometheus/prometheus/template/template.go:312 +0x5f3
2018-04-09_15:52:13.02285   github.com/prometheus/prometheus/web.(*Handler).executeTemplate()
2018-04-09_15:52:13.02285       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:785 +0x387
2018-04-09_15:52:13.02286   github.com/prometheus/prometheus/web.(*Handler).targets()
2018-04-09_15:52:13.02286       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:624 +0x4c7
2018-04-09_15:52:13.02287   github.com/prometheus/prometheus/web.(*Handler).(github.com/prometheus/prometheus/web.targets)-fm()
2018-04-09_15:52:13.02288       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:219 +0x5f
2018-04-09_15:52:13.02288   github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1()
2018-04-09_15:52:13.02289       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/http.go:309 +0x2a2
2018-04-09_15:52:13.02289   github.com/prometheus/prometheus/web.(*Handler).testReady.func1()
2018-04-09_15:52:13.02290       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:357 +0x81
2018-04-09_15:52:13.02290   github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).handle.func1()
2018-04-09_15:52:13.02290       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:60 +0x2db
2018-04-09_15:52:13.02291   github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP()
2018-04-09_15:52:13.02291       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter/router.go:299 +0xbad
2018-04-09_15:52:13.02292   github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP()
2018-04-09_15:52:13.02293       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:98 +0x67
2018-04-09_15:52:13.02293   net/http.(*ServeMux).ServeHTTP()
2018-04-09_15:52:13.02294       /usr/local/go/src/net/http/server.go:2337 +0x9f
2018-04-09_15:52:13.02294   github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2()
2018-04-09_15:52:13.02295       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x543
2018-04-09_15:52:13.02295   net/http.HandlerFunc.ServeHTTP()
2018-04-09_15:52:13.02296       /usr/local/go/src/net/http/server.go:1947 +0x51
2018-04-09_15:52:13.02296   net/http.serverHandler.ServeHTTP()
2018-04-09_15:52:13.02296       /usr/local/go/src/net/http/server.go:2694 +0xb9
2018-04-09_15:52:13.02297   net/http.(*conn).serve()
2018-04-09_15:52:13.02297       /usr/local/go/src/net/http/server.go:1830 +0x7dc
2018-04-09_15:52:13.02298 
2018-04-09_15:52:13.02298 Previous write at 0x00c43aeddb80 by goroutine 109:
2018-04-09_15:52:13.02298   runtime.slicecopy()
2018-04-09_15:52:13.02299       /usr/local/go/src/runtime/slice.go:192 +0x0
2018-04-09_15:52:13.02299   github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
2018-04-09_15:52:13.02300       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/target.go:114 +0x39d
2018-04-09_15:52:13.02300   github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
2018-04-09_15:52:13.02301       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/scrape.go:270 +0x805
2018-04-09_15:52:13.02301   github.com/prometheus/prometheus/scrape.(*Manager).reload()
2018-04-09_15:52:13.02302       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/manager.go:170 +0x30d
2018-04-09_15:52:13.02302   github.com/prometheus/prometheus/scrape.(*Manager).Run()
2018-04-09_15:52:13.02303       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/manager.go:69 +0x5c
2018-04-09_15:52:13.02303   main.main.func12()
2018-04-09_15:52:13.02304       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:425 +0xc7
2018-04-09_15:52:13.02304   github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1()
2018-04-09_15:52:13.02304       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x34
2018-04-09_15:52:13.02305 
2018-04-09_15:52:13.02305 Goroutine 314 (running) created at:
2018-04-09_15:52:13.02305   net/http.(*Server).Serve()
2018-04-09_15:52:13.02306       /usr/local/go/src/net/http/server.go:2795 +0x364
2018-04-09_15:52:13.02306   github.com/prometheus/prometheus/web.(*Handler).Run.func5()
2018-04-09_15:52:13.02307       /home/bjoern/gocode/src/github.com/prometheus/prometheus/web/web.go:455 +0x53
2018-04-09_15:52:13.02308 
2018-04-09_15:52:13.02308 Goroutine 109 (running) created at:
2018-04-09_15:52:13.02308   github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run()
2018-04-09_15:52:13.02309       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0x10b
2018-04-09_15:52:13.02309   main.main()
2018-04-09_15:52:13.02310       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:581 +0x95f9
2018-04-09_15:52:13.02310 ==================
@krasi-georgiev

This comment has been minimized.

Sign in to view
Copy link
Member

krasi-georgiev commented Apr 9, 2018

will check it again tomorrow.

@krasi-georgiev krasi-georgiev referenced this issue Apr 9, 2018

Merged

add mutex for DiscoveredLabels #4066

@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member Author

beorn7 commented Apr 10, 2018

Looks like #4066 fixed it. Closing for now.

@beorn7 beorn7 closed this Apr 10, 2018

@krasi-georgiev krasi-georgiev referenced this issue Aug 14, 2018

Closed

Extend prombench to trigger a job that will catch race conditions /race #134

@krasi-georgiev krasi-georgiev referenced this issue Aug 27, 2018

Open

Add another test to benchmark Prometheus with Race enabled. #86

@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 22, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 22, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.