Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race when reloading the config #4067

Closed
krasi-georgiev opened this Issue Apr 9, 2018 · 6 comments

Comments

Assignees
No one assigned
Labels
component/scraping kind/bug priority/P1
Projects
None yet
Milestone
No milestone
3 participants
@krasi-georgiev @beorn7 @simonpasquier
@krasi-georgiev
Copy link
Member

krasi-georgiev commented Apr 9, 2018 •
edited

What did you do?

  • started prometheus to scrape itself -
    go run --race main.go uname_default.go fdlimits_default.go --storage.tsdb.min-block-duration=5m --storage.tsdb.max-block-duration=15m --config.file=../../promBench.yml
  • wait few seconds until prometheus registeres all targets and starts scraping.
  • in a separate terminal run watch -n 1 killall -1 main to keep reloading the config file
==================
WARNING: DATA RACE
Read at 0x00c420013838 by goroutine 14:
  github.com/prometheus/prometheus/scrape.(*scrapePool).appender()
      /home/krasi/src/github.com/prometheus/prometheus/scrape/scrape.go:404 +0x262
  github.com/prometheus/prometheus/scrape.(*scrapePool).(github.com/prometheus/prometheus/scrape.appender)-fm()
      /home/krasi/src/github.com/prometheus/prometheus/scrape/scrape.go:171 +0x41
  github.com/prometheus/prometheus/scrape.(*scrapeLoop).report()
      /home/krasi/src/github.com/prometheus/prometheus/scrape/scrape.go:991 +0x207
  github.com/prometheus/prometheus/scrape.(*scrapeLoop).run()
      /home/krasi/src/github.com/prometheus/prometheus/scrape/scrape.go:720 +0xbf7

Previous write at 0x00c420013838 by goroutine 86:
  github.com/prometheus/prometheus/scrape.(*scrapePool).reload()
      /home/krasi/src/github.com/prometheus/prometheus/scrape/scrape.go:214 +0x1b2
  github.com/prometheus/prometheus/scrape.(*Manager).ApplyConfig()
      /home/krasi/src/github.com/prometheus/prometheus/scrape/manager.go:104 +0x392
  github.com/prometheus/prometheus/scrape.(*Manager).ApplyConfig-fm()
      /home/krasi/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:306 +0x4b
  main.reloadConfig()
      /home/krasi/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:606 +0x2eb
  main.main.func14()
      /home/krasi/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:450 +0x57e
  github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1()
      /home/krasi/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x34

Goroutine 14 (running) created at:
  github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
      /home/krasi/src/github.com/prometheus/prometheus/scrape/scrape.go:312 +0x589
  github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
      /home/krasi/src/github.com/prometheus/prometheus/scrape/scrape.go:270 +0x74d
  github.com/prometheus/prometheus/scrape.(*Manager).reload()
      /home/krasi/src/github.com/prometheus/prometheus/scrape/manager.go:170 +0x30d
  github.com/prometheus/prometheus/scrape.(*Manager).Run()
      /home/krasi/src/github.com/prometheus/prometheus/scrape/manager.go:69 +0x5c
  main.main.func12()
      /home/krasi/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:425 +0xc7
  github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1()
      /home/krasi/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x34

Goroutine 86 (running) created at:
  github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run()
      /home/krasi/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0x10b
  main.main()
      /home/krasi/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:581 +0x95f3
==================
  • Prometheus version:
    2.2
  • Prometheus configuration file:
global:
  scrape_interval:     500ms # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  - job_name: 'prometheus1'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'prometheus2'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'prometheus3'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'prometheus4'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'prometheus5'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'prometheus6'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'prometheus7'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'prometheus8'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'prometheus9'
    static_configs:
      - targets: ['localhost:9090']

@krasi-georgiev krasi-georgiev added priority/P1 component/scraping kind/bug labels Apr 9, 2018

@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Apr 10, 2018

Confirmed. Could even find a slightly different stack trace:

2018-04-10_13:56:06.49425 ==================
2018-04-10_13:56:06.49427 WARNING: DATA RACE
2018-04-10_13:56:06.49427 Read at 0x00c43d222738 by goroutine 635:
2018-04-10_13:56:06.49428   github.com/prometheus/prometheus/scrape.(*scrapePool).mutateSampleLabels()
2018-04-10_13:56:06.49428       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/scrape.go:346 +0x1cb
2018-04-10_13:56:06.49428   github.com/prometheus/prometheus/scrape.newScrapePool.func2.1()
2018-04-10_13:56:06.49429       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/scrape.go:169 +0x6f
2018-04-10_13:56:06.49429   github.com/prometheus/prometheus/scrape.(*scrapeLoop).append()
2018-04-10_13:56:06.49430       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/scrape.go:880 +0x87f
2018-04-10_13:56:06.49430   github.com/prometheus/prometheus/scrape.(*scrapeLoop).run()
2018-04-10_13:56:06.49430       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/scrape.go:704 +0xa5f
2018-04-10_13:56:06.49431 
2018-04-10_13:56:06.49431 Previous write at 0x00c43d222738 by goroutine 110:
2018-04-10_13:56:06.49432   github.com/prometheus/prometheus/scrape.(*scrapePool).reload()
2018-04-10_13:56:06.49433       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/scrape.go:214 +0x1b2
2018-04-10_13:56:06.49434   github.com/prometheus/prometheus/scrape.(*Manager).ApplyConfig()
2018-04-10_13:56:06.49434       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/manager.go:104 +0x392
2018-04-10_13:56:06.49435   github.com/prometheus/prometheus/scrape.(*Manager).ApplyConfig-fm()
2018-04-10_13:56:06.49435       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:306 +0x4b
2018-04-10_13:56:06.49436   main.reloadConfig()
2018-04-10_13:56:06.49436       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:606 +0x2eb
2018-04-10_13:56:06.49437   main.main.func14()
2018-04-10_13:56:06.49437       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:450 +0x57e
2018-04-10_13:56:06.49437   github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1()
2018-04-10_13:56:06.49438       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x34
2018-04-10_13:56:06.49438 
2018-04-10_13:56:06.49438 Goroutine 635 (running) created at:
2018-04-10_13:56:06.49440   github.com/prometheus/prometheus/scrape.(*scrapePool).reload.func1()
2018-04-10_13:56:06.49440       /home/bjoern/gocode/src/github.com/prometheus/prometheus/scrape/scrape.go:235 +0x93
2018-04-10_13:56:06.49441 
2018-04-10_13:56:06.49441 Goroutine 110 (running) created at:
2018-04-10_13:56:06.49441   github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run()
2018-04-10_13:56:06.49442       /home/bjoern/gocode/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0x10b
2018-04-10_13:56:06.49443   main.main()
2018-04-10_13:56:06.49443       /home/bjoern/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:581 +0x95f9
2018-04-10_13:56:06.49444 ==================
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Apr 10, 2018

The above built at f8dcf9b
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Apr 12, 2018

Would be great if this could be fixed. It's hard to hunt down conventional bugs with the looming suspicion that all the weird things we see might just be caused by a data race.
@simonpasquier

This comment has been minimized.

Sign in to view
Copy link
Member

simonpasquier commented Apr 12, 2018

@beorn7 @krasi-georgiev I'm having a look at it right now.
@krasi-georgiev

This comment has been minimized.

Sign in to view
Copy link
Member Author

krasi-georgiev commented Apr 12, 2018 •
edited

@simonpasquier thanks and good luck :) this one doesn't look so easy at first glance.

This was referenced Apr 13, 2018

Closed
Merged

@beorn7 beorn7 closed this in #4096 Apr 25, 2018

@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 22, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 22, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.