Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

thanos query takes high cpu when added self as a store #1328

Closed
infa-ddeore opened this issue Jul 14, 2019 · 2 comments
Closed

thanos query takes high cpu when added self as a store #1328

infa-ddeore opened this issue Jul 14, 2019 · 2 comments

Comments

@infa-ddeore
Copy link

@infa-ddeore infa-ddeore commented Jul 14, 2019

Thanos: 0.5.0
go: go version go1.11.2 darwin/amd64

thanos, version 0.5.0 (branch: HEAD, revision: 72820b3f41794140403fd04d6da82299f2c16447)
  build user:       root@7d72e9360b09
  build date:       20190606-10:49:59
  go version:       go1.12.5

What happened
If thanos query is added itself as a store then it takes high cpu after running any query.

What you expected to happen
It should give warning about adding self or shouldn't allow adding itself

How to reproduce it (as minimally and precisely as possible):

  1. start thanos query
./thanos-0.5.0.darwin-amd64/thanos query --http-address 0.0.0.0:9091 --store=localhost:10901
  1. access the ui and run any query, eg. up
  2. wait for ~30-45 seconds
  3. local machine's cpu boosts up and thanos process needs kill -9

Full logs to relevant components

Uncomment if you would like to post collapsible logs:

level=warn ts=2019-07-14T03:09:27.682971Z caller=proxy.go:349 err="failed to receive any data from Addr: localhost:10901 Labels: [] Mint: 0 Maxt: 9223372036854775807: context canceled" msg="returning partial response"
level=warn ts=2019-07-14T03:09:27.683081Z caller=proxy.go:349 err="failed to receive any data from Addr: localhost:10901 Labels: [] Mint: 0 Maxt: 9223372036854775807: context canceled" msg="returning partial response"
level=warn ts=2019-07-14T03:09:27.683321Z caller=proxy.go:349 err="failed to receive any data from Addr: localhost:10901 Labels: [] Mint: 0 Maxt: 9223372036854775807: context canceled" msg="returning partial response"
level=warn ts=2019-07-14T03:09:27.683494Z caller=proxy.go:349 err="failed to receive any data from Addr: localhost:10901 Labels: [] Mint: 0 Maxt: 9223372036854775807: context canceled" msg="returning partial response"
level=warn ts=2019-07-14T03:09:27.683669Z caller=proxy.go:349 err="failed to receive any data from Addr: localhost:10901 Labels: [] Mint: 0 Maxt: 9223372036854775807: context canceled" msg="returning partial response"
level=warn ts=2019-07-14T03:09:27.68383Z caller=proxy.go:349 err="failed to receive any data from Addr: localhost:10901 Labels: [] Mint: 0 Maxt: 9223372036854775807: context canceled" msg="returning partial response"

Anything else we need to know

@GiedriusS
Copy link
Member

@GiedriusS GiedriusS commented Jul 14, 2019

Yeah... that's to be expected 😕 , however, how would you solve this? Simply checking for 127.0.0.1 or localhost is not an answer since someone could a proxy in front and then we would be back to square one. We could print a message at best in such situations, IMHO. Also, Having something like "correlation IDs" on top of the queries would be nice however that would probably incur a heavy performance penalty and bring more harm than good.

@stale
Copy link

@stale stale bot commented Jan 11, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Jan 11, 2020
@stale stale bot closed this Jan 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants