Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

/metrics calls on ingesters taking multiple seconds #1281

Closed
bboreham opened this issue Mar 14, 2019 · 3 comments
Closed

/metrics calls on ingesters taking multiple seconds #1281

bboreham opened this issue Mar 14, 2019 · 3 comments

Comments

@bboreham
Copy link
Contributor

According to Jaeger traces about half are <20ms and half are 2-10 seconds.
According to Prometheus the mean is 1.5 seconds.
Traces have no detail, haven't really looked into it.

Similar: #283
@bboreham
Copy link
Contributor Author

I added some tracing and this lead to a ReadMemStats call in prometheus/client_golang.

So I believe the root cause is (a) my ingester has a 30GB Go heap which can take more than 10 seconds to run a GC cycle and (b) ReadMemStats() is blocked for that cycle - golang/go#19812

@beorn7 beorn7 mentioned this issue Mar 21, 2019
Closed
@bboreham
Copy link
Contributor Author

This should be fixed by prometheus/client_golang#568, when Cortex updates to that version.

@gouthamve
Copy link
Contributor

This should be fixed now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gouthamve @bboreham