-
Notifications
You must be signed in to change notification settings - Fork 9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
heavy memory usage #3774
Comments
yes the promql defenetely needs optimising and there is already an idea to use streaming. I am also trying few things, but this will take time. |
There's in sufficient information here to know if anything is wrong. What is your ingestion rate and what sort of queries are you using? In particular are they pulling in hours or days of data? |
99,5% of the queries are just pulling the last 5 minutes of data. Also no complex aggregations are performed. There is no mentionable cpu load |
What about the other .5%? What is your ingestion rate? There's still insufficient information to know if this is unexpected. Can you run this dashboard against your Prometheus: https://grafana.com/dashboards/3834 |
It looks like you have some expensive rules that are taking too long to evaluate, 2.1 will show you how long each rule is taking so you can pinpoint which rule it is in particular. That may be what's pulling in all the data. |
Ive done futher investigations. The rule evalutation time is around 20s, but the memory seems to be allocated for multiple minutes. record: statistic_cpu_intensive
expr: avg(avg_over_time(cpu_percent[12h]))
BY (target) > 95 I know the query needs to load all memory series, but it wouldn't scale well if this would be done fully in memory. E.g PostgreSQL allows to configure a memory limit, larger memory allocations will trigger disk based aggregations. |
Prometheus doesn't spill to disk, and if you have a query so expensive that it doesn't fit in RAM you should probably be doing it outside of Prometheus as that's getting into heavy reporting use cases. That rule is also filtering, that is not a good idea. You should only filter in alerting rules. You want the |
Ah ok, good to know. |
I'm running Prometheus 2.2 now, I wonder how I would see how long each rule is taking so I can pinpoint which rule is taking too long (if any) ? |
For those running 2.2 and trying to find how long each rule is taking, the run time per rule should be showing if you go to http:///rules |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
The prometheus server is currently allocating the whole series of 30GB in memory. By just putting hardware on it, its back again and faster than before ;D. But in general its not an option 😄
What did you do?
Queries, often!
Environment
The text was updated successfully, but these errors were encountered: