Limit maximum metrics to be deleted simultaniously #1125
Conversation
|
Hi everyone, |
felixhuettner
force-pushed
the
limit_cleanup
branch
from
b7344bb
to
2bb4e76
Compare
|
Implemented a configuration option. Not sure about the name though |
felixhuettner
force-pushed
the
limit_cleanup
branch
from
2bb4e76
to
3122bc6
Compare
felixhuettner
force-pushed
the
limit_cleanup
branch
from
3122bc6
to
cd2de29
Compare
|
I hope i got the release note format correctly :) |
releasenotes/notes/configurable_clenaup_batch_size-a785c1aa6ee29058.yaml
Outdated
Show resolved
Hide resolved
|
The CI seems broken of the latest tenacity, somebody should rewrite the current code to stop using the deprecated now-removed API. |
felixhuettner
force-pushed
the
limit_cleanup
branch
from
cd2de29
to
18a1f3d
Compare
|
What is then the best way to go forward here? Should we just leave it sitting here until the CI is fixed (that would be fine with me) |
|
Would you have any details where i could start taking a look? Then i'll see if i can get some time for that |
|
The latest tenacity release https://pypi.org/project/tenacity/ 7.0.0 removed old compat code for retry functions. |
|
I've put together a PR to fix CI by pinning tenacity and sqlalchemy, #1124 |
|
@Mergifyio rebase |
|
Command
|
|
@Mergifyio update |
|
Command
|
|
@felixhuettner you really need to click that checkbox 😆 |
|
:D ok that is fixed now. |
|
@felixhuettner is not allowed to run commands |
felixhuettner
force-pushed
the
limit_cleanup
branch
from
18a1f3d
to
cecf952
Compare
|
:D ok, let's do it manually |
|
@Mergifyio rebase |
Currently gnocchi-metricd tries to collect all metrics that should be deleted and deletes them in one big loop. If a large amount of old metrics has accumulated this will cause high load on the database as it needs to find and sort all of these rows. It will also cause a high amount of memory usage for gnocchi-metricd as it needs to also sort all of these rows in memory. To fix this the amount of rows being deleted in one run is limited to 10000. This is sufficient to allow gnocchi to cleanup metrics in a fast way under normal circumstances while still allowing it to catch up if no deletion has been performed for a while. Signed-off-by: Felix Huettner <felix.huettner@mail.schwarz>
|
Command
|
Currently gnocchi-metricd tries to collect all metrics that should be deleted
and deletes them in one big loop. If a large amount of old metrics has
accumulated this will cause high load on the database as it needs to find
and sort all of these rows. It will also cause a high amount of memory usage
for gnocchi-metricd as it needs to also sort all of these rows in memory.
To fix this the amount of rows being deleted in one run is limited to 10000.
This is sufficient to allow gnocchi to cleanup metrics in a fast way under
normal circumstances while still allowing it to catch up if no deletion has
been performed for a while.
Signed-off-by: Felix Huettner felix.huettner@mail.schwarz