New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix flaky tests caused by OPTIMIZE FINAL failing memory budget check #49764
Conversation
This is an automated comment for commit a4abb81 with description of existing statuses. It's updated for the latest CI running
|
if (!is_background_memory_usage_ok(out_disable_reason)) | ||
{ | ||
constexpr auto poll_interval = std::chrono::seconds(1); | ||
Int64 attempts = timeout / poll_interval; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This time is quite big (120 seconds). So we can wait here for a long time + additionally we can wait for currently_merging_mutating_parts
another 120 seconds (next lines after loop). Maybe update timeout value after wait here?
@al13n321 Maybe it worth to add |
Changelog category (leave one):
02461_prewhere_row_level_policy_lightweight_delete and 02458_relax_too_many_parts failed in the same way: their
OPTIMIZE ... FINAL
query didn't do the merge becauseCurrent background tasks memory usage (31.77 GiB) is more than the limit (31.01 GiB)
.This PR does two things:
A lot of the 31+ GB background memory usage seems to come from tests
02581_share_big_sets_*
. This PR marks 2/4 of them as no-parallel (the ones that are also marked long; I guessed that the other 2 use much less memory, but don't actually know what I'm doing).OPTIMIZE FINAL
already has a retry loop for something similar: to wait for other merges to complete. This PR moves the memory budget check into a similar retry loop. It re-checks the limit every second for up to 2 minutes (lock_acquire_timeout_for_background_operations
).