branch-4.1: [fix](file cache) keep the cache monitor off the LRU recorder lock #67315 - #67495
Open
github-actions[bot] wants to merge 1 commit into
Open
branch-4.1: [fix](file cache) keep the cache monitor off the LRU recorder lock #67315#67495github-actions[bot] wants to merge 1 commit into
github-actions[bot] wants to merge 1 commit into
Conversation
…67315) run_background_monitor() ended its loop with update_shadow_queue_element_count_metrics(), which takes _mutex_lru_log. The LRU log replay thread holds that lock for as long as it takes to drain the log queue, so a slow consumer froze the monitor with it: check_disk_resource_limit(), check_need_evict_cache_in_advance() and every gauge stopped running, and the disk resource limit mode stayed at whatever value it happened to hold. Gauges were observed frozen for up to 40 minutes in production. The call was redundant from the start. #64798 added the shadow queue element count gauge and published it in two places: inside replay_queue_event(), under the same lock that mutates the shadow queue, and again from the monitor every file_cache_background_monitor_interval_ms as a periodic refresh. Nothing outside replay_queue_event() mutates a shadow queue, so that refresh could only rewrite a value that had just been published and could not have changed since. What it did add was a dependency from the disk protection loop onto a lock owned by a background consumer. Drop the call, and update_shadow_queue_element_count_metrics() with it: it existed only for that refresh, and leaving a public method that takes _mutex_lru_log invites the next background loop to reintroduce the coupling. The gauge is still published by replay, now on the replay interval instead of the monitor interval. Its test is replaced by one asserting that replay publishes the gauge on its own. How long replay holds the lock is a separate problem, addressed separately.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #67315