[v23.3.x] cloud_storage: Implement "carryover" cache trimming mechanism #18138
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(cherry picked from commit 57985b7)
Signed-off-by: Michael Boquard <michael@redpanda.com> (cherry picked from commit e088ac3)
Add new parameter that controls cache carryover behavior. (cherry picked from commit 940fcd4)
The "carryover" behavior allows cache to use information from the previous trim to quickly trim the cache without scanning the whole directory. This allows cache to avoid blocking readers. In a situation when the cache cntains very large number of files the recursive directory walk could take few minutes. We're not allowing number of objects stored in the cache to overshoot so all the readers are blocked until the walk is finished. This commit adds new "carryover" trim mechanism which is running before the normal trim and uses information obtained during the previous fast or full trim to delete some objects wihtout walking the directory tree. (cherry picked from commit 5ece9d4)
(cherry picked from commit fdf3498)
(cherry picked from commit 1256594)
Change the configuration parameter and treat the value as number of bytes that we can use to store carryover data. (cherry picked from commit e1c30bc)
Reserve memory units for the carryover mechanism in materialized_resrouces. (cherry picked from commit 0f84fdb)
(cherry picked from commit 6b57e0c)
(cherry picked from commit 61a09b4)
In case if carryover trim was able to release enough space start trim in the background and return early. This unblocks the hydration that reserved space and triggered the trim. We need to run normal trim anyway to avoid corner case when the carryover list becomes empty and we have to block readers for the duration of the full trim. (cherry picked from commit 9f1b51b)
Lazin
force-pushed
the
backport-pr-18056-v23.3.x-62
branch
from
ada7062
to
bc434c6
Compare
abhijat
approved these changes
Apr 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of the #18056
Fixes #18133
The
carryoveris a collection of deletion candidates found during the previous trim. The trim collects full list of objects in the directory and sorts them in LRU order. Then it deletes first N objects. We're saving first M objects that remain in the list after the trim. These objects are deletion candidates.During the next trim the cache service first uses
carryoverlist to a quick cache trim. This trim doesn't need a directory scan and it can quickly decrease bytes/objects counts so other readers could reserve space successfully. The trim doesn't delete objects from thecarryoverlist blindly. It compares access time recorded in the carryover list to the access time stored in theaccesstime_tracker. If the time is different then the object was accessed and the trim is not deleting it during this phase.New configuration option
cloud_storage_cache_trim_carryoveris added. This config option sets the limit on the size of the carryover list. The list is stored on shard 0. The default value is 512. We are storing a file path for every object so this list shouldn't be too big. Even relatively small carryover list might be able to make a difference and prevent readers from being blocked.Backports Required
Release Notes
Improvements