[v24.1.x] cloud_storage: Implement "carryover" cache trimming mechanism #18134
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(cherry picked from commit 57985b7)
Add new parameter that controls cache carryover behavior. (cherry picked from commit 940fcd4)
The "carryover" behavior allows cache to use information from the previous trim to quickly trim the cache without scanning the whole directory. This allows cache to avoid blocking readers. In a situation when the cache cntains very large number of files the recursive directory walk could take few minutes. We're not allowing number of objects stored in the cache to overshoot so all the readers are blocked until the walk is finished. This commit adds new "carryover" trim mechanism which is running before the normal trim and uses information obtained during the previous fast or full trim to delete some objects wihtout walking the directory tree. (cherry picked from commit 5ece9d4)
(cherry picked from commit fdf3498)
(cherry picked from commit 1256594)
Change the configuration parameter and treat the value as number of bytes that we can use to store carryover data. (cherry picked from commit e1c30bc)
Reserve memory units for the carryover mechanism in materialized_resrouces. (cherry picked from commit 0f84fdb)
(cherry picked from commit 6b57e0c)
In case if carryover trim was able to release enough space start trim in the background and return early. This unblocks the hydration that reserved space and triggered the trim. We need to run normal trim anyway to avoid corner case when the carryover list becomes empty and we have to block readers for the duration of the full trim. (cherry picked from commit 9f1b51b)
Lazin
approved these changes
Apr 29, 2024
Feediver1
reviewed
Apr 29, 2024
| sm::make_counter( | ||
| "carryover_trims", | ||
| [this] { return _carryover_trims; }, | ||
| sm::description("Number of times we invoked carryover trim.")) |
There was a problem hiding this comment.
Suggested change
| sm::description("Number of times we invoked carryover trim.")) | |
| sm::description("Number of times carryover trim has been invoked.")) |
Feediver1
reviewed
Apr 29, 2024
| vlog( | ||
| cst_log.info, | ||
| "Failed to reserve {} units for the cache carryover " | ||
| "mechanism because tiered-storage is likely under memory " |
There was a problem hiding this comment.
Suggested change
| "mechanism because tiered-storage is likely under memory " | |
| "mechanism because Tiered Storage is likely under memory " |
Feediver1
reviewed
Apr 29, 2024
| @@ -2359,6 +2359,18 @@ configuration::configuration() | |||
| // Enough for a >1TiB cache of 16MiB objects. Decrease this in case | |||
| // of issues with trim performance. | |||
| 100000) | |||
| , cloud_storage_cache_trim_carryover_bytes( | |||
| *this, | |||
| "cloud_storage_cache_trim_carryover_bytes", | |||
Feediver1
approved these changes
Apr 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #18056