[v23.3.x] Made storage::readers_cache size limited
#16963
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 4278057)
Added property which allows controlling maximum number of readers kept in cache per ntp. The value was previously unbounded which may lead to a situation in which readers cache grew and caused out of memory error. Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 4399ba4)
Added size based eviction to `storage::readers_cache`. Previously the eviction was only time based which may lead to a problem when multiple readers were created for the same NTP. Now with the size based eviction the readers cache size is bounded and will not grow indefinitely. Note on backward compatibility: Previously the size of cache was unbounded. In this commit we change the policy to keep up to 200 readers per ntp. This number is still large enough not cause any issues as it would mean that there are 200 concurrent readers reading a single partition at the same time. Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit a494de6)
Added jitter to readers cache eviction timer to prevent all eviction timers from firing at the same time. Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit d3b07de)
mmaslankaprv
force-pushed
the
vbotbuildovich/backport-16846-v23.3.x-53
branch
from
6b23f49
to
412d589
Compare
StephanDollberg
approved these changes
Mar 8, 2024
This was referenced Mar 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #16846
Fixes: #16947