[v23.1.x] cloud_storage: tolerate exceptions in the hydration loop #10917
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
May 22, 2023 12:30
This commit extends `remote_segment` to track the status of the hydration loop. Previously, if a hydration is requested via `remote_segment::hydrate()` whilst the loop is not running, the request will stall until the client times it out. This also prevents Redpanda from cleaing up the Kafka connection properly. With this patch, the client gets an error instead an the connection is released.
Previously, any unexpected exceptions thrown in the hydration loop caused it to stop, leaving the `remote_segment` in an undetermined state. We have seen this in the field when permissions for the cloud storage cache prevent writes. With this patch, these exceptions are tolerated and hydration can be retried.
When there are multiple consumers in a fanout with a deep queue and some consumers have progressed ahead, if a slow consumer exits early by detaching, this causes many buffers in the front of the queue to have their bitmasks set. When a fast consumer then does the next read, it finds a buffer to read from, sets its own bit in the mask, and then finds the buffer eligible to pop. But the invariant that the popped buffer should be at the front of the queue is now broken. The changes here clean up the front of the queue before setting any masks, so that any buffers which have their mask already set to all bits are popped out before we proceed with the current read. This maintains the invariant. (cherry picked from commit bc13bca)
andrwng
approved these changes
May 22, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #10828 and #9786.
The fix in #10828 unveiled a deadlock in input_fanout_stream which was fixed in #9786.