Backport "Fix archival upload data corruption" #3293 #3309
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The bug manifested as a truncated segment in S3. This happens only if the cloud_storage_segment_max_upload_interval_sec option is enabled. This is what causes the bug. The code reads the size of the segment at the begining and uses it to set the initial content-length value. Then it performs an asynchronous operation. During this asynchronous operation the size of the segment might change but the content-length value will stay the same. After that we're getting the segment size once again and here we can have a larger value. This value is evenutally used to compute the size of the tail region of the segment which shouldn't be uploaded (file size minus the position where the upload should stop). Because the code uses the new file size here it subtracts larger value which results in truncation of the segment. The fix is to just read the size of the segment once. (cherry picked from commit 3a25251)
When the cloud_storage_segment_max_upload_interval_sec option is used archival subsystem tries to read from segment to find the right locations of the begining and the end of the segment. This reads didn't use the right io_priority_class that the rest of the archival uses. This commit fixes this by propagating the right io_priority_class. (cherry picked from commit 5285353)
If transform_stream throws we're handling an exception correctly by closing the streams and propagating the exception. But if it returns an error we might just create an incorrect upload. This commit changes this behavior by throwing exception in case of any error. (cherry picked from commit e12105b)
ztlpn
approved these changes
Dec 17, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cover letter
Backport PR #3293