CQ: Fix shared store crashes #11076
Merged
CQ: Fix shared store crashes #11076
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lhoguin
commented
Apr 24, 2024
lhoguin
referenced
this pull request
Apr 24, 2024
Crashes could happen because compaction would wrongly write over valid messages, or truncate over valid messages, because when looking for messages into the files it would encounter leftover data that made it look like there was a message, which prompted compaction to not look for the real messages hidden within. To avoid this we ensure that there can't be leftover data as a result of compaction. We get this guarantee by blanking data in the holes in the file before we start copying messages closer to the start of the file. This requires us to do a few more writes but we know that the only data in the files at any point are valid messages. Note that it's possible that some of the messages in the files are no longer referenced; that's OK. We filter them out after scanning the file. This was also a good time to merge two almost identical scan functions, and be more explicit about what messages should be dropped after scanning the file (the messages no longer in the ets index and the fan-out messages that ended up re-written in a more recent file).
essen
force-pushed
the
loic-fix-cq-shared-store-crashes
branch
from
4d39cd7 to
fcd011f
Compare
lhoguin
changed the title
1 task
lhoguin
added a commit
that referenced
this pull request
Apr 29, 2024
CQ: Fix shared store crashes (backport #11076)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Crashes could happen because compaction would wrongly write over valid messages, or truncate over valid messages, because when looking for messages into the files it would encounter leftover data that made it look like there was a message, which prompted compaction to not look for the real messages hidden within.
To avoid this we ensure that there can't be leftover data as a result of compaction. We get this guarantee by blanking data in the holes in the file before we start copying messages closer to the start of the file. This requires us to do a few more writes but we know that the only data in the files at any point are valid messages.
Related discussion: #10902
Types of Changes
What types of changes does your code introduce to this project?
Put an
xin the boxes that apply