Cut output files at compaction cursors #10227
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
littlepig2013
force-pushed
the
cut_output_at_cmpct_cursors
branch
5 times, most recently
from
cf949b1
to
0ecab03
Compare
littlepig2013
changed the title
jay-zhuang
reviewed
Jun 23, 2022
littlepig2013
force-pushed
the
cut_output_at_cmpct_cursors
branch
from
0ecab03
to
15f02dd
Compare
littlepig2013
force-pushed
the
cut_output_at_cmpct_cursors
branch
from
15f02dd
to
d7f26fe
Compare
littlepig2013
force-pushed
the
cut_output_at_cmpct_cursors
branch
from
d7f26fe
to
83ce16f
Compare
|
@littlepig2013 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
ajkr
reviewed
Jun 30, 2022
| const InternalKey output_split_key = c->GetOutputSplitKey(); | ||
| if (output_split_key.Valid()) { | ||
| output_split_user_key = ExtractUserKey(output_split_key.Encode()); | ||
| bounds.emplace_back(output_split_key.Encode()); |
There was a problem hiding this comment.
Confusingly, InternalKey::Encode() does not make a copy. I think that makes it unsafe for use in bounds.
There was a problem hiding this comment.
Replace output_split_key in 648 with c->GetOutputSplitKey() should be fine?
facebook-github-bot
pushed a commit
that referenced
this pull request
Jul 27, 2022
Summary: Update HISTORY.md for CompactionPri::kRoundRobin. Detailed implementation can be found in [PR10107](#10107), [PR10227](#10227), [PR10250](#10250), [PR10278](#10278), [PR10316](#10316), and [PR10341](#10341) Pull Request resolved: #10421 Reviewed By: ajkr Differential Revision: D38194070 Pulled By: littlepig2013 fbshipit-source-id: 4ce153dc0bf22cd865d09c5429955023dbc90f37
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
The files behind the compaction cursor contain newer data than the files ahead of it. If a compaction writes a file that spans from before its output level’s cursor to after it, then data before the cursor will be contaminated with the old timestamp from the data after the cursor. To avoid this, we can split the output file into two – one entirely before the cursor and one entirely after the cursor. Note that, in rare cases, we DO NOT need to cut the file if it is a trivial move since the file will not be contaminated by older files. In such case, the compact cursor is not guaranteed to be the boundary of the file, but it does not hurt the round-robin selection process.
Test Plan: Add 'RoundRobinCutOutputAtCompactCursor' unit test in
db_compaction_testTask: T122216351