New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature(2.3): add compaction task for delta files #1945
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zipper-meng
changed the title
Feature/lts 2.3/delta compaction v3
Feature(2.3): add compaction task for delta files
Jan 19, 2024
|
roseboy-liu
force-pushed
the
feature/lts_2.3/delta_compaction_v3
branch
from
January 24, 2024 02:48
7d8773e
to
3a7a957
Compare
zipper-meng
force-pushed
the
feature/lts_2.3/delta_compaction_v3
branch
from
January 24, 2024 02:52
3a7a957
to
c7e7ea8
Compare
roseboy-liu
force-pushed
the
feature/lts_2.3/delta_compaction_v3
branch
from
January 24, 2024 03:16
c7e7ea8
to
e36f193
Compare
zipper-meng
force-pushed
the
feature/lts_2.3/delta_compaction_v3
branch
from
January 25, 2024 12:29
6b17266
to
14d5f0d
Compare
…fix(e2e_test): fix windows build fail
zipper-meng
force-pushed
the
feature/lts_2.3/delta_compaction_v3
branch
from
January 26, 2024 09:15
14d5f0d
to
a5689ab
Compare
roseboy-liu
previously approved these changes
Jan 31, 2024
roseboy-liu
approved these changes
Feb 1, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Related #1244.
Conclusion
Others
DataBlock
- Add methodsplit_at(self, index)-> (DataBlock, DataBlock)
andintersection(self, time_range) -> Option<DataBlock>
.ColumnFile
,LevelInfo
- Implementstd::fmt::Display
, also addColumnFiles<F: AsRef<ColumnFile>>(&[F])
andLevelInfos(&[LevelInfo])
to decrease code when we need to log them.TimeRange
, add functioncompact(time_ranges: &mut Vec<TimeRange>)
to compact time ranges.TimeRanges
, add methodpush(&mut self, time_range: TimeRange)
,extend_from_slice(&mut self, time_ranges: &[TimeRange])
to modify aTimeRanges
.Compaction
"db", "ts_family", "level"
to"db", "ts_family", "in_level", "out_level"
.delta_compact.rs
for delta compaction:out_level_max_ts
of the out_level.Vec<CompactingBlockMetaGroup>
.CompactingBlockMetaGroup
, get merge-split blocks(only data in time_range0..=out_level_max_ts
is needed) and write them into files in the out_level, store these files inVersionEdit
.xxxxxxx.tombstone.compact.tmp
, includes all it's field_ids and time_range0..=out_level_max_ts
, because data in the time_range is already merged into the out_level. Tombstones will be compacted bycompact(time_ranges: &mut Vec<TimeRange>)
before write.VersionEdit
into summary, and rename thesexxxxxxx.tombstone.compact.tmp
files toxxxxxxx.tombstone
.Task definition
Some changes in
CompactTask
:Some changes in
CompactReq
:lv0_files: Option<Vec<Arc<ColumnFile>>>
to store files in delta compaction.Pick
Picker: Send + Sync + Debug
, add two function instead of it:pick_level_compaction(compact_task: CompactTask, version: Arc<Version>) -> Option<CompactReq>
async pick_delta_compaction(compact_task: CompactTask, version: Arc<Version>) -> Option<CompactReq>
pick_delta_compaction
only for picking delta files with a tsm file to partly merges data in level-0 files into the destination level, the merged data will leave a tombstone file for the source level-0 files.Scheduler
schedule_compaction()
not only check if a tseries family is cold but also check num of level-0 files with configcompact_trigger_file_num
or constantDEFAULT_COMPACT_TRIGGER_DETLA_FILE_NUM
(This change makes unit tests in summary.rs won't stop, so I fixed it by some changes in unit test cases in summary.rs).schedule_compaction()
holds a shared reference ofArc<TseriesFamily
, now we must manually stop it byTseriesFamily::close()
. Implementation ofDrop
forDatabase
will call that method of all it's tseries families.compaction::job::CompactProcessor
is changed to hold the map of(TsFamilyId, IsDeltaCompaction)
toShouldFlushBeforeCompaction
.Tombstone
Add tombstone type
ALL
, now a tombstone can not only specifies an excluded time range for a field but can also specify the time range for all fields.As we know the tombstone file is record file, and each record is like:
We can control the decoding of tombstones by
data_type
in record header.Tombstone record v1
The v1 tombstone record, I think it's sparse.
Tombstone record v2
Each record may contains many fields and time ranges.
TODOs
There may be some logs for debug may be deleted.
Are there any user-facing changes?
No.