Feature(2.3): add compaction task for delta files

[zipper-meng]

Rationale for this change

Related #1244.

Conclusion

Others

1. DataBlock - Add method split_at(self, index)-> (DataBlock, DataBlock) and intersection(self, time_range) -> Option<DataBlock>.
2. ColumnFile, LevelInfo - Implement std::fmt::Display, also add ColumnFiles<F: AsRef<ColumnFile>>(&[F]) and LevelInfos(&[LevelInfo]) to decrease code when we need to log them.
3. TimeRange, add function compact(time_ranges: &mut Vec<TimeRange>) to compact time ranges.

Compaction

• Changes measurement of compaction from "db", "ts_family", "level" to "db", "ts_family", "in_level", "out_level".
• Add file delta_compact.rs for delta compaction:
  1. Pick delta files and the out_level, get the out_level_max_ts of the out_level.
  2. Now we have many delta files, for each field_id in these files, find grouped blocks which should be merged together: Vec<CompactingBlockMetaGroup>.
  3. For each CompactingBlockMetaGroup, get merge-split blocks(only data in time_range 0..=out_level_max_ts is needed) and write them into files in the out_level, store these files in VersionEdit.
  4. Write a temporary tsm-tombstone file for the delta-file, named xxxxxxx.tombstone.compact.tmp, includes all it's field_ids and time_range 0..=out_level_max_ts, because data in the time_range is already merged into the out_level. Tombstones will be compacted by compact(time_ranges: &mut Vec<TimeRange>) before write.
  5. Write the VersionEdit into summary, and rename these xxxxxxx.tombstone.compact.tmp files to xxxxxxx.tombstone.

Task definition

Some changes in CompactTask:

• Normal(from old CompactTask::Vnode) - Compact the files in the in_level into the out_level.
• Cold(from old CompactTask::ColdVnode) - Flush memcaches and then compact the files in the in_level into the out_level.
• Delta - Compact the files in level-0 into the out_level.

Pick

• Remove trait Picker: Send + Sync + Debug, add two function instead of it:
  • pick_level_compaction(version: Arc<Version>) -> Option<CompactReq>
  • pick_delta_compaction(version: Arc<Version>) -> Option<CompactReq>
• Add function pick_delta_compaction only for picking delta files, it partly merges data in level-0 files to the destination level, the merged data will leave a tombstone file for the source level-0 files.

Scheduler

• Now schedule_compaction() not only check if a tseries family is cold but also check num of level-0 files with config compact_trigger_file_num or constant DEFAULT_COMPACT_TRIGGER_DETLA_FILE_NUM (This change makes unit tests in summary.rs won't stop, so I fixed it by some changes in unit test cases in summary.rs).
• schedule_compaction() holds a shared reference of Arc<TseriesFamily, now we must manually stop it by TseriesFamily::close(). Implementation of Drop for Database will call that method of all it's tseries families.
• Compact job structure compaction::job::CompactProcessor is changed to hold the map of (TsFamilyId, IsDeltaCompaction) to ShouldFlushBeforeCompaction.

TODOs

There may be some logs for debug may be deleted.

Are there any user-facing changes?

No.

[roseboy-liu]

try optimize the algo merge-intervals

[roseboy-liu]

simply the code

[roseboy-liu]

define enum compaction type to replace CompactTaskKey

[Subsegment]

1. pick file instead of pick level(to avoid timerange overlap)
2. modify max level ts before flush(to avoid timerange overlap)

[zipper-meng]

Please see the new PR #1945.