Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(storage): hummock support ttl ability with compaction_filter #3438

Merged
merged 7 commits into from
Jun 27, 2022

Conversation

Li0k
Copy link
Contributor

@Li0k Li0k commented Jun 23, 2022

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

hummock support ttl ability with compaction_filter

  • bitflags to control compaction_filter
  • multi_compaction_filter (state_clean_up and ttl)
  • transfer table_opiton and mask to compactor for compaction_filter
  • unit-test for ttl_compaction_filter

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)

@Li0k Li0k force-pushed the li0k/feat_table_option_for_compaction_group branch from 032c782 to ccd14ad Compare June 24, 2022 03:10
@Li0k Li0k force-pushed the li0k/feat_hummock_ttl_compactin branch from 9f66533 to ec9c2d7 Compare June 24, 2022 03:11
@Li0k Li0k force-pushed the li0k/feat_table_option_for_compaction_group branch from ccd14ad to c8de7ba Compare June 24, 2022 05:20
Base automatically changed from li0k/feat_table_option_for_compaction_group to main June 24, 2022 07:14
@Li0k Li0k force-pushed the li0k/feat_hummock_ttl_compactin branch from ec9c2d7 to bee1f68 Compare June 24, 2022 14:39
@Li0k Li0k marked this pull request as ready for review June 24, 2022 14:40
@Li0k Li0k added the component/storage Storage label Jun 24, 2022
match table_id {
Some(table_id) => {
let ttl = self.table_id_to_ttl[&table_id];
let epoch = get_epoch(key);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Get epoch and table_id in once call for performance

Copy link
Contributor Author

@Li0k Li0k Jun 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we need a new function to get epoch and table_id in once call ? (not exist now)

fn extract_table_id_and_epoch(full_key: &[u8]) -> (Option<u32>, Epoch)

in this case it means that we no need to get_epoch if table_id not exist

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. A new function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix it

@Li0k Li0k force-pushed the li0k/feat_hummock_ttl_compactin branch 2 times, most recently from 3df6c68 to 5244321 Compare June 25, 2022 09:28
@codecov
Copy link

codecov bot commented Jun 25, 2022

Codecov Report

Merging #3438 (0ae198e) into main (7eca500) will increase coverage by 0.04%.
The diff coverage is 96.21%.

@@            Coverage Diff             @@
##             main    #3438      +/-   ##
==========================================
+ Coverage   74.37%   74.42%   +0.04%     
==========================================
  Files         769      769              
  Lines      107047   107270     +223     
==========================================
+ Hits        79621    79838     +217     
- Misses      27426    27432       +6     
Flag Coverage Δ
rust 74.42% <96.21%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/meta/src/hummock/compaction_group/manager.rs 90.47% <ø> (+0.68%) ⬆️
src/common/src/config.rs 46.42% <75.00%> (+1.05%) ⬆️
src/meta/src/hummock/hummock_manager.rs 85.72% <93.33%> (+0.10%) ⬆️
src/storage/src/hummock/compactor_tests.rs 93.88% <96.02%> (+1.48%) ⬆️
src/storage/src/hummock/compactor.rs 77.67% <98.07%> (+1.42%) ⬆️
...c/meta/src/hummock/compaction/compaction_config.rs 92.68% <100.00%> (+0.18%) ⬆️
src/meta/src/hummock/compaction/level_selector.rs 96.63% <100.00%> (+0.01%) ⬆️
src/meta/src/hummock/compaction/mod.rs 80.20% <100.00%> (+0.20%) ⬆️
src/meta/src/hummock/compaction_group/mod.rs 89.89% <100.00%> (ø)
src/meta/src/hummock/compactor_manager.rs 98.87% <100.00%> (+0.01%) ⬆️
... and 8 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@Li0k Li0k force-pushed the li0k/feat_hummock_ttl_compactin branch from 5244321 to c1719bb Compare June 27, 2022 03:11
Copy link
Contributor

@Little-Wallace Little-Wallace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Li0k Li0k added mergify/can-merge Indicates that the PR can be added to the merge queue and removed mergify/can-merge Indicates that the PR can be added to the merge queue labels Jun 27, 2022
@Li0k Li0k added the mergify/can-merge Indicates that the PR can be added to the merge queue label Jun 27, 2022
@mergify mergify bot merged commit 6c84273 into main Jun 27, 2022
@mergify mergify bot deleted the li0k/feat_hummock_ttl_compactin branch June 27, 2022 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/storage Storage mergify/can-merge Indicates that the PR can be added to the merge queue type/feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants