gc_worker: make unsafe detroy range compatible with multi-rocks db #13485

SpadeA-Tang · 2022-09-19T01:43:56Z

What is changed and how it works?

Issue Number: Close #13406

What's Changed:

This PR makes unsafe-destroy-range compatible with multi-rocks db.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test

Release note

None

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

ti-chi-bot · 2022-09-19T01:43:58Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

MyonKeminta
tonyxuqqi

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

MyonKeminta · 2022-09-19T03:42:56Z

src/server/gc_worker/gc_worker.rs

+        } else {
+            let cfs = &[CF_LOCK, CF_DEFAULT, CF_WRITE];
+            let keys = vec![start_key.clone(), end_key.clone()];
+            let regions = get_regions_for_gc(self.store_id, &keys, regions_provider)?;


Does this return all regions covered by the range?
It seems to me that get_regions_for_gc is only responsible for returning regions that contains one of the keys in the parameter keys. Otherwise the function get_regions_for_gc is better to be changed to something like get_regions_in_range.

Generally, it is. But sometimes keys may only contain one key, so in this case, it is not appropriate to call range?

It can be considered as a range [key, key.append(0)) where key is in raw format, thus there is at most one key in the range.

So，just rename the method? Or I should change the keys to Range type?

Maybe you can rename it to something like get_regions_for_range_of_keys without changing the parameter it accepts. And explain in comments that the function is used to find all regions in the range inferred by a list of sorted keys.

Thanks for the suggestion.

MyonKeminta · 2022-09-19T04:00:22Z

src/server/gc_worker/gc_worker.rs

+            let count = regions.len();
+
+            let mut region_modifies = HashMap::default();
+            for i in 0..count {


consider: for (i, region) in regions.into_iter().enumerate()

Good suggestion.

MyonKeminta · 2022-09-19T04:07:02Z

src/server/gc_worker/gc_worker.rs

+
+                let mut modifies = Vec::new();
+                for cf in cfs {
+                    modifies.push(Modify::DeleteRange(


cc @BusyJay
Is it possible to delete large ranges in a safe way (instead of bypassing raft) after supporting multi-rocks?

Yes, just send them like normal commands.

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

tonyxuqqi · 2022-09-19T21:39:55Z

src/server/gc_worker/gc_worker.rs

-        // to do it in somewhere of the same layer with apply_worker.
-        let start_data_key = keys::data_key(start_key.as_encoded());
-        let end_data_key = keys::data_end_key(end_key.as_encoded());
+        if let Some(local_storage) = self.engine.kv_engine() {


please add comment this branch is for single rocksdb case.

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

MyonKeminta · 2022-09-20T08:58:32Z

/merge

ti-chi-bot · 2022-09-20T08:58:33Z

@MyonKeminta: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-09-20T08:58:36Z

This pull request has been accepted and is ready to merge.

Commit hash: bba951a

ti-chi-bot · 2022-09-20T08:58:49Z

@SpadeA-Tang: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

SpadeA-Tang added 6 commits September 8, 2022 10:03

update

f3603a1

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

init

8726f06

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

update

3a095c8

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

resolve merge

c2491fb

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

add test: why get_cf cannot get value while get on region can

a213645

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

finish test

f0f6cd3

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

ti-chi-bot added release-note contribution Type: PR - From contributors size/L release-note-none and removed release-note labels Sep 19, 2022

MyonKeminta reviewed Sep 19, 2022

View reviewed changes

SpadeA-Tang and others added 3 commits September 19, 2022 12:59

update

5576ccb

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

Merge branch 'master' into multi-unsafe-destroy

124e434

update

2c39334

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

ti-chi-bot added size/XL and removed size/L labels Sep 19, 2022

tonyxuqqi reviewed Sep 19, 2022

View reviewed changes

tonyxuqqi approved these changes Sep 19, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Status: PR - There is already 1 approval label Sep 19, 2022

add comment

bba951a

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

SpadeA-Tang requested a review from MyonKeminta September 20, 2022 01:43

MyonKeminta approved these changes Sep 20, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Status: PR - There are already 2 approvals and removed status/LGT1 Status: PR - There is already 1 approval labels Sep 20, 2022

ti-chi-bot added the status/can-merge Status: Can merge to base branch label Sep 20, 2022

Merge branch 'master' into multi-unsafe-destroy

b17011e

ti-chi-bot merged commit 2e11398 into tikv:master Sep 20, 2022

ti-chi-bot added this to the Pool milestone Sep 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gc_worker: make unsafe detroy range compatible with multi-rocks db #13485

gc_worker: make unsafe detroy range compatible with multi-rocks db #13485

SpadeA-Tang commented Sep 19, 2022 •

edited

ti-chi-bot commented Sep 19, 2022 •

edited

MyonKeminta Sep 19, 2022

SpadeA-Tang Sep 19, 2022

MyonKeminta Sep 19, 2022

SpadeA-Tang Sep 19, 2022

MyonKeminta Sep 19, 2022

SpadeA-Tang Sep 19, 2022

MyonKeminta Sep 19, 2022

SpadeA-Tang Sep 19, 2022

MyonKeminta Sep 19, 2022

BusyJay Sep 19, 2022

tonyxuqqi Sep 19, 2022

SpadeA-Tang Sep 20, 2022

MyonKeminta commented Sep 20, 2022

ti-chi-bot commented Sep 20, 2022

ti-chi-bot commented Sep 20, 2022

ti-chi-bot commented Sep 20, 2022

gc_worker: make unsafe detroy range compatible with multi-rocks db #13485

gc_worker: make unsafe detroy range compatible with multi-rocks db #13485

Conversation

SpadeA-Tang commented Sep 19, 2022 • edited

What is changed and how it works?

Related changes

Check List

Release note

ti-chi-bot commented Sep 19, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MyonKeminta commented Sep 20, 2022

ti-chi-bot commented Sep 20, 2022

ti-chi-bot commented Sep 20, 2022

ti-chi-bot commented Sep 20, 2022

SpadeA-Tang commented Sep 19, 2022 •

edited

ti-chi-bot commented Sep 19, 2022 •

edited