Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: Add APIs for RawKV GC #4937

Merged
merged 22 commits into from
May 26, 2022

Conversation

AmoebaProtozoa
Copy link
Contributor

Signed-off-by: AmoebaProtozoa 8039876+AmoebaProtozoa@users.noreply.github.com

What problem does this PR solve?

Add APIs for RawKV GC (storage part of #4866).
Issue Number: ref #4865
RFC: tikv/rfcs#90

What is changed and how does it work?

Add APIs for RawKV GC.

Check List

Tests

  • Unit test

Code changes

Side effects
No

Release note

Add APIs for RawKV GC.

Signed-off-by: AmoebaProtozoa <8039876+AmoebaProtozoa@users.noreply.github.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented May 12, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • AndreMouche
  • nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the release-note The PR should write the release note. label May 12, 2022
@AmoebaProtozoa
Copy link
Contributor Author

/cc @nolouch @AndreMouche
PTAL, thanks!

Signed-off-by: AmoebaProtozoa <8039876+AmoebaProtozoa@users.noreply.github.com>
Signed-off-by: AmoebaProtozoa <8039876+AmoebaProtozoa@users.noreply.github.com>
Signed-off-by: AmoebaProtozoa <8039876+AmoebaProtozoa@users.noreply.github.com>
Signed-off-by: AmoebaProtozoa <8039876+AmoebaProtozoa@users.noreply.github.com>
server/storage/endpoint/gc_safe_point.go Outdated Show resolved Hide resolved
server/storage/endpoint/key_path.go Outdated Show resolved Hide resolved
server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
Signed-off-by: AmoebaProtozoa <8039876+AmoebaProtozoa@users.noreply.github.com>
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Copy link
Contributor

@pingyu pingyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM~

@ti-chi-bot
Copy link
Member

@pingyu: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

LGTM~

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@AmoebaProtozoa AmoebaProtozoa mentioned this pull request May 13, 2022
5 tasks
server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
server/storage/endpoint/gc_key_space.go Show resolved Hide resolved
server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Copy link
Member

@AndreMouche AndreMouche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
}

// KeySpaceServiceSafePointPath returns the path of given service's service safe point.
// Path: /key_space/gc_safepoint/{space_id}/service/{service_id}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we know the endpoint types from service_id? such as it's from tidb, raw_kv client, cdc or others?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we can tell by service_id alone,
for example, both TiDB and raw_kv's gc_worker may use gc_worker as their service_id

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So... Is it different in space_id? like cdc_xxx, tidb_xxx, client_xxxx . I would like to know who use the path.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currently, we use spaceID to differentiate key spaces, things like
default_rawkv, default_txnkv, default_tidb
serviceID is used to identify the specific service under that KeySpace, like
cdc, br, etc.

tidb cdc and rawkv cdc may have the same serviceID but different spaceIDs
rawkv cdc and rawkv br have the same spaceID but different serviceIDs

server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
server/storage/endpoint/gc_key_space.go Outdated Show resolved Hide resolved
server/storage/storage_gc_test.go Outdated Show resolved Hide resolved
server/storage/storage_gc_test.go Outdated Show resolved Hide resolved
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Copy link
Member

@AndreMouche AndreMouche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codecov
Copy link

codecov bot commented May 17, 2022

Codecov Report

Merging #4937 (d8dd130) into master (4a9d7c0) will decrease coverage by 0.13%.
The diff coverage is 68.75%.

@@            Coverage Diff             @@
##           master    #4937      +/-   ##
==========================================
- Coverage   75.57%   75.43%   -0.14%     
==========================================
  Files         307      308       +1     
  Lines       30201    30281      +80     
==========================================
+ Hits        22823    22842      +19     
- Misses       5396     5453      +57     
- Partials     1982     1986       +4     
Flag Coverage Δ
unittests 75.43% <68.75%> (-0.14%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
server/storage/storage.go 70.27% <ø> (+5.40%) ⬆️
server/storage/endpoint/gc_key_space.go 66.66% <66.66%> (ø)
server/storage/endpoint/key_path.go 100.00% <100.00%> (ø)
pkg/errs/errs.go 75.00% <0.00%> (-25.00%) ⬇️
pkg/dashboard/adapter/manager.go 74.71% <0.00%> (-11.50%) ⬇️
server/tso/allocator_manager.go 59.73% <0.00%> (-3.33%) ⬇️
server/region_syncer/server.go 83.15% <0.00%> (-3.27%) ⬇️
server/election/lease.go 84.05% <0.00%> (-2.90%) ⬇️
server/schedule/operator_controller.go 84.56% <0.00%> (-1.21%) ⬇️
server/schedulers/evict_leader.go 78.82% <0.00%> (-0.91%) ⬇️
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a9d7c0...d8dd130. Read the comment docs.

Copy link
Contributor

@nolouch nolouch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. better merge it to master after release-6.1 checkout.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 18, 2022
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AndreMouche
Copy link
Member

Could we merge this PR?

@nolouch
Copy link
Contributor

nolouch commented May 25, 2022

We can merge it now, cc @AndreMouche

Copy link
Member

@AndreMouche AndreMouche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 26, 2022
@nolouch
Copy link
Contributor

nolouch commented May 26, 2022

/merge

@ti-chi-bot
Copy link
Member

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 769a40b

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 26, 2022
@ti-chi-bot ti-chi-bot merged commit 86a6821 into tikv:master May 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note The PR should write the release note. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants