Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gc: config gc io limit dynamically #5769

Merged
merged 9 commits into from Nov 18, 2019
Merged

gc: config gc io limit dynamically #5769

merged 9 commits into from Nov 18, 2019

Conversation

youjiali1995
Copy link
Contributor

Signed-off-by: youjiali1995 zlwgx1023@gmail.com

What have you changed?

tikv-ctl can change gc io limit dynamically now. Usage:

tikv-ctl --host=host modify-tikv-config -m server -n gc.max_write_bytes_per_sec -v 10MB

It's used to limit the write flow when gc influences normal requests, and remove the limit when there is no influence.

What is the type of the changes?

  • Improvement (a change which is an improvement to an existing feature)

How is the PR tested?

  • Unit test
  • Manual test (add detailed scripts or steps below)
    image

Does this PR affect documentation (docs) or should it be mentioned in the release notes?

Yes. I will file one.

Does this PR affect tidb-ansible?

No

Refer to a related PR or issue link (optional)

#5263

@youjiali1995
gc: config gc io limit dynamically
30cf4ed 
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
@youjiali1995 youjiali1995 added type/enhancement Type: Issue - Enhancement component/tikv-ctl Component: TiKV Control component/server Component: Server labels Oct 31, 2019
@youjiali1995 youjiali1995 added needs-cherry-pick-release-3.0 Type: Need cherry pick to release 3.0 needs-cherry-pick-release-3.1 Type: Need cherry pick to release 3.1 labels Oct 31, 2019
@youjiali1995
Copy link
Contributor Author

youjiali1995 commented Oct 31, 2019 •
edited

The implementation may be changed totally because many configurations will be modified dynamically this quarter. So only gc io limiter is supported in this PR.

@youjiali1995
Copy link
Contributor Author

/run-integration-common-test

@overvenus
Copy link
Member

FYI @NingLin-P , this PR changes GC config dynamically.

sticnarf
sticnarf reviewed Nov 8, 2019
View reviewed changes
src/server/gc_worker.rs Outdated Show resolved Hide resolved
@sticnarf
Copy link
Contributor

sticnarf commented Nov 8, 2019

Is it possible that accumulated GC tasks blocks GCTask::ChangeIoLimit from executing?

For example, originally we set the value too small and find the speed too slow. Then, we want to use tikv-ctl to make the limit larger. However, the previous GC task runs too slow so that the change limit task cannot run. Can we solve the problem?

@youjiali1995
Copy link
Contributor Author

Is it possible that accumulated GC tasks blocks GCTask::ChangeIoLimit from executing?

For example, originally we set the value too small and find the speed too slow. Then, we want to use tikv-ctl to make the limit larger. However, the previous GC task runs too slow so that the change limit task cannot run. Can we solve the problem?

Yes, it's possible. I can create the IoLimiter protected by a mutex in the GCWorker and pass it to the GCRunner. Then we can change the limit in the GCWorker.

@youjiali1995
Mutex protected limiter
b785b0d 
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
@youjiali1995
Copy link
Contributor Author

/test

@sre-bot sre-bot mentioned this pull request Nov 13, 2019
Merged
@youjiali1995
Copy link
Contributor Author

@MyonKeminta @sticnarf PTAL.

@youjiali1995 youjiali1995 mentioned this pull request Nov 18, 2019
Merged
@youjiali1995 youjiali1995 added the priority/release-blocker This PR blocks a release. Please review it ASAP. label Nov 18, 2019
sticnarf
sticnarf reviewed Nov 18, 2019
View reviewed changes
Copy link
Contributor

@sticnarf sticnarf left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest using RwLock because the limiter is accessed frequently but seldom changes. (I know the time acquring lock is really small compared to the whole gc operation. It's more reasonable :)

The rest LGTM

MyonKeminta
MyonKeminta approved these changes Nov 18, 2019
View reviewed changes
Copy link
Contributor

@MyonKeminta MyonKeminta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@youjiali1995
Copy link
Contributor Author

I suggest using RwLock because the limiter is accessed frequently but seldom changes. (I know the time acquring lock is really small compared to the whole gc operation. It's more reasonable :)

The rest LGTM

@sticnarf I think changing limit and requesting quota are both write operations.

@sticnarf
Copy link
Contributor

But IOLimiter::request only needs &self...

sticnarf
sticnarf approved these changes Nov 18, 2019
View reviewed changes
Copy link
Contributor

@sticnarf sticnarf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway. It's not so important.

LGTM

@youjiali1995
Copy link
Contributor Author

/merge

@sre-bot sre-bot added the status/can-merge Status: Can merge to base branch label Nov 18, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Nov 18, 2019

/run-all-tests

@youjiali1995
Copy link
Contributor Author

/test

@youjiali1995
Copy link
Contributor Author

/merge

@sre-bot
Copy link
Contributor

sre-bot commented Nov 18, 2019

/run-all-tests

@sre-bot
Copy link
Contributor

sre-bot commented Nov 18, 2019

@youjiali1995 merge failed.

@youjiali1995
Copy link
Contributor Author

/test

4 similar comments
@youjiali1995
Copy link
Contributor Author

/test

@youjiali1995
Copy link
Contributor Author

/test

@youjiali1995
Copy link
Contributor Author

/test

@youjiali1995
Copy link
Contributor Author

/test

@youjiali1995
Copy link
Contributor Author

/merge

@sre-bot
Copy link
Contributor

sre-bot commented Nov 18, 2019

Your auto merge job has been accepted, waiting for 5903

@sre-bot
Copy link
Contributor

sre-bot commented Nov 18, 2019

/run-all-tests

@sre-bot sre-bot merged commit 1d5e120 into tikv:master Nov 18, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Nov 18, 2019

cherry pick to release-3.0 failed

@sre-bot
Copy link
Contributor

sre-bot commented Nov 18, 2019

cherry pick to release-3.1 failed

b41sh pushed a commit to b41sh/tikv that referenced this pull request Nov 18, 2019
@youjiali1995 @b41sh
gc: config gc io limit dynamically (tikv#5769)
d36cd34 
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
youjiali1995 added a commit to youjiali1995/tikv that referenced this pull request Nov 19, 2019
@youjiali1995
gc: config gc io limit dynamically (tikv#5769)
c3c912a 
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
@youjiali1995 youjiali1995 mentioned this pull request Nov 19, 2019
Merged
youjiali1995 added a commit to youjiali1995/tikv that referenced this pull request Nov 19, 2019
@youjiali1995
gc: config gc io limit dynamically (tikv#5769)
5b6b36d 
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
@youjiali1995 youjiali1995 deleted the dynamically-config-gc branch November 19, 2019 08:37
sre-bot pushed a commit that referenced this pull request Nov 20, 2019
@youjiali1995 @sre-bot
gc: config gc io limit dynamically (#5769) (#5957)
a285a54 
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
youjiali1995 added a commit to youjiali1995/tikv that referenced this pull request Nov 25, 2019
@youjiali1995
gc: config gc io limit dynamically (tikv#5769) (tikv#5957)
043d003 
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
@youjiali1995 youjiali1995 mentioned this pull request Nov 25, 2019
Merged
sre-bot pushed a commit that referenced this pull request Nov 25, 2019
@youjiali1995 @sre-bot
gc: config gc io limit dynamically (#5769) (#5957) (#6036)
cedef10 
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
hawkingrei pushed a commit to hawkingrei/tikv that referenced this pull request Dec 1, 2019
@youjiali1995
gc: config gc io limit dynamically (tikv#5769)
4e4e1ac 
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
@youjiali1995 youjiali1995 mentioned this pull request Nov 19, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MyonKeminta MyonKeminta MyonKeminta approved these changes

@sticnarf sticnarf sticnarf approved these changes

Assignees

@AndreMouche AndreMouche

@MyonKeminta MyonKeminta

@sticnarf sticnarf

Labels
component/server Component: Server component/tikv-ctl Component: TiKV Control needs-cherry-pick-release-3.0 Type: Need cherry pick to release 3.0 needs-cherry-pick-release-3.1 Type: Need cherry pick to release 3.1 priority/release-blocker This PR blocks a release. Please review it ASAP. status/can-merge Status: Can merge to base branch type/enhancement Type: Issue - Enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@youjiali1995 @overvenus @sticnarf @sre-bot @MyonKeminta @AndreMouche