*: optimize heartbeat process with concurrent runner - part 1 #8053

nolouch · 2024-04-10T10:09:56Z

What problem does this PR solve?

Issue Number: Ref #7897

Part of #8052

What is changed and how does it work?

*: Optimize the heartbeat process with the concurrent runner 
- add a switch about the runner
- move some tasks out, not block the heartbeat process

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Release note

None.

ti-chi-bot · 2024-04-10T10:09:58Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

HuSharp
JmPotato

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: nolouch <nolouch@gmail.com>

pkg/mcs/scheduling/server/cluster.go

codecov · 2024-04-11T07:54:48Z

Codecov Report

Merging #8053 (85e6950) into master (340c58c) will increase coverage by 0.00%.
The diff coverage is 79.73%.

Additional details and impacted files

@@           Coverage Diff            @@
##           master    #8053    +/-   ##
========================================
  Coverage   77.36%   77.36%            
========================================
  Files         468      469     +1     
  Lines       60943    61092   +149     
========================================
+ Hits        47146    47262   +116     
- Misses      10252    10278    +26     
- Partials     3545     3552     +7

Flag	Coverage Δ
unittests	`77.36% <79.73%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

pkg/cluster/cluster.go

pkg/core/region.go

server/cluster/cluster.go

Signed-off-by: nolouch <nolouch@gmail.com>

…async_1

Signed-off-by: nolouch <nolouch@gmail.com>

nolouch · 2024-04-15T02:12:11Z

PTAL @HuSharp @JmPotato

pkg/core/region.go

Signed-off-by: nolouch <nolouch@gmail.com>

…async_1

Signed-off-by: nolouch <nolouch@gmail.com>

pkg/core/context.go

rleungx · 2024-04-15T08:02:03Z

pkg/core/region.go

+		debug, info := d, i
+		if taskRunner != nil {
+			debug = func(msg string, fields ...zap.Field) {
+				taskRunner.RunTask(


Is the error needed?

I will add metrics about this error in next PR.

rleungx · 2024-04-15T08:05:50Z

pkg/mcs/scheduling/server/cluster.go

@@ -81,6 +91,9 @@ func NewCluster(parentCtx context.Context, persistConfig *config.PersistConfig,
 		storage:           storage,
 		clusterID:         clusterID,
 		checkMembershipCh: checkMembershipCh,
+
+		taskRunner:           ratelimit.NewConcurrentRunner(hbConcurrentRunner, time.Minute),


Will it cause the memory to increase a lot if there are many hb requests?

it depends on the number of waiting queue closures (tasks). we suppose each costs 128 bytes, 1M is about 128MB. but it's released very quickly I think.

Signed-off-by: nolouch <nolouch@gmail.com>

nolouch · 2024-04-15T09:12:03Z

/merge

ti-chi-bot · 2024-04-15T09:12:06Z

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2024-04-15T09:12:09Z

This pull request has been accepted and is ready to merge.

Commit hash: 85e6950

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none labels Apr 10, 2024

ti-chi-bot bot requested review from HunDunDM and JmPotato April 10, 2024 10:10

ti-chi-bot bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Apr 10, 2024

nolouch force-pushed the optimize-heartbeat/async_1 branch from 0b177f8 to 1247655 Compare April 10, 2024 10:12

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 10, 2024

nolouch force-pushed the optimize-heartbeat/async_1 branch 2 times, most recently from f4380e2 to 8961f5b Compare April 10, 2024 10:14

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label Apr 10, 2024

nolouch force-pushed the optimize-heartbeat/async_1 branch 5 times, most recently from e670b0b to a75ae5f Compare April 11, 2024 06:28

*: optimize heartbeat process with async runner

39dfc43

Signed-off-by: nolouch <nolouch@gmail.com>

nolouch force-pushed the optimize-heartbeat/async_1 branch from a75ae5f to 39dfc43 Compare April 11, 2024 06:36

nolouch requested review from rleungx and removed request for HunDunDM April 11, 2024 06:41

nolouch force-pushed the optimize-heartbeat/async_1 branch from f6c36d3 to 39dfc43 Compare April 11, 2024 07:15

nolouch requested a review from HuSharp April 11, 2024 07:15

fix

06a1903

Signed-off-by: nolouch <nolouch@gmail.com>

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 11, 2024

HuSharp reviewed Apr 11, 2024

View reviewed changes

pkg/mcs/scheduling/server/cluster.go Outdated Show resolved Hide resolved

JmPotato reviewed Apr 11, 2024

View reviewed changes

pkg/cluster/cluster.go Outdated Show resolved Hide resolved

pkg/core/region.go Outdated Show resolved Hide resolved

HuSharp reviewed Apr 11, 2024

View reviewed changes

server/cluster/cluster.go Show resolved Hide resolved

address

d22d9c4

Signed-off-by: nolouch <nolouch@gmail.com>

nolouch changed the title *: optimize heartbeat process with async runner - part 1 *: optimize heartbeat process with concurrent runner - part 1 Apr 11, 2024

nolouch added 3 commits April 11, 2024 19:17

rename

08f8c58

Signed-off-by: nolouch <nolouch@gmail.com>

Merge remote-tracking branch 'origin/master' into optimize-heartbeat/…

2a91f8c

…async_1

rename

4de6106

Signed-off-by: nolouch <nolouch@gmail.com>

HuSharp reviewed Apr 15, 2024

View reviewed changes

pkg/core/region.go Show resolved Hide resolved

nolouch added 3 commits April 15, 2024 13:19

address context

9991d28

Signed-off-by: nolouch <nolouch@gmail.com>

Merge remote-tracking branch 'origin/master' into optimize-heartbeat/…

13c66bc

…async_1

fix race

d85ac59

Signed-off-by: nolouch <nolouch@gmail.com>

JmPotato approved these changes Apr 15, 2024

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 15, 2024

HuSharp approved these changes Apr 15, 2024

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 15, 2024

rleungx reviewed Apr 15, 2024

View reviewed changes

address

85e6950

Signed-off-by: nolouch <nolouch@gmail.com>

rleungx approved these changes Apr 15, 2024

View reviewed changes

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 15, 2024

ti-chi-bot bot merged commit 0214778 into tikv:master Apr 15, 2024
26 checks passed

nolouch deleted the optimize-heartbeat/async_1 branch April 15, 2024 09:15

nolouch mentioned this pull request Apr 15, 2024

Enhance heartbeat processing capabilities #7897

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: optimize heartbeat process with concurrent runner - part 1 #8053

*: optimize heartbeat process with concurrent runner - part 1 #8053

nolouch commented Apr 10, 2024 •

edited

ti-chi-bot bot commented Apr 10, 2024 •

edited

codecov bot commented Apr 11, 2024 •

edited

nolouch commented Apr 15, 2024

rleungx Apr 15, 2024

nolouch Apr 15, 2024

rleungx Apr 15, 2024 •

edited

nolouch Apr 15, 2024

nolouch commented Apr 15, 2024

ti-chi-bot bot commented Apr 15, 2024

ti-chi-bot bot commented Apr 15, 2024

*: optimize heartbeat process with concurrent runner - part 1 #8053

*: optimize heartbeat process with concurrent runner - part 1 #8053

Conversation

nolouch commented Apr 10, 2024 • edited

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Apr 10, 2024 • edited

codecov bot commented Apr 11, 2024 • edited

Codecov Report

nolouch commented Apr 15, 2024

rleungx Apr 15, 2024

Choose a reason for hiding this comment

nolouch Apr 15, 2024

Choose a reason for hiding this comment

rleungx Apr 15, 2024 • edited

Choose a reason for hiding this comment

nolouch Apr 15, 2024

Choose a reason for hiding this comment

nolouch commented Apr 15, 2024

ti-chi-bot bot commented Apr 15, 2024

ti-chi-bot bot commented Apr 15, 2024

nolouch commented Apr 10, 2024 •

edited

ti-chi-bot bot commented Apr 10, 2024 •

edited

codecov bot commented Apr 11, 2024 •

edited

rleungx Apr 15, 2024 •

edited