statistic: collect both read and write pending influence at the same time #5521

lhy1024 · 2022-09-16T07:32:49Z

Signed-off-by: lhy1024 admin@liudos.us

What problem does this PR solve?

Issue Number: Ref #4949

Although there is a regionPendings in the hot scheduler, there is only read or write load during tryAddPendingInfluence.

Hot write scheduler and hot read scheduler may affect each other but cannot be perceived by each other.

What is changed and how it works?

Check List

Tests

Unit test
Integration test

Release note

None.

ti-chi-bot · 2022-09-16T07:32:50Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

HunDunDM
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

codecov · 2022-09-16T07:53:46Z

Codecov Report

Base: 75.60% // Head: 75.56% // Decreases project coverage by -0.04% ⚠️

Coverage data is based on head (1a76711) compared to base (1e1732c).
Patch coverage: 86.95% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5521      +/-   ##
==========================================
- Coverage   75.60%   75.56%   -0.05%     
==========================================
  Files         325      325              
  Lines       32017    32055      +38     
==========================================
+ Hits        24208    24221      +13     
- Misses       5722     5737      +15     
- Partials     2087     2097      +10

Flag	Coverage Δ
unittests	`75.56% <86.95%> (-0.05%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/cluster/cluster.go	`82.84% <0.00%> (-0.17%)`	⬇️
server/statistics/hot_cache_task.go	`83.63% <72.72%> (-2.73%)`	⬇️
server/statistics/kind.go	`43.42% <83.33%> (+3.42%)`	⬆️
server/statistics/hot_cache.go	`90.47% <91.66%> (+0.19%)`	⬆️
pkg/mock/mockcluster/mockcluster.go	`94.18% <100.00%> (+0.01%)`	⬆️
server/schedulers/hot_region.go	`85.82% <100.00%> (+0.08%)`	⬆️
server/statistics/hot_peer_cache.go	`96.89% <100.00%> (+0.02%)`	⬆️
server/schedulers/shuffle_hot_region.go	`55.55% <0.00%> (-10.11%)`	⬇️
pkg/tempurl/tempurl.go	`60.00% <0.00%> (-10.00%)`	⬇️
...erver/config/service_middleware_persist_options.go	`91.66% <0.00%> (-8.34%)`	⬇️
... and 23 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

lhy1024 · 2022-09-16T10:00:00Z

/run-build-arm64 comment=true

sre-bot · 2022-09-16T10:02:30Z

download pd binary(linux arm64) at http://fileserver.pingcap.net/download/builds/pingcap/test/pd/1cdd39bbbcaa5dac101c3e2d1e81a81bde651cf6/centos7/pd-linux-arm64.tar.gz

nolouch · 2022-09-17T03:09:35Z

server/statistics/kind.go

+	case Read:
+		return Write
+	}
+	return Read


why default read?

== Case Write:

there are only two kind in RWType

I think we can panic if not match

nolouch · 2022-09-17T03:19:50Z

server/schedulers/hot_region.go

+	infl := statistics.Influence{Loads: make([]float64, statistics.RegionStatCount), Count: 1}
+	bs.rwTy.SetFullLoadRates(infl.Loads, peer.GetLoads())
+	inverse := bs.rwTy.Inverse()
+	another := bs.GetHotPeerStat(inverse, peer.RegionID, peer.StoreID)


I prefer summary all loads at a same time (including read and write) in prepareForBalance, then get it from the stLoadInfos. it is at a same "snapshot" rather than get it one by one.

Now we don't need to use read and write statistics together. I think we can do this when we unify read and write schedulers later.

yes, it will be placed into together when we unify read and write schedulers.

…time Signed-off-by: lhy1024 <admin@liudos.us>

HunDunDM

rest LGTM

HunDunDM · 2022-09-19T07:47:43Z

server/statistics/hot_peer_cache.go

-		if stat := peers.Get(region.GetID()); stat != nil {
-			return stat.(*HotPeerStat).HotDegree >= hotDegree
+		if stat := peers.Get(regionID); stat != nil {
+			return stat.(*HotPeerStat)


Are there times when there may be insufficient statistical confidence?

HunDunDM · 2022-09-19T08:00:06Z

/merge

ti-chi-bot · 2022-09-19T08:00:07Z

@HunDunDM: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-09-19T08:00:09Z

This pull request has been accepted and is ready to merge.

Commit hash: 1a76711

ref #5521, fix #5527 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: 混沌DM <hundundm@gmail.com>

lhy1024 requested a review from HunDunDM September 16, 2022 07:32

ti-chi-bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none labels Sep 16, 2022

lhy1024 force-pushed the rw-pending branch from ebdad7e to 3a73672 Compare September 16, 2022 09:06

ti-chi-bot removed the do-not-merge/needs-linked-issue label Sep 16, 2022

lhy1024 marked this pull request as ready for review September 16, 2022 09:22

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 16, 2022

lhy1024 mentioned this pull request Sep 16, 2022

Improvement of hot scheduler when meets dimension conflict #4949

Closed

25 tasks

lhy1024 added the component/statistics Statistics for scheduling. label Sep 16, 2022

nolouch reviewed Sep 17, 2022

View reviewed changes

lhy1024 force-pushed the rw-pending branch from c292b54 to d180cc0 Compare September 17, 2022 09:01

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 18, 2022

statistic: collect both read and write pending influence at the same …

aa3a121

…time Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 force-pushed the rw-pending branch from d180cc0 to aa3a121 Compare September 18, 2022 10:05

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 18, 2022

Merge branch 'master' into rw-pending

9bc82e9

lhy1024 mentioned this pull request Sep 19, 2022

metrics: add scheduling hot peer load #5525

Merged

rleungx approved these changes Sep 19, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 19, 2022

HunDunDM reviewed Sep 19, 2022

View reviewed changes

Merge branch 'master' into rw-pending

1a76711

HunDunDM approved these changes Sep 19, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 19, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 19, 2022

ti-chi-bot merged commit ccb97f3 into tikv:master Sep 19, 2022

ti-chi-bot pushed a commit that referenced this pull request Oct 14, 2022

metrics: add scheduling hot peer load (#5525)

bc2019e

ref #5521, fix #5527 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: 混沌DM <hundundm@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistic: collect both read and write pending influence at the same time #5521

statistic: collect both read and write pending influence at the same time #5521

lhy1024 commented Sep 16, 2022 •

edited

ti-chi-bot commented Sep 16, 2022 •

edited

codecov bot commented Sep 16, 2022 •

edited

lhy1024 commented Sep 16, 2022

sre-bot commented Sep 16, 2022

nolouch Sep 17, 2022

HunDunDM Sep 17, 2022

lhy1024 Sep 17, 2022

nolouch Sep 19, 2022

nolouch Sep 17, 2022

HunDunDM Sep 17, 2022

lhy1024 Sep 17, 2022

HunDunDM left a comment

HunDunDM Sep 19, 2022

HunDunDM commented Sep 19, 2022

ti-chi-bot commented Sep 19, 2022

ti-chi-bot commented Sep 19, 2022

statistic: collect both read and write pending influence at the same time #5521

statistic: collect both read and write pending influence at the same time #5521

Conversation

lhy1024 commented Sep 16, 2022 • edited

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Sep 16, 2022 • edited

codecov bot commented Sep 16, 2022 • edited

Codecov Report

lhy1024 commented Sep 16, 2022

sre-bot commented Sep 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HunDunDM left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HunDunDM commented Sep 19, 2022

ti-chi-bot commented Sep 19, 2022

ti-chi-bot commented Sep 19, 2022

lhy1024 commented Sep 16, 2022 •

edited

ti-chi-bot commented Sep 16, 2022 •

edited

codecov bot commented Sep 16, 2022 •

edited