*: add region heartbeat duration breakdown metrics #7871

nolouch · 2024-03-04T09:20:09Z

What problem does this PR solve?

Issue Number: Close #7868

What is changed and how does it work?

*: add region heartbeat duration breakdown metrics
- add a tracer during the heartbeat process
- statistic the lock wait time in RegionsInfo

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

No code

Code changes

Has the configuration change
Has HTTP API interfaces changed (Don't forget to add the declarative for the new API)
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tiup:
Need to cherry-pick to the release branch

Release note

None.

ti-chi-bot · 2024-03-04T09:20:12Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JmPotato
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

codecov · 2024-03-04T09:30:05Z

Codecov Report

Merging #7871 (1bf72a3) into master (c1eabda) will increase coverage by 0.17%.
The diff coverage is 95.45%.

❗ Current head 1bf72a3 differs from pull request most recent head e6d9cd9. Consider uploading reports for the commit e6d9cd9 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7871      +/-   ##
==========================================
+ Coverage   73.32%   73.49%   +0.17%     
==========================================
  Files         435      435              
  Lines       48195    48225      +30     
==========================================
+ Hits        35337    35442     +105     
+ Misses       9784     9728      -56     
+ Partials     3074     3055      -19

Flag	Coverage Δ
unittests	`73.49% <95.45%> (+0.17%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Signed-off-by: nolouch <nolouch@gmail.com>

pkg/core/metrics.go

JmPotato · 2024-03-05T09:23:05Z

pkg/core/metrics.go

+	preCheckDurationSum.Add(h.preCheckDuration.Seconds())
+	preCheckCount.Inc()


Is it possible to flush these metrics asynchronously? It appears that setting the metrics in the middle of processing may affect the performance.

I think it's ok because all of them use atomic. and we can dynamically disable the trace by the config enable-heartbeat-breakdown-metrics

Signed-off-by: nolouch <nolouch@gmail.com>

nolouch · 2024-03-08T06:46:32Z

ptal @JmPotato @rleungx

rleungx

BTW, will it cause too much lock contention?

rleungx · 2024-03-11T03:36:07Z

pkg/core/region.go

@@ -1653,6 +1695,42 @@ func (r *RegionsInfo) GetRegionSizeByRange(startKey, endKey []byte) int64 {
 	return size
 }

+// metrics default poll interval
+const magicCount = 15 * time.Second


How about just DefaultPollInterval?

Signed-off-by: nolouch <nolouch@gmail.com>

…-metrics

nolouch · 2024-03-11T06:43:00Z

@rleungx This PR does not introduce additional locking and unlocking processes. All trace operations are added within the existing locks. Since metrics are directly atomic operations, I think it's okay, but it may increase some CPU time (holding locks). However, this CPU time should be relatively small compared to other parts.

nolouch · 2024-03-12T03:30:46Z

ptal @JmPotato

JmPotato · 2024-03-12T03:42:18Z

Should we conduct a benchmark test on this feature to assess any potential impact on performance?

nolouch · 2024-03-12T08:53:04Z

@JmPotato Here is a benchmark result with the follower code :

func BenchmarkRandomAtomicPutRegion(b *testing.B) {
	regions := NewRegionsInfo()
	var items []*RegionInfo
	const treeSize = 1000000
	for i := 0; i < treeSize; i++ {
		peers := []*metapb.Peer{
			{StoreId: uint64((i % 20) + 1), Id: uint64(i*3 + 1)},
			{StoreId: uint64((i % 20) + 2), Id: uint64(i*3 + 2)},
			{StoreId: uint64((i % 20)) + 3, Id: uint64(i*3 + 3)}}
		region := NewRegionInfo(&metapb.Region{
			Id:       uint64(i + 3000001),
			Peers:    peers,
			StartKey: []byte(fmt.Sprintf("%20d", i)),
			EndKey:   []byte(fmt.Sprintf("%20d", i+1)),
		}, peers[0])
		origin, overlaps, rangeChanged := regions.SetRegion(region)
		regions.UpdateSubTree(region, origin, overlaps, rangeChanged)
		items = append(items, region)
	}
	order := mrand.Perm(treeSize)
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		tracer := NewNoopHeartbeatProcessTracer()
		idx := order[i%treeSize]
		item := items[idx]
		item.approximateKeys += int64(200000)
		item.approximateSize += int64(20)
		item.leader = item.meta.Peers[2]
		regions.AtomicCheckAndPutRegion(item, tracer)
	}
}

Result

Noop tracer

➜  pd git:(add-lock-metrics) ✗ go test -bench ^BenchmarkRandomAtomicPutRegion$ -benchtime=1000000x -count=3  github.com/tikv/pd/pkg/core -cpu=16
 goos: linux
goarch: amd64
pkg: github.com/tikv/pd/pkg/core
cpu: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
BenchmarkRandomAtomicPutRegion-16        1000000              2221 ns/op
BenchmarkRandomAtomicPutRegion-16        1000000              2046 ns/op
BenchmarkRandomAtomicPutRegion-16        1000000              2030 ns/op
PASS

Metrics Tracer

➜  pd git:(add-lock-metrics) ✗ go test -bench ^BenchmarkRandomAtomicPutRegion$ -benchtime=1000000x -count=3  github.com/tikv/pd/pkg/core -cpu=16
goos: linux
goarch: amd64
pkg: github.com/tikv/pd/pkg/core
cpu: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
BenchmarkRandomAtomicPutRegion-16        1000000              2238 ns/op
BenchmarkRandomAtomicPutRegion-16        1000000              2213 ns/op
BenchmarkRandomAtomicPutRegion-16        1000000              2252 ns/op
PASS

nolouch · 2024-03-12T12:09:59Z

already approved by @easonn7

/merge

ti-chi-bot · 2024-03-12T12:10:01Z

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2024-03-12T12:10:03Z

This pull request has been accepted and is ready to merge.

Commit hash: 1bf72a3

HuSharp · 2024-03-14T03:12:10Z

why not add metrics in grafana/pd.json?

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none labels Mar 4, 2024

ti-chi-bot bot requested review from JmPotato and rleungx March 4, 2024 09:20

ti-chi-bot bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Mar 4, 2024

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label Mar 4, 2024

nolouch force-pushed the add-lock-metrics branch from 23be40e to e36b9c1 Compare March 4, 2024 11:14

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 4, 2024

nolouch force-pushed the add-lock-metrics branch from bd05e2d to 25e3e45 Compare March 4, 2024 12:09

*: add region heartbeat duration breakdown metrics

ecf93b6

Signed-off-by: nolouch <nolouch@gmail.com>

nolouch force-pushed the add-lock-metrics branch from 25e3e45 to ecf93b6 Compare March 4, 2024 12:32

Merge remote-tracking branch 'origin/master' into add-lock-metrics

509e5d9

nolouch force-pushed the add-lock-metrics branch from 7a5d15a to 509e5d9 Compare March 4, 2024 12:32

JmPotato reviewed Mar 5, 2024

View reviewed changes

nolouch added 3 commits March 7, 2024 15:25

address

bd59983

Signed-off-by: nolouch <nolouch@gmail.com>

Merge remote-tracking branch 'origin/master' into add-lock-metrics

eecb7d7

fix

4d9a680

Signed-off-by: nolouch <nolouch@gmail.com>

nolouch force-pushed the add-lock-metrics branch from a17a5cf to 4d9a680 Compare March 7, 2024 07:36

Merge branch 'master' into add-lock-metrics

846ddec

rleungx reviewed Mar 11, 2024

View reviewed changes

nolouch added 2 commits March 11, 2024 14:39

address

37f3a98

Signed-off-by: nolouch <nolouch@gmail.com>

Merge remote-tracking branch 'nolouch/add-lock-metrics' into add-lock…

1bf72a3

…-metrics

rleungx approved these changes Mar 11, 2024

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 11, 2024

JmPotato approved these changes Mar 12, 2024

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 12, 2024

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 12, 2024

Merge branch 'master' into add-lock-metrics

e6d9cd9

ti-chi-bot bot merged commit 96590de into tikv:master Mar 12, 2024
22 checks passed

nolouch deleted the add-lock-metrics branch March 12, 2024 12:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: add region heartbeat duration breakdown metrics #7871

*: add region heartbeat duration breakdown metrics #7871

nolouch commented Mar 4, 2024 •

edited

ti-chi-bot bot commented Mar 4, 2024 •

edited

codecov bot commented Mar 4, 2024 •

edited

JmPotato Mar 5, 2024

nolouch Mar 7, 2024 •

edited

nolouch commented Mar 8, 2024

rleungx left a comment •

edited

rleungx Mar 11, 2024

nolouch commented Mar 11, 2024

nolouch commented Mar 12, 2024

JmPotato commented Mar 12, 2024

nolouch commented Mar 12, 2024 •

edited

nolouch commented Mar 12, 2024 •

edited

ti-chi-bot bot commented Mar 12, 2024

ti-chi-bot bot commented Mar 12, 2024

HuSharp commented Mar 14, 2024

		preCheckDurationSum.Add(h.preCheckDuration.Seconds())
		preCheckCount.Inc()

*: add region heartbeat duration breakdown metrics #7871

*: add region heartbeat duration breakdown metrics #7871

Conversation

nolouch commented Mar 4, 2024 • edited

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Mar 4, 2024 • edited

codecov bot commented Mar 4, 2024 • edited

Codecov Report

JmPotato Mar 5, 2024

Choose a reason for hiding this comment

nolouch Mar 7, 2024 • edited

Choose a reason for hiding this comment

nolouch commented Mar 8, 2024

rleungx left a comment • edited

Choose a reason for hiding this comment

rleungx Mar 11, 2024

Choose a reason for hiding this comment

nolouch commented Mar 11, 2024

nolouch commented Mar 12, 2024

JmPotato commented Mar 12, 2024

nolouch commented Mar 12, 2024 • edited

Result

Noop tracer

Metrics Tracer

nolouch commented Mar 12, 2024 • edited

ti-chi-bot bot commented Mar 12, 2024

ti-chi-bot bot commented Mar 12, 2024

HuSharp commented Mar 14, 2024

nolouch commented Mar 4, 2024 •

edited

ti-chi-bot bot commented Mar 4, 2024 •

edited

codecov bot commented Mar 4, 2024 •

edited

nolouch Mar 7, 2024 •

edited

rleungx left a comment •

edited

nolouch commented Mar 12, 2024 •

edited

nolouch commented Mar 12, 2024 •

edited