server/schedule: refine metrics about store limit by wangrzneu · Pull Request #2404 · tikv/pd

wangrzneu · 2020-05-08T05:12:34Z

What problem does this PR solve?

refine metrics about store limit: add storeLimitCostCounter and storeLimitAvailableGauge to monitor the status of store limiter.

What is changed and how it works?

refine metrics about store limit

add storeLimitCostCounter to monitor the step cost
add storeLimitAvailableGauge to monitor the available status of storeLimit, and it is collected in a background job.
add storeLimitRateGauge to show the store limit setting in pd.

Check List

Tests

Manual test (add detailed scripts or steps below)

run a tidb cluste with 4 tikv instances
prepare some data using sysbench like : sysbench --mysql-host=127.0.0.1 --mysql-port=4000 --mysql-user=root --table-size=1000000 oltp_insert prepare
run pd-ctl -u 0.0.0.0:32774 store limit all 1 region-add and pd-ctl -u 0.0.0.0:32774 store limit all 1 region-remove to set the store limit
run pd-ctl -u 0.0.0.0:32774 store delete 3057 to remove one store
run curl 0.0.0.0:32774/metrics | grep store_limit to get the metrics. The metrics should be like:

# HELP pd_schedule_store_limit_available available limit rate of store.
# TYPE pd_schedule_store_limit_available gauge
pd_schedule_store_limit_available{limit_type="region-add",store="1"} 1
pd_schedule_store_limit_available{limit_type="region-add",store="3057"} 1
pd_schedule_store_limit_available{limit_type="region-add",store="4"} 1
pd_schedule_store_limit_available{limit_type="region-add",store="5"} 1
pd_schedule_store_limit_available{limit_type="region-remove",store="1"} 1
pd_schedule_store_limit_available{limit_type="region-remove",store="3057"} 100000
pd_schedule_store_limit_available{limit_type="region-remove",store="4"} 1
pd_schedule_store_limit_available{limit_type="region-remove",store="5"} 1
# HELP pd_schedule_store_limit_cost limit rate cost of store.
# TYPE pd_schedule_store_limit_cost counter
pd_schedule_store_limit_cost{limit_type="region-add",store="1"} 1
pd_schedule_store_limit_cost{limit_type="region-add",store="4"} 2
pd_schedule_store_limit_cost{limit_type="region-remove",store="3057"} 3
# HELP pd_schedule_store_limit_rate the limit rate of store.
# TYPE pd_schedule_store_limit_rate gauge
pd_schedule_store_limit_rate{limit_type="region-add",store="1"} 1.000000016666667
pd_schedule_store_limit_rate{limit_type="region-add",store="3057"} 6e+06
pd_schedule_store_limit_rate{limit_type="region-add",store="4"} 1.000000016666667
pd_schedule_store_limit_rate{limit_type="region-add",store="5"} 1.000000016666667
pd_schedule_store_limit_rate{limit_type="region-remove",store="1"} 1.000000016666667
pd_schedule_store_limit_rate{limit_type="region-remove",store="3057"} 6e+06
pd_schedule_store_limit_rate{limit_type="region-remove",store="4"} 1.000000016666667
pd_schedule_store_limit_rate{limit_type="region-remove",store="5"} 1.000000016666667

Side effects

Breaking backward compatibility
The metrics pd_schedule_store_limit is removed.

rleungx · 2020-05-11T04:19:01Z

I am not sure the metrics of cost or available token is useful (due to the time interval of metrics collection?), but we can record the capacity of the store limit to avoid using pd-ctl to find it. What do you think?

codecov-io · 2020-05-11T04:46:44Z

Codecov Report

Merging #2404 into master will decrease coverage by 0.04%.
The diff coverage is 32.00%.

@@            Coverage Diff             @@
##           master    #2404      +/-   ##
==========================================
- Coverage   76.32%   76.28%   -0.05%     
==========================================
  Files         206      206              
  Lines       22018    22035      +17     
==========================================
+ Hits        16805    16809       +4     
- Misses       3936     3941       +5     
- Partials     1277     1285       +8

Impacted Files	Coverage Δ
server/cluster/cluster.go	`80.23% <0.00%> (-0.53%)`	⬇️
server/schedule/operator_controller.go	`78.85% <27.27%> (-1.99%)`	⬇️
server/schedule/metrics.go	`100.00% <100.00%> (ø)`
server/region_syncer/client.go	`80.15% <0.00%> (-6.11%)`	⬇️
server/schedulers/random_merge.go	`61.19% <0.00%> (-2.99%)`	⬇️
server/cluster/coordinator.go	`74.52% <0.00%> (-2.20%)`	⬇️
server/tso/tso.go	`81.02% <0.00%> (-2.19%)`	⬇️
server/member/leader.go	`74.31% <0.00%> (-1.17%)`	⬇️
server/core/storage.go	`74.38% <0.00%> (-0.83%)`	⬇️
server/server.go	`77.53% <0.00%> (-0.47%)`	⬇️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update feb738d...bd0a047. Read the comment docs.

wangrzneu · 2020-05-12T14:13:19Z

I am not sure the metrics of cost or available token is useful (due to the time interval of metrics collection?), but we can record the capacity of the store limit to avoid using pd-ctl to find it. What do you think?

I add a metric pd_schedule_store_limit_rate to show the store limit setting of pd.

shafreeck · 2020-05-13T09:49:37Z

 }
+
+// CollectStoreLimitMetrics collects the metrics about store limit
+func (oc *OperatorController) CollectStoreLimitMetrics() {


Where is the new added function called?

Where is the new added function called?

It is called in server/cluster/cluster.go:115.

nolouch · 2020-05-15T05:13:28Z

PTAL @rleungx

shafreeck

LGTM

nolouch

thanks, @wangrzneu. rest LGTM.

nolouch · 2020-05-18T16:41:20Z

 			Subsystem: "schedule",
-			Name:      "store_limit",
-			Help:      "Limit of store.",
-		}, []string{"store", "type", "limit_type"})


The metrics will not be compatible with the old version of grafana template, why can't it keep the original way?

The metrics will not be compatible with the old version of grafana template, why can't it keep the original way?

The old way has two type, "take" and "available". I use a new metrics storeLimitAvailableGauge to record the "available" type storeLimitGauge, but use a different metrics storeLimitCostCounter (it is a counter, not gauge like before) to record the original "take" type storeLimitGauge metrics.

nolouch · 2020-05-18T16:58:34Z

+				}
+				storeIDStr := strconv.FormatUint(storeID, 10)
+				storeLimitAvailableGauge.WithLabelValues(storeIDStr, n).Set(float64(storeLimit.Available()) / float64(storelimit.RegionInfluence[v]))
+				storeLimitRateGauge.WithLabelValues(storeIDStr, n).Set(storeLimit.Rate() * StoreBalanceBaseTime)


Users may more likely want to know the store how many operators can be produced in one minute, for example, default is 15 opm.

The metrics storeLimitRateGauge can show the operator producing rate.

rleungx · 2020-06-01T09:19:07Z

/merge

sre-bot · 2020-06-01T09:19:10Z

Your auto merge job has been accepted, waiting for:

2467

sre-bot · 2020-06-01T09:22:19Z

/run-all-tests

nolouch · 2020-06-01T11:26:31Z

/merge

sre-bot · 2020-06-01T11:28:49Z

/run-all-tests

codecov-commenter · 2020-06-01T11:40:22Z

Codecov Report

Merging #2404 into master will increase coverage by 0.05%.
The diff coverage is 60.00%.

@@            Coverage Diff             @@
##           master    #2404      +/-   ##
==========================================
+ Coverage   77.01%   77.07%   +0.05%     
==========================================
  Files         205      205              
  Lines       21948    22016      +68     
==========================================
+ Hits        16903    16968      +65     
- Misses       3755     3768      +13     
+ Partials     1290     1280      -10

Impacted Files	Coverage Δ
server/cluster/coordinator.go	`71.95% <7.14%> (-2.57%)`	⬇️
server/schedule/operator_controller.go	`78.50% <33.33%> (-2.51%)`	⬇️
server/cluster/cluster.go	`80.06% <66.66%> (-0.23%)`	⬇️
client/base_client.go	`90.60% <100.00%> (+3.70%)`	⬆️
client/client.go	`70.60% <100.00%> (+1.00%)`	⬆️
pkg/cache/ttl.go	`90.66% <100.00%> (+4.72%)`	⬆️
server/cluster/cluster_worker.go	`72.95% <100.00%> (+0.22%)`	⬆️
server/grpc_service.go	`59.96% <100.00%> (+1.98%)`	⬆️
server/schedule/metrics.go	`100.00% <100.00%> (ø)`
pkg/mock/mockhbstream/mockhbstream.go	`89.23% <0.00%> (-1.54%)`	⬇️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06fbee6...4965bf3. Read the comment docs.

lhy1024 · 2020-06-01T11:45:01Z

/merge

sre-bot · 2020-06-01T11:45:17Z

/run-all-tests

sre-bot · 2020-06-01T11:49:17Z

@wangrzneu merge failed.

wangrzneu · 2020-06-02T02:20:38Z

/merge

wangrzneu · 2020-06-02T02:31:15Z

/run-all-tests

nolouch · 2020-06-03T11:56:45Z

/merge

sre-bot · 2020-06-03T11:58:57Z

/run-all-tests

sre-bot · 2020-06-03T12:02:26Z

@wangrzneu merge failed.

wangrzneu · 2020-06-03T13:09:13Z

/run-all-tests

lhy1024 · 2020-06-03T14:17:07Z

/merge

sre-bot · 2020-06-03T14:17:29Z

/run-all-tests

sre-bot · 2020-06-03T14:23:26Z

@wangrzneu merge failed.

nolouch · 2020-06-08T11:34:17Z

/merge

sre-bot · 2020-06-08T11:36:46Z

/run-all-tests

wangrzneu added 2 commits May 4, 2020 02:34

refine store limit related metrics

4655959

add CollectStoreLimitMetrics method

018c827

sre-bot added the contribution This PR is from a community contributor. label May 8, 2020

wangrzneu added 2 commits May 7, 2020 22:15

add comment of CollectStoreLimitMetrics

a6ef421

Merge branch 'master' into store_limit_metric

e1c651a

rleungx requested review from lhy1024, rleungx and shafreeck May 8, 2020 08:02

Merge branch 'master' into store_limit_metric

bd0a047

wangrzneu added 2 commits May 12, 2020 21:47

add metric storeLimitRateGauge

0c69089

Merge branch 'master' into store_limit_metric

70f34c8

shafreeck reviewed May 13, 2020

View reviewed changes

Merge branch 'master' into store_limit_metric

a6754d8

shafreeck approved these changes May 15, 2020

View reviewed changes

nolouch reviewed May 18, 2020

View reviewed changes

show default store limit rate in metrics interface

b2dfaa0

wangrzneu force-pushed the store_limit_metric branch from 7005cd3 to b2dfaa0 Compare May 25, 2020 05:07

Merge branch 'master' into store_limit_metric

e80494d

nolouch approved these changes May 31, 2020

View reviewed changes

sre-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 1, 2020

Merge branch 'master' into store_limit_metric

1f45ea9

Merge branch 'master' into store_limit_metric

4965bf3

lhy1024 approved these changes Jun 1, 2020

View reviewed changes

Merge branch 'master' into store_limit_metric

d9f400e

Merge branch 'master' into store_limit_metric

7ed9f58

Merge branch 'master' into store_limit_metric

3b901f1

sre-bot merged commit 0d2eb9c into tikv:master Jun 8, 2020

Conversation

wangrzneu commented May 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Uh oh!

rleungx commented May 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-io commented May 11, 2020

Codecov Report

Uh oh!

wangrzneu commented May 12, 2020

Uh oh!

shafreeck May 13, 2020

Choose a reason for hiding this comment

Uh oh!

wangrzneu May 13, 2020

Choose a reason for hiding this comment

Uh oh!

nolouch commented May 15, 2020

Uh oh!

shafreeck left a comment

Choose a reason for hiding this comment

Uh oh!

nolouch left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nolouch May 18, 2020

Choose a reason for hiding this comment

Uh oh!

wangrzneu May 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nolouch May 18, 2020

Choose a reason for hiding this comment

Uh oh!

wangrzneu May 25, 2020

Choose a reason for hiding this comment

Uh oh!

nolouch May 31, 2020

Choose a reason for hiding this comment

Uh oh!

rleungx commented Jun 1, 2020

Uh oh!

sre-bot commented Jun 1, 2020

Uh oh!

sre-bot commented Jun 1, 2020

Uh oh!

nolouch commented Jun 1, 2020

Uh oh!

sre-bot commented Jun 1, 2020

Uh oh!

codecov-commenter commented Jun 1, 2020

Codecov Report

Uh oh!

lhy1024 commented Jun 1, 2020

Uh oh!

sre-bot commented Jun 1, 2020

Uh oh!

sre-bot commented Jun 1, 2020

Uh oh!

wangrzneu commented Jun 2, 2020

Uh oh!

wangrzneu commented Jun 2, 2020

Uh oh!

nolouch commented Jun 3, 2020

Uh oh!

sre-bot commented Jun 3, 2020

Uh oh!

sre-bot commented Jun 3, 2020

Uh oh!

wangrzneu commented Jun 3, 2020

Uh oh!

lhy1024 commented Jun 3, 2020

Uh oh!

sre-bot commented Jun 3, 2020

wangrzneu commented May 8, 2020 •

edited

Loading

rleungx commented May 11, 2020 •

edited

Loading

nolouch left a comment •

edited

Loading

wangrzneu May 25, 2020 •

edited

Loading