document and improve HA algorithm #670

cofyc · 2019-07-18T13:51:41Z

What problem does this PR solve?

fixes #652

What is changed and how does it work?

Check List

Tests

Unit test

Code changes

Has Go code change

Does this PR introduce a user-facing change?:

NONE

cofyc · 2019-07-19T02:58:30Z

/run-e2e-tests

xiaojingchen · 2019-07-19T08:00:03Z

LGTM

gregwebs · 2019-07-22T18:47:14Z

pkg/scheduler/predicates/ha.go

-		if podsCount+1 >= int(replicas+1)/2 {
+		// When replicas equals 3, pods on each node should not be greater than 1.
+		maxPodsPerNode := 1
+		if replicas > 3 {


3 is hard-coded, but it should be gathered from configuration (in case the user sets it to 5)

I think 3 is the number of copies of data regions in TiKV. Is it possible to has more than 3 copies when the replicas of TiKV instances increases, e.g. replicates one data region into more than 3 instances? cc @tennix @weekface

3 is the tikv replicas.

This pr just fix the corner case when replicas is 4.

cofyc · 2019-07-23T07:09:27Z

it has some conflicts, I will resolve it later

cofyc · 2019-07-23T07:28:21Z

@xiaojingchen @weekface conflicts are resolved, PTAL again. Thanks!
@gregwebs I'm going to fix the bug first.

cofyc · 2019-07-23T07:28:42Z

/run-e2e-tests

xiaojingchen · 2019-07-23T08:36:57Z

pkg/scheduler/predicates/ha.go

+		maxPodsPerNode := 1
+		if replicas > 3 {
+			// When replicas is greater than 3, we allow more than one pods on one node.
+			maxPodsPerNode = int((replicas + 1) / 2)


if replicas =5 , the maxPodsPerNode would be 3. Is it right?

it seems not right... 2, 3 will be possible when there are only two nodes.

int((replicas + 1) / 2) should be ceil(replicas / 3)

replicas maxPodsPerNode three nodes less than three nodes

< 3 not limited, because HA is not possible possible, no HA is required possible, no HA is required

= 3 1 1, 1, 1 not possible, requires HA in three nodes

4 2 1, 1, 2 not possible, requires HA in three nodes

5 2 1, 2, 2 not possible, requires HA in three nodes

6 2 2, 2, 2 not possible, requires HA in three nodes

7 3 2, 2, 3 not possible, requires HA in three nodes

8 3 2, 3, 3 not possible, requires HA in three nodes

8 3 3, 3, 3 not possible, requires HA in three nodes

what do you think? @weekface

In the table above, the maxPodsPerNode is 2 when replicas is 4, this is not HA for pd.

I think the maxPodsPerNode is right when replicas is 4, because in three nodes, 1, 1, 2 is ok, but when there are two instances in two nodes (1, 1), maxPodsPerNode must be 1 at this moment to force the scheduler to find a node which has no TiKV pod.

I guess we need to update the HA algorithm to take more information into account.

maxPodsPerNode is always 1 when scheduled TiKV pods less than 3?

The current algorithm is ok for pd, but not tikv. We need a new HA algorithm for tikv.

cofyc · 2019-07-23T12:27:12Z

@weekface @xiaojingchen PTAL again.

cofyc · 2019-07-23T12:31:32Z

pkg/scheduler/predicates/ha.go

+	if component != label.PDLabelVal && component != label.TiKVLabelVal {
+		glog.V(4).Infof("component %s is ignored in HA predicate", component)
+		return nodes, nil
+	}


this is a safe guard and improves readability

cofyc · 2019-07-23T12:32:49Z

pkg/scheduler/predicates/ha_test.go

-			name:          "three nodes, three pods scheduled on these three nodes, replicas is 4, can't scheduled",
-			podFn:         newHAPDPod,
+			name:          "three nodes, three pods scheduled on these three nodes, replicas is 4, return all the three nodes",
+			podFn:         newHATiKVPod,


this case is for TiKV, we need a TiKV pod

weekface · 2019-07-24T09:49:46Z

/run-e2e-tests

xiaojingchen · 2019-07-25T06:24:28Z

pkg/scheduler/predicates/ha.go

+			 * ---------------------------
+			 * 1			1
+			 * 2			1
+			 * 3		    1


please align

oh, I mixed tab and spaces, fixed now.

cofyc · 2019-07-29T07:13:12Z

/run-e2e-tests

cofyc · 2019-07-29T08:47:11Z

/run-e2e-tests

cofyc · 2019-07-29T10:50:46Z

@weekface should this be merged before 1.0 GA?

weekface · 2019-07-29T11:18:27Z

Yes, merging it to master.

But I will not cherry-pick it to the release-1.0. The release-1.0 is frozen.

sre-bot · 2019-07-30T06:35:40Z

cherry pick to release-1.0 failed

sre-bot · 2019-07-30T06:36:59Z

cherry pick to release-1.1 failed

weekface · 2019-08-01T08:19:48Z

/run-cherry-picker

weekface · 2019-08-01T08:32:01Z

/run-cherry-picker

cofyc · 2019-08-05T08:13:51Z

/run-cherry-picker

* fix the corner case when replicas is 4 * improve HA algorithm for TiKV/PD

fix the corner case when replicas is 4

45d6b18

cofyc requested review from weekface, xiaojingchen and tennix July 18, 2019 13:51

weekface previously approved these changes Jul 19, 2019

View reviewed changes

gregwebs reviewed Jul 22, 2019

View reviewed changes

Merge branch 'master' into fix652

7e66e18

cofyc dismissed weekface’s stale review via 7e66e18 July 23, 2019 07:23

cofyc force-pushed the fix652 branch from ca95464 to 7e66e18 Compare July 23, 2019 07:23

xiaojingchen reviewed Jul 23, 2019

View reviewed changes

cofyc changed the title ~~fix the corner case when replicas is 4~~ WIP: fix the corner case when replicas is 4 Jul 23, 2019

cofyc changed the title ~~WIP: fix the corner case when replicas is 4~~ document and improve HA algorithm Jul 23, 2019

cofyc force-pushed the fix652 branch 2 times, most recently from cd80f80 to 131f76d Compare July 23, 2019 12:28

cofyc commented Jul 23, 2019

View reviewed changes

xiaojingchen reviewed Jul 25, 2019

View reviewed changes

improve HA algorithm for TiKV/PD

dd012e7

cofyc force-pushed the fix652 branch from 131f76d to dd012e7 Compare July 25, 2019 06:30

weekface approved these changes Jul 25, 2019

View reviewed changes

Merge branch 'master' into fix652

0dd81eb

xiaojingchen approved these changes Jul 29, 2019

View reviewed changes

cofyc merged commit 6047095 into pingcap:master Jul 29, 2019

weekface added needs-cherry-pick-1.1 needs-cherry-pick-1.0 labels Jul 29, 2019

weekface added needs-cherry-pick-1.0 and removed needs-cherry-pick-1.0 labels Aug 1, 2019

cofyc added a commit to cofyc/tidb-operator that referenced this pull request Aug 5, 2019

document and improve HA algorithm (pingcap#670)

48aaddd

* fix the corner case when replicas is 4 * improve HA algorithm for TiKV/PD

cofyc mentioned this pull request Aug 5, 2019

cherry pick PR-670: document and improve HA algorithm #730

Merged

cofyc added a commit that referenced this pull request Aug 6, 2019

document and improve HA algorithm (#670) (#730)

8a00b02

* fix the corner case when replicas is 4 * improve HA algorithm for TiKV/PD

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

document and improve HA algorithm #670

document and improve HA algorithm #670

cofyc commented Jul 18, 2019

cofyc commented Jul 19, 2019

xiaojingchen commented Jul 19, 2019

gregwebs Jul 22, 2019

cofyc Jul 23, 2019 •

edited

weekface Jul 23, 2019

weekface Jul 23, 2019

cofyc commented Jul 23, 2019

cofyc commented Jul 23, 2019

cofyc commented Jul 23, 2019

xiaojingchen Jul 23, 2019

cofyc Jul 23, 2019 •

edited

weekface Jul 23, 2019 •

edited

cofyc Jul 23, 2019

cofyc Jul 23, 2019

weekface Jul 23, 2019

cofyc commented Jul 23, 2019

cofyc Jul 23, 2019

cofyc Jul 23, 2019

weekface commented Jul 24, 2019

xiaojingchen Jul 25, 2019

cofyc Jul 25, 2019

cofyc commented Jul 29, 2019

cofyc commented Jul 29, 2019

cofyc commented Jul 29, 2019

weekface commented Jul 29, 2019

sre-bot commented Jul 30, 2019

sre-bot commented Jul 30, 2019

weekface commented Aug 1, 2019

weekface commented Aug 1, 2019

cofyc commented Aug 5, 2019

replicas	maxPodsPerNode	three nodes	less than three nodes
< 3	not limited, because HA is not possible	possible, no HA is required	possible, no HA is required
= 3	1	1, 1, 1	not possible, requires HA in three nodes
4	2	1, 1, 2	not possible, requires HA in three nodes
5	2	1, 2, 2	not possible, requires HA in three nodes
6	2	2, 2, 2	not possible, requires HA in three nodes
7	3	2, 2, 3	not possible, requires HA in three nodes
8	3	2, 3, 3	not possible, requires HA in three nodes
8	3	3, 3, 3	not possible, requires HA in three nodes

document and improve HA algorithm #670

document and improve HA algorithm #670

Conversation

cofyc commented Jul 18, 2019

What problem does this PR solve?

What is changed and how does it work?

Check List

Does this PR introduce a user-facing change?:

cofyc commented Jul 19, 2019

xiaojingchen commented Jul 19, 2019

Choose a reason for hiding this comment

cofyc Jul 23, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cofyc commented Jul 23, 2019

cofyc commented Jul 23, 2019

cofyc commented Jul 23, 2019

Choose a reason for hiding this comment

cofyc Jul 23, 2019 • edited

Choose a reason for hiding this comment

weekface Jul 23, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cofyc commented Jul 23, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weekface commented Jul 24, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cofyc commented Jul 29, 2019

cofyc commented Jul 29, 2019

cofyc commented Jul 29, 2019

weekface commented Jul 29, 2019

sre-bot commented Jul 30, 2019

sre-bot commented Jul 30, 2019

weekface commented Aug 1, 2019

weekface commented Aug 1, 2019

cofyc commented Aug 5, 2019

cofyc Jul 23, 2019 •

edited

cofyc Jul 23, 2019 •

edited

weekface Jul 23, 2019 •

edited