server/constraints: add replication constraints #402

huachaohuang · 2016-11-29T03:05:27Z

This is an example constraints:

[[constraints]]
labels = ["zone=us", "disk=ssd"]
replicas = 2
[[constraints]]
labels = ["zone=cn", "disk=hdd"]
replicas = 1

We need to ensure all constraints will be satisfied in the replica checker, and maintain the constraints when scheduling peers.

siddontang · 2016-11-30T05:52:08Z

should check constraints validation. E,g. if we have max 3 replicas, but the total replicas number > 3, we may panic.

huachaohuang · 2016-12-05T07:34:12Z

PTAL @siddontang @overvenus

siddontang · 2016-12-05T12:20:55Z

server/constraints.go

+// Although we can use a more optimize algorithm, but it may be hard
+// to understand by most users.
+// So we decide to use a straight forward algorithm to match stores
+// and constraints one by one in order, and it works find in most


it works fine

siddontang · 2016-12-05T12:47:49Z

Maybe we should support the exclusive labels like:

If two constraints have same value for this label, no matter other label values are different, they are still not exclusive.

siddontang · 2016-12-05T13:08:04Z

server/config.go

+			return nil, errors.Errorf("invalid constraint %q", label)
+		}
+		k, v := kv[0], kv[1]
+		if !validLabel.MatchString(k) || !validLabel.MatchString(v) {


check key, value separately in two if.

siddontang · 2016-12-05T13:17:55Z

server/constraints.go

+// Match returns true if store matches the constraint.
+func (c *Constraint) Match(store *storeInfo) bool {
+	labels := make(map[string]string)
+	for _, label := range store.GetLabels() {


why not checking len(store.GetLabels() == len(c.Labels) first, then:

for _, label := range store.GetLabels() { if v := c.Labels[label.GetKey()); v != label.GetValue() { return false } }

So we don't need using another map here.

But the store labels don't need to be exactly the same as the constraint, we just require that the store contains all labels in the constraint.
For example, if the constraint is zone=us, a store with labels zone=us,disk=ssd can match the constraint.

siddontang · 2016-12-05T13:24:35Z

server/balancer.go

+		return r.removePeer(region, peer)
+	}
+
+	// Now we have redundant replicas, we can remove unmatched peers.


do we need to consider redundant but matched replicas here?

There are no such cases. If a constraint requires 2 replicas, then even if we have 3 replicas can match the constraint, the third replicas will not show up in the result, and we will consider that it is redundant.

huachaohuang · 2016-12-05T13:36:03Z

I don't understand the comment about exclusive labels, can you give me an example?

siddontang · 2016-12-05T13:43:05Z

E.g,

Store A: zone=1 disk=ssd
Store B: zone=1 disk=hdd

zone is an exclusive label, so if a region has already covered A, we can't use B for it.

huachaohuang · 2016-12-06T05:30:08Z

That feature will not included in this PR, we will design and support it later.

siddontang · 2016-12-07T02:47:03Z

LGTM

siddontang · 2016-12-07T02:47:25Z

PTAL @overvenus

overvenus

Rest LGTM

overvenus · 2016-12-06T11:18:00Z

server/config_test.go

+		{"abc=123*"},
+		{".123=-abc"},
+		{"abc,123=123.abc"},
+		{"abc=123", "abc=abc"},


Is k= v a valid lable? (A space after =)

No, that's not valid.

overvenus · 2016-12-06T11:23:04Z

server/config.go

@@ -70,6 +71,9 @@ type Config struct {

 	MetricCfg MetricConfig `toml:"metric" json:"metric"`

+	ConstraintCfg []ConstraintConfig `toml:"constraints" json:"constraints"`


Please add some examples in config.toml.

overvenus · 2016-12-07T02:57:05Z

server/balancer_test.go

+	checkAddPeer(c, rc.Check(region), 3)
+	peer3, _ := cluster.allocPeer(3)
+	region.Peers = append(region.Peers, peer3)
+	log.Debug(region)


Does it help debug? If not, please remove it.

overvenus · 2016-12-07T03:07:21Z

server/config.go

 	o := &scheduleOption{}
-	o.store(cfg)
+	o.store(&cfg.ScheduleCfg)
+	o.constraints = cfg.constraints


Potential data race? Why not swap line 345 and 346?

Constraints are read only and will not be changed dynamically, so we don't need to protect it.

overvenus · 2016-12-07T03:09:45Z

Please merge master in order to run tests in circleci.

I think the score things are too complicated, we can calculate the scores more easily.

Coordinator can control the speed of different schedulers. Every scheduler has a unique name, we can add API to run or stop any schedulers dynamically, and different schedulers can run concurrently.

I think the score things are too complicated, we can calculate the scores more easily.

* server/selector: add selector interface and balance selector (#388) We can use selector to abstract different strategies to select the source and target stores. * server: use selector to schedule region peer (#389) * server/resource: replace score type with resource kind (#393) I think the score things are too complicated, we can calculate the scores more easily. * server/metrics: remove redundant metrics (#395) * server/balancer: add scheduler and simplify schedule configs (#396) The original balance configs are complicated and some of them are very confusing, now we simplify them to make it more easy to understand. We replace the concept of "balance" with "schedule". Balance means to keep the resource balance, but we will introduce different kinds of strategies in the future which may not balance, and balance is just one kind of strategies to schedule resources between stores. * server/coordinator: replace balancer worker with coordinator (#398) Coordinator can control the speed of different schedulers. Every scheduler has a unique name, we can add API to run or stop any schedulers dynamically, and different schedulers can run concurrently. * server/constraints: add replication constraints (#402) * server/scheduler: add grant-leader-scheduler (#406) * server/coordinator: combine scheduler and controller * server/api: add scheduler api (#407) We can now use API to list all schedulers, add or remove a scheduler. * Add shuffle-leader-scheduler (#409) * server/scheduler: add shuffle-leader-scheduler

huachaohuang changed the title ~~server/constraints: add replication constraints~~ [DNM] server/constraints: add replication constraints Nov 29, 2016

huachaohuang force-pushed the huachao/coordinator branch 2 times, most recently from ae3b984 to bc8e554 Compare November 29, 2016 09:16

huachaohuang force-pushed the huachao/labels branch from 3751016 to a939d22 Compare November 29, 2016 10:50

huachaohuang force-pushed the huachao/coordinator branch from 3c2c3e7 to 4d5ae40 Compare November 29, 2016 10:53

huachaohuang force-pushed the huachao/labels branch from a939d22 to 501e53c Compare November 30, 2016 02:58

huachaohuang force-pushed the huachao/labels branch from 9ca675f to 21a0191 Compare December 1, 2016 10:04

huachaohuang changed the base branch from huachao/coordinator to develop December 1, 2016 10:05

huachaohuang changed the title ~~[DNM] server/constraints: add replication constraints~~ server/constraints: add replication constraints Dec 1, 2016

siddontang reviewed Dec 5, 2016

View reviewed changes

overvenus approved these changes Dec 7, 2016

View reviewed changes

huachaohuang force-pushed the develop branch from 81bee11 to 8a2fc58 Compare December 7, 2016 04:07

huachaohuang added 6 commits December 7, 2016 12:14

server/resource: replace score type with resource kind

96fb9a2

I think the score things are too complicated, we can calculate the scores more easily.

server: resource ratio

c96268e

server/resource: replace score type with resource kind

b404f57

I think the score things are too complicated, we can calculate the scores more easily.

server: resource ratio

ebc44da

server/coordinator: replace balancer worker with coordinator

db1cc55

Coordinator can control the speed of different schedulers. Every scheduler has a unique name, we can add API to run or stop any schedulers dynamically, and different schedulers can run concurrently.

server/balancer: tiny fix

036123e

huachaohuang added 8 commits December 7, 2016 12:14

server/resource: replace score type with resource kind

7a686da

I think the score things are too complicated, we can calculate the scores more easily.

server: resource ratio

b91fd8a

server/constraints: add replication constraints

4c79f5d

server/cluster: fix put store

a003a2a

server/balancer: ensure max replicas

f241399

server/constraints: remove default constraint

b1bb864

server/constraints: check max replicas and constraints

9f3bf91

address comment

e9bf3bb

huachaohuang force-pushed the huachao/labels branch from e0f0ae0 to e9bf3bb Compare December 7, 2016 04:15

address comment

e861af5

huachaohuang merged commit 247fa4f into develop Dec 7, 2016

huachaohuang added a commit that referenced this pull request Dec 8, 2016

server/constraints: add replication constraints (#402)

a691ee9

huachaohuang mentioned this pull request Dec 8, 2016

check duplicated address for existed stores #394

Closed

huachaohuang deleted the huachao/labels branch December 8, 2016 08:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

server/constraints: add replication constraints #402

server/constraints: add replication constraints #402

huachaohuang commented Nov 29, 2016

siddontang commented Nov 30, 2016

huachaohuang commented Dec 5, 2016

siddontang Dec 5, 2016

siddontang commented Dec 5, 2016

siddontang Dec 5, 2016

siddontang Dec 5, 2016

huachaohuang Dec 5, 2016

siddontang Dec 5, 2016

huachaohuang Dec 5, 2016

huachaohuang commented Dec 5, 2016

siddontang commented Dec 5, 2016

huachaohuang commented Dec 6, 2016

siddontang commented Dec 7, 2016

siddontang commented Dec 7, 2016

overvenus left a comment

overvenus Dec 6, 2016

huachaohuang Dec 7, 2016

overvenus Dec 6, 2016

overvenus Dec 7, 2016

overvenus Dec 7, 2016

huachaohuang Dec 7, 2016

overvenus commented Dec 7, 2016

		@@ -70,6 +71,9 @@ type Config struct {

		MetricCfg MetricConfig `toml:"metric" json:"metric"`

		ConstraintCfg []ConstraintConfig `toml:"constraints" json:"constraints"`

server/constraints: add replication constraints #402

server/constraints: add replication constraints #402

Conversation

huachaohuang commented Nov 29, 2016

siddontang commented Nov 30, 2016

huachaohuang commented Dec 5, 2016

Choose a reason for hiding this comment

siddontang commented Dec 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huachaohuang commented Dec 5, 2016

siddontang commented Dec 5, 2016

huachaohuang commented Dec 6, 2016

siddontang commented Dec 7, 2016

siddontang commented Dec 7, 2016

overvenus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

overvenus commented Dec 7, 2016