Skip to content

Commit

Permalink
rebalancer: introduce pinned bucket concept into rebalancer algo
Browse files Browse the repository at this point in the history 
Pinned bucket is the bucket, that can not be sent out of its
replicaset. Taking pinned buckets into account changes rebalancer
algorithm, since now on some replicasets the perfect balance can
not be reached.

Iterative algorithm is used to learn the best balance in a
cluster. On each step it calculates perfect bucket count for each
replicaset. If this count can not be satisfied due to pinned
buckets, the algorithm does best effort to get the perfect
balance. This is done via ignoring of replicasets disbalanced via
pinning, and their pinned buckets. After that a new balance is
calculated. And it can happen, that it can not be satisfied too.
It is possible, because ignoring of pinned buckets in
overpopulated replicasets leads to decrease of perfect bucket
count in other replicasets, and a new values can become less that
their pinned bucket count.

Part of #71
  • Loading branch information
@Gerold103
Gerold103 committed Mar 26, 2018
1 parent e95d040 commit e3e0d69
Show file tree
Hide file tree
Showing 5 changed files with 500 additions and 37 deletions.
333 changes: 332 additions & 1 deletion test/unit/rebalancer.result
View file
Expand Up @@ -369,7 +369,8 @@ _bucket:replace{3, consts.BUCKET.SENT}
...
get_state()
---
- 2
- bucket_active_count: 2
bucket_pinned_count: 0
...
_bucket:replace{1, consts.BUCKET.RECEIVING}
---
Expand Down Expand Up @@ -643,6 +644,336 @@ replicasets
ethalon_bucket_count: 34
bucket_count: 25
...
--
-- gh-71: allow to pin buckets. A pinned bucket can not be sent
-- out of its replicaset even to satisfy perfect balance.
--
-- For this case the rebalancer does best effort balance. The
-- perfect balance here is unrechable, since on each replicaset
-- too many buckets are pinned.
test_run:cmd("setopt delimiter ';'")
---
- true
...
replicasets = {
uuid1 = {bucket_count = 33, pinned_count = 26, weight = 1},
uuid2 = {bucket_count = 33, pinned_count = 24, weight = 1},
uuid3 = {bucket_count = 34, pinned_count = 30, weight = 1},
uuid4 = {bucket_count = 0, weight = 1},
};
---
...
test_run:cmd("setopt delimiter ''");
---
- true
...
calc_ethalon(replicasets, 100)
---
...
replicasets
---
- uuid4:
weight: 1
ethalon_bucket_count: 20
bucket_count: 0
uuid1:
bucket_count: 33
ethalon_bucket_count: 26
ignore_disbalance: true
weight: 1
pinned_count: 26
uuid3:
bucket_count: 34
ethalon_bucket_count: 30
ignore_disbalance: true
weight: 1
pinned_count: 30
uuid2:
bucket_count: 33
ethalon_bucket_count: 24
ignore_disbalance: true
weight: 1
pinned_count: 24
...
calc_metrics(replicasets, consts.DEFAULT_REBALANCER_MAX_RECEIVING)
---
- 100
...
replicasets
---
- uuid4:
needed: 20
weight: 1
ethalon_bucket_count: 20
bucket_count: 0
uuid1:
bucket_count: 33
ethalon_bucket_count: 26
ignore_disbalance: true
needed: -7
weight: 1
pinned_count: 26
uuid3:
bucket_count: 34
ethalon_bucket_count: 30
ignore_disbalance: true
needed: -4
weight: 1
pinned_count: 30
uuid2:
bucket_count: 33
ethalon_bucket_count: 24
ignore_disbalance: true
needed: -9
weight: 1
pinned_count: 24
...
--
-- Here the disbalance is ok for the replicaset with uuid1 only -
-- other replicasets have pinned buckets too, but not enough to
-- break the balance: buckets are moved ok to uuid4.
--
test_run:cmd("setopt delimiter ';'")
---
- true
...
replicasets = {
uuid1 = {bucket_count = 33, pinned_count = 30, weight = 1},
uuid2 = {bucket_count = 33, pinned_count = 10, weight = 1},
uuid3 = {bucket_count = 34, pinned_count = 15, weight = 1},
uuid4 = {bucket_count = 0, weight = 1},
};
---
...
test_run:cmd("setopt delimiter ''");
---
- true
...
calc_ethalon(replicasets, 100)
---
...
replicasets
---
- uuid4:
weight: 1
ethalon_bucket_count: 23
bucket_count: 0
uuid1:
bucket_count: 33
ethalon_bucket_count: 30
ignore_disbalance: true
weight: 1
pinned_count: 30
uuid3:
ethalon_bucket_count: 23
bucket_count: 34
pinned_count: 15
weight: 1
uuid2:
ethalon_bucket_count: 24
bucket_count: 33
pinned_count: 10
weight: 1
...
calc_metrics(replicasets, consts.DEFAULT_REBALANCER_MAX_RECEIVING)
---
- 100
...
replicasets
---
- uuid4:
needed: 23
weight: 1
ethalon_bucket_count: 23
bucket_count: 0
uuid1:
bucket_count: 33
ethalon_bucket_count: 30
ignore_disbalance: true
needed: -3
weight: 1
pinned_count: 30
uuid3:
bucket_count: 34
ethalon_bucket_count: 23
needed: -11
weight: 1
pinned_count: 15
uuid2:
bucket_count: 33
ethalon_bucket_count: 24
needed: -9
weight: 1
pinned_count: 10
...
--
-- Non-locked replicaset with any pinned bucket count can receive
-- more buckets, if the rebalancer decides it is the best balance.
--
test_run:cmd("setopt delimiter ';'")
---
- true
...
replicasets = {
uuid1 = {bucket_count = 30, pinned_count = 25, weight = 0},
uuid2 = {bucket_count = 25, pinned_count = 25, weight = 1},
uuid3 = {bucket_count = 25, pinned_count = 25, weight = 1},
uuid4 = {bucket_count = 20, weight = 0},
};
---
...
test_run:cmd("setopt delimiter ''");
---
- true
...
calc_ethalon(replicasets, 100)
---
...
replicasets
---
- uuid4:
weight: 0
ethalon_bucket_count: 0
bucket_count: 20
uuid1:
bucket_count: 30
ethalon_bucket_count: 25
ignore_disbalance: true
weight: 0
pinned_count: 25
uuid3:
ethalon_bucket_count: 37
bucket_count: 25
pinned_count: 25
weight: 1
uuid2:
ethalon_bucket_count: 38
bucket_count: 25
pinned_count: 25
weight: 1
...
calc_metrics(replicasets, consts.DEFAULT_REBALANCER_MAX_RECEIVING)
---
- inf
...
replicasets
---
- uuid4:
needed: -20
weight: 0
ethalon_bucket_count: 0
bucket_count: 20
uuid1:
bucket_count: 30
ethalon_bucket_count: 25
ignore_disbalance: true
needed: -5
weight: 0
pinned_count: 25
uuid3:
bucket_count: 25
ethalon_bucket_count: 37
needed: 12
weight: 1
pinned_count: 25
uuid2:
bucket_count: 25
ethalon_bucket_count: 38
needed: 13
weight: 1
pinned_count: 25
...
--
-- Check that the rebalancer can calculate a complex case, when a
-- perfect balance is learned in several steps of the algorithm.
-- Here on the first step it is calculated, that each replicaset
-- must contain 25 buckets. But UUID1 can not satisfy it, so it
-- is ignored. On the next step there are 100 - 30 pinned buckets
-- from UUID1 = 70 buckets, and 3 replicasets. A new perfect
-- balance is 23-23-24. But it can not be satisfied too - UUID2
-- has 25 pinned buckets, so it is ignored. On the third step
-- there are 70 - 25 = 45 buckets and 2 replicasets. A new perfect
-- balance is 22-23. But it is unreachable too, because UUID3 has
-- 24 pinned buckets. So only UUID4 is not ignored, and it
-- receives all non-pinned buckets: 45 - 24 = 21.
--
test_run:cmd("setopt delimiter ';'")
---
- true
...
replicasets = {
uuid1 = {bucket_count = 33, pinned_count = 30, weight = 1},
uuid2 = {bucket_count = 33, pinned_count = 25, weight = 1},
uuid3 = {bucket_count = 34, pinned_count = 24, weight = 1},
uuid4 = {bucket_count = 0, weight = 1},
};
---
...
test_run:cmd("setopt delimiter ''");
---
- true
...
calc_ethalon(replicasets, 100)
---
...
replicasets
---
- uuid4:
weight: 1
ethalon_bucket_count: 21
bucket_count: 0
uuid1:
bucket_count: 33
ethalon_bucket_count: 30
ignore_disbalance: true
weight: 1
pinned_count: 30
uuid3:
bucket_count: 34
ethalon_bucket_count: 24
ignore_disbalance: true
weight: 1
pinned_count: 24
uuid2:
bucket_count: 33
ethalon_bucket_count: 25
ignore_disbalance: true
weight: 1
pinned_count: 25
...
calc_metrics(replicasets, consts.DEFAULT_REBALANCER_MAX_RECEIVING)
---
- 100
...
replicasets
---
- uuid4:
needed: 21
weight: 1
ethalon_bucket_count: 21
bucket_count: 0
uuid1:
bucket_count: 33
ethalon_bucket_count: 30
ignore_disbalance: true
needed: -3
weight: 1
pinned_count: 30
uuid3:
bucket_count: 34
ethalon_bucket_count: 24
ignore_disbalance: true
needed: -10
weight: 1
pinned_count: 24
uuid2:
bucket_count: 33
ethalon_bucket_count: 25
ignore_disbalance: true
needed: -8
weight: 1
pinned_count: 25
...
_bucket:drop()
---
...

0 comments on commit e3e0d69

Please sign in to comment.