Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to lock a replicaset's buckets from rebalancing #71

Closed
Gerold103 opened this issue Feb 13, 2018 · 2 comments
Closed

Allow to lock a replicaset's buckets from rebalancing #71

Gerold103 opened this issue Feb 13, 2018 · 2 comments
Assignees
Labels
customer feature A new functionality

Comments

@Gerold103
Copy link
Collaborator

Gerold103 commented Feb 13, 2018

Add a method like

vshard.storage.lock_buckets()

This method must protect active buckets of this replicaset from rebalancing, and must forbid to receive new buckets. In other words, rebalancer must consider such replicaset as balanced, even if it contradicts with weights.

@Gerold103 Gerold103 added feature A new functionality customer labels Feb 13, 2018
@Gerold103 Gerold103 self-assigned this Feb 13, 2018
@Gerold103
Copy link
Collaborator Author

Gerold103 commented Feb 14, 2018

And allow to lock a concrete bucket. For example, by:

vshard.storage.bucket_pin(bucket_id)

@Gerold103 Gerold103 added prio2 and removed prio2 labels Feb 14, 2018
@Gerold103
Copy link
Collaborator Author

Why?

  1. Replicaset lock allows, for example, to separate a replicaset for testsing
    from production replicasets. Or to fix some application metadata, that must not
    be sharded for a while.

  2. Bucket pin allows the same, but in the smaller scope.

How it works?

Replicaset lock is not the same as pinning all of its buckets. When a replicaset
is locked, it can neither receive new buckets nor send its own ones.
When a bucket is individually pinned, then it can not be sent, but its
replicaset can receive new buckets.

Replicaset lock and rebalancing

When a replicaset is locked, it does not participate in rebalancing. It means,
that even if its ethalon bucket count is not equal to an actual one, this
disbalance can not be fixed due to lock. When a rebalancer detects that one of
replicasets appears to be locked, it recalculates ethalon bucket count of
non-locked replicasets as if the locked replicaset and its buckets does not
exist.

Bucket pin and rebalancing

It is the much more complex case.

  1. A rebalancer calculates ethalon bucket count as if all buckets are not
    pinned. Then it looks at each replicaset and compares its new ethalon bucket
    count against pinned bucket count. If the pinned one is less, then it is ok. A
    non-locked replicaset with pinned buckets can receive new ones.

  2. If ethalon bucket count is less, than pinned one, this disbalance can not be
    fixed - the rebalancer can not move pinned buckets out of this replicaset. In
    such a case ethalon bucket count of this replicasets are set to exactly the
    pinned one. These replicasets are not considered by a rebalancer then, and their
    pinned count is subtracted from a total bucket count.

  3. The described procecure is restarted from the step one with new total
    bucket count and with replicasets, those ethalon bucket count >= pinned one,
    until it appears, that on all replicasets ethalon bucket count >= pinned one.

Pseudocode:

function cluster_calculate_ethalon_balance(replicasets, bucket_count)
	-- spread buckets over replicasets using weights --
end;

cluster = <all of non-locked replicasets>;
bucket_count = <total bucket count in the cluster>;
next_cluster = <empty>;
while next_cluster != cluster do
	cluster_calculate_ethalon_balance(cluster, bucket_count);
	foreach replicaset in cluster do
		if replicaset.ethalon_bucket_count <
		   replicaset.pinned_bucket_count then
			bucket_count -= replicaset.pinned_bucket_count;
			replicaset.ethalon_bucket_count =
				replicaset.pinned_bucket_count;
		else
			new_cluster.add(replicaset);
		end;
	end;
end;
cluster_calculate_ethalon_balance(cluster, bucket_count);

Gerold103 added a commit that referenced this issue Mar 19, 2018
Locked replicaset can neither receive new buckets nor send its
own ones. Actually, it and its buckets do not participate in
rebalancing, and non-locked replicasets are rebalanced
independently.

For example, consider a cluster:
replicaset1: locked, weight = 1, bucket count = 1500
replicaset2:         weight = 1, bucket count = 1500

When a replicaset3 is added, only rs3 and rs2 participate in
rebalancing (respecting their weights):
replicaset1: locked, weight = 1, bucket count = 1500
replicaset2:         weight = 1, bucket_count = 500
replicaset3:         weight = 2, bucket_count = 1000

The lock is useful for example to hold some test data on a
particular replicaset, or to store on the replicaset a special
data, that must be stored together.

Part of #71
Gerold103 added a commit that referenced this issue Mar 26, 2018
Locked replicaset can neither receive new buckets nor send its
own ones. Actually, it and its buckets do not participate in
rebalancing, and non-locked replicasets are rebalanced
independently.

For example, consider a cluster:
replicaset1: locked, weight = 1, bucket count = 1500
replicaset2:         weight = 1, bucket count = 1500

When a replicaset3 is added, only rs3 and rs2 participate in
rebalancing (respecting their weights):
replicaset1: locked, weight = 1, bucket count = 1500
replicaset2:         weight = 1, bucket_count = 500
replicaset3:         weight = 2, bucket_count = 1000

The lock is useful for example to hold some test data on a
particular replicaset, or to store on the replicaset a special
data, that must be stored together.

Part of #71
Gerold103 added a commit that referenced this issue Mar 26, 2018
Locked replicaset can neither receive new buckets nor send its
own ones. Actually, it and its buckets do not participate in
rebalancing, and non-locked replicasets are rebalanced
independently.

For example, consider a cluster:
replicaset1: locked, weight = 1, bucket count = 1500
replicaset2:         weight = 1, bucket count = 1500

When a replicaset3 is added, only rs3 and rs2 participate in
rebalancing (respecting their weights):
replicaset1: locked, weight = 1, bucket count = 1500
replicaset2:         weight = 1, bucket_count = 500
replicaset3:         weight = 2, bucket_count = 1000

The lock is useful for example to hold some test data on a
particular replicaset, or to store on the replicaset a special
data, that must be stored together.

Part of #71
Gerold103 added a commit that referenced this issue Mar 26, 2018
Pinned bucket is the bucket, that can not be sent out of its
replicaset. Taking pinned buckets into account changes rebalancer
algorithm, since now on some replicasets the perfect balance can
not be reached.

Iterative algorithm is used to learn the best balance in a
cluster. On each step it calculates perfect bucket count for each
replicaset. If this count can not be satisfied due to pinned
buckets, the algorithm does best effort to get the perfect
balance. This is done via ignoring of replicasets disbalanced via
pinning, and their pinned buckets. After that a new balance is
calculated. And it can happen, that it can not be satisfied too.
It is possible, because ignoring of pinned buckets in
overpopulated replicasets leads to decrease of perfect bucket
count in other replicasets, and a new values can become less that
their pinned bucket count.

Part of #71
Gerold103 added a commit that referenced this issue Mar 26, 2018
Gerold103 added a commit that referenced this issue Mar 26, 2018
Pinned bucket is the bucket, that can not be sent out of its
replicaset. Taking pinned buckets into account changes rebalancer
algorithm, since now on some replicasets the perfect balance can
not be reached.

Iterative algorithm is used to learn the best balance in a
cluster. On each step it calculates perfect bucket count for each
replicaset. If this count can not be satisfied due to pinned
buckets, the algorithm does best effort to get the perfect
balance. This is done via ignoring of replicasets disbalanced via
pinning, and their pinned buckets. After that a new balance is
calculated. And it can happen, that it can not be satisfied too.
It is possible, because ignoring of pinned buckets in
overpopulated replicasets leads to decrease of perfect bucket
count in other replicasets, and a new values can become less that
their pinned bucket count.

Part of #71
Gerold103 added a commit that referenced this issue Mar 26, 2018
Gerold103 added a commit that referenced this issue Mar 27, 2018
Locked replicaset can neither receive new buckets nor send its
own ones. Actually, it and its buckets do not participate in
rebalancing, and non-locked replicasets are rebalanced
independently.

For example, consider a cluster:
replicaset1: locked, weight = 1, bucket count = 1500
replicaset2:         weight = 1, bucket count = 1500

When a replicaset3 is added, only rs3 and rs2 participate in
rebalancing (respecting their weights):
replicaset1: locked, weight = 1, bucket count = 1500
replicaset2:         weight = 1, bucket_count = 500
replicaset3:         weight = 2, bucket_count = 1000

The lock is useful for example to hold some test data on a
particular replicaset, or to store on the replicaset a special
data, that must be stored together.

Part of #71
Gerold103 added a commit that referenced this issue Mar 27, 2018
Pinned bucket is the bucket, that can not be sent out of its
replicaset. Taking pinned buckets into account changes rebalancer
algorithm, since now on some replicasets the perfect balance can
not be reached.

Iterative algorithm is used to learn the best balance in a
cluster. On each step it calculates perfect bucket count for each
replicaset. If this count can not be satisfied due to pinned
buckets, the algorithm does best effort to get the perfect
balance. This is done via ignoring of replicasets disbalanced via
pinning, and their pinned buckets. After that a new balance is
calculated. And it can happen, that it can not be satisfied too.
It is possible, because ignoring of pinned buckets in
overpopulated replicasets leads to decrease of perfect bucket
count in other replicasets, and a new values can become less that
their pinned bucket count.

Part of #71
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
customer feature A new functionality
Projects
None yet
Development

No branches or pull requests

1 participant