Automate incrementally growing a cluster

[wohali]

@janl:

With this we will be able to grow clusters one by one instead of hitting a wall when eventually each shard lives on a single machine. E.g. when you add a node to the cluster, all other nodes share 1/Nth of their data with the new node, and everything can keep going. Same for removing a node and shrinking the cluster.

Couchbase has this and it is really nice.

Step A: move a shard around...

[janl]

@nickva is working on a shard splitting proposal which would be the technical foundation of this,

[wohali]

Shard splitting landed in #1920 and related. Shard merging does not yet exist, but the shard splitting gives us most of the infrastructure necessary to make it happen.

We still lack tools to visualise node balancing.

[kocolosk]

It would be great to have full shard rebalancing across nodes in 3.0, but I don't think the timelines are going to work out.

[wohali]

@nickva is there any chance of adding shard merging before 3.0?

[nickva]

@wohali I am thinking probably not.

On the surface it would appear it's just the reverse of splitting, but there are some corner cases to handle that make things a bit more complicated such as:

1. What happens to update sequences when we merge shards.

2. Since we only split by 2 now, we have a more constrained set of possible shard overlap configurations, such that shards either completely enclose other shards' ranges, or are completely enclosed by other shards' ranges. With merges and splits we'd have arbitrary overlaps. Even though I think our current logic will handle it , it would require extra care to test and all the new corner cases

3. Would need to do some job managements updates, say, to avoid having jobs that both split and merge a shard at the same time. This is not difficult, but it would just be extra work and testing.

All in all, it is not impossible but it's also probably too much work for 3.x. If FDB wasn't in play, it we'd likely head that way, and even add automatic rebalancing, but now, kind of feels less motivating to continue on that path.

[wohali]

@nickva then we may simply want to close this as WONTFIX and point people towards something like https://github.com/neighbourhoodie/couch-continuum , which is an ugly workaround but at least reliable. What do you think?

[nickva]

@wohali Agree. Closing as wontfix as suggested.

https://github.com/neighbourhoodie/couch-continuum seems like a good tool