AZ447: Improve partition claim algorithm to reduce unneeded partition transfers

[jtuple]

This pull-request adds a new partition claim algorithm in riak_core_new_claim.erl that aims to reduce the number of unneeded partition transfers. However, the previous algorithm (default_claim) remains the default for 1.0.

The default algorithm falls back to re-diagonalization far too often, making it so that adding a new node to the more often than not requires more than half of the partitions to be handed off, rather than the expected 1/N (where N is the number of nodes, including the new node).

The new algorithm tries more closely hit the 1/N target while still maintaining the target_n spread constraint.

In addition to the new algorithm, this pull-request also merges in a claim simulation harness originally contributed by Greg Nelson at Dropcam (https://gist.github.com/992317). This can be used to compare the efficiency of the new and old algorithms.

[kellymclaughlin]

How about adding two calls to run, one with the default claim and one with the new?

[jtuple]

The claim simulation code is setup as an eunit test, and can be invoked through rebar:
./rebar app=riak_core eunit suite=claim_simulation

The code writes out a file in /tmp of the form /tmp/rings_RINGSIZE_NODES.txt. For example, the default in the test is to test a 32 element ring with 8 nodes, so you'll see /tmp/rings_32_8.txt.

It should be easy to modify that file test/claim_simulation.erl in order to change ring, node, and claim algorithm parameters. Current, the new algorithm is the default in the code, comment/uncomment the obvious lines will enable the default algorithm.

To test a real cluster, be sure to set lager debugging to the debug level. Then, on the claimant (typically dev1) you'll see messages of the form Pending ownership transfers: X printed the the console/console.log. It makes it easy to add/remove nodes and see the outcome of the rebalance. Setting the app.config as appropriate will enable you to test out a old/new claim cluster and compare the pending transfers number:

{riak_core, [
%%  {choose_claim_fun, {riak_core_claim, default_choose_claim}},
%%  {wants_claim_fun, {riak_core_claim, default_wants_claim}},
    {choose_claim_fun, {riak_core_new_claim, new_claim}},
    {wants_claim_fun, {riak_core_new_claim, new_wants_claim}},

There's also new pretty-printer for riak_core_ring. To demonstrate:

%% Single-line pretty-print of ring, grouped into target_n segments
> riak_core_ring:pretty_print(Ring, []).            
abcc|abcc|abcc|abcc|abcc|abcc|abcc|abcc|abcc|abcc|abca|abba|abba|abba|abba|abba|

%% Include legend in pretty-print
> riak_core_ring:pretty_print(Ring, [legend]). 
==================================== Nodes ====================================
Node a:  34.4% dev1@127.0.0.1
Node b:  32.8% dev2@127.0.0.1
Node c:  32.8% dev3@127.0.0.1
==================================== Ring =====================================
abcc|abcc|abcc|abcc|abcc|abcc|abcc|abcc|abcc|abcc|abca|abba|abba|abba|abba|abba|

%% Print using numeric ids instead of letters (useful for cluster > 26 nodes)
riak_core_ring:pretty_print(Ring, [numeric]).
["1","2","3","3","1","2","3","3","1","2","3","3","1","2","3","3","1","2","3",
 "3","1","2","3","3","1","2","3","3","1","2","3","3","1","2","3","3","1","2",
 "3","3","1","2","3","1","1","2","2","1","1","2","2","1","1","2","2","1","1",
 "2","2","1","1","2","2","1"]

[jtuple]

Another interesting thing to point out is that the default target_n_val is 4 while the default N is 3. This leads to what may appear as strange behavior with the new claim algorithm, versus the old.

For example, the old algorithm's ring progression going from 1 to 4 nodes clearly shows that with 3 nodes most N=3 preflists are on distinct nodes:

1: aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|
2: abab|abab|abab|abab|abab|abab|abab|abab|
3: abca|bcab|cabc|abca|bcab|cabc|abca|bcab|
4: abcd|abcd|abcd|abcd|abcd|abcd|abcd|abcd|

For the new algorithm, the default settings result in the 3 node ring having repeated nodes in most N=3 preflists, because the new algorithm doesn't maximize spread until after the number of nodes is greater than target_n_val. In other words, both algorithms are safe with N=3 when there are 4 nodes, but the old is better when there are only 3 nodes:

1: aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|
2: abba|abba|abba|abba|abba|abba|abba|abba|
3: abcc|abcc|abcc|abcc|abca|abca|abba|abba|
4: abcd|abcd|abcd|abcd|abcd|abcd|abcd|abcd|

If, however, target_n_val was 3 to begin with, then the new algorithm also has 3 distinct nodes per N=3 preflist with only 3 nodes (except the same wrap around preflist that also plagues the old algorithm due to 3 not evenly diving the ring size):

aaa|aaa|aaa|aaa|aaa|aaa|aaa|aaa|aaa|aaa|aa
abb|abb|abb|abb|abb|aba|aba|aba|aba|aba|ab
abc|abc|abc|abc|abc|abc|abc|abc|abc|abc|ab
adc|adc|adc|abd|abd|abc|dbc|dbc|abc|abc|db

[kellymclaughlin]

Code review, test runs, and local cluster testing look good. +1 to merge.