Fix riak dynamic join leave

[albsch]

• Implemented delete handoff functionality to remove nodes from a cluster.
  • This does not mean that the interaction with the inter-dc communictation is preserved, this has to be checked separately. Nevertheless, removing a node now works on the riak_core side of things.
  • Updated metrics to account for log size decrease
• Added missing function clause for the handoff commands for the materializer_vnode
  • Fixes a bug where a function clause is missing during handoff
  • This does not mean the handoff is implemented correctly, unexpected requests coming from processes which try to access the vnode in the handoff lifecycle can hang indefinitely because the timeout of riak calls is infinity
• Improved documentation for handoff

[albsch]

Adding/Deleting monitoring:

1. Node 1 is started and receives operations
2. Node 2 joined, node 1 transfers half of its log (half of the vnodes start handoff), log shrinks
3. More data, this time only half of the operations arrive at node 1
4. Node 3 joined, rebalancing all vnodes again. Node 1 loses and gains some vnodes

[bieniusa]

Handoff and leave is a cool feature, but it interferes with the Cure protocol (which assumes a fixed one-to-one mapping of nodes on different DCs).

@preguica @marc-shapiro @shamouda Opinions? Should we try to fix the protocol accordingly? How does this relate to partial replication?

[preguica]

On Mon, Dec 9, 2019 at 8:07 AM Annette Bieniusa ***@***.***> wrote: Handoff and leave is a cool feature, but it interferes with the Cure protocol (which assumes a fixed one-to-one mapping of nodes on different DCs).

You are talking about hadoff and leaving inside a DC?

…

@preguica <https://github.com/preguica> @marc-shapiro <https://github.com/marc-shapiro> @shamouda <https://github.com/shamouda> Opinions? Should we try to fix the protocol accordingly? How does this relate to partial replication? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#410?email_source=notifications&email_token=ACLFD4U5MG7C6U5A44ASKPDQXX4CPA5CNFSM4JWZJHMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGIHCCQ#issuecomment-563114250>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACLFD4RFWVHZWEGQDZK5ANTQXX4CPANCNFSM4JWZJHMA> .

[bieniusa]

Yes, it is about dynamicity of nodes within a DC.

[albsch]

Currently, to add a node dynamically to not break cure one needs to do the following manually:

1. Block client requests
2. Each DC forgets every other DC
3. Change any DC configuration in any way
4. Wait until handoff is finished (can take a long time)
5. Reconnect all DCs
6. Receive client requests

Once handoff is implemented correctly, step 4 can be skipped. I'm not sure if 1. and 6. is needed, but added it just in case.

An easier approach would be to design Inter-DC communication in such a way that each DC does not know the internal ring structure of another DC. Then changing the structure and replication inside a DC would not affect another DC, reducing complexity of the problem.

[marc-shapiro]

Le 9 déc. 2019 à 14h18, Albert Schimpf ***@***.***> a écrit : […] An easier approach would be to design Inter-DC communication in such a way that each DC does not know the internal ring structure of another DC. Then changing the structure and replication inside a DC would not affect another DC, reducing complexity of the problem.

Not easily done. Long-term, I would suggest the following approach. All DCs are configured with the same relatively high number of logical shards (2048?). We keep the logical relation of one logical log per shard and one logical FIFO inter-DC connection between corresponding shards. Physically, multiple shards are assigned to some server — this can be different in each DC. However all the shards on the same server share physical resources, e.g., all its logical logs are coalesced to a single physical log. Similarly, redundant inter-DC channels are coalesced. It’s still possible to move logical shards between physical servers, by de-coalescing and re-coalescing accordingly. Marc

[albsch]

The discussion for that is best left for another issue or meeting, this PR doesn't address the issues of proper join leave mechanics and replication.

[albsch]

Will merge this once the rebar3 as test coveralls send stops failing the build.