Change the handoff sender to use RPC to query the handoff_ip from the handoff listener rather than directly issuing a gen_server call. The call approach crashes against older nodes that do not expect a handoff_ip message.
Claimant transitions are triggered by ring update events. However, if a ring update event is somehow missed, and the ring reaches a steady state, then there will be no more update events and the claimant will stall. Likewise, when a older node joins the cluster, there is a race between when the capabilities system negotiates that the claimant should perform automatic joining and when the claimant transition occurs. Thus, the claimant may not properly auto join an older node in a mixed cluster. This is resolved in this commit by having the claimant periodically check for a stalled ring and trigger a force update if the ring is stalled. A stall is detected by checking if running the claimant logic against the current ring would generate a ring that has different cluster state than the current ring. If so, it is considered stalled and the update forced.
Reintroduce gen_server to riak_core_stat rather than spawn per update Due to review, also make stat cache ets table protected. Add infinity timeout to stat calculation call Doc the TTL param Make error response more meaningful Expose staleness Calculate cache timestamp as early as possible Add unit test for the cache
Treat old-nodes joining the cluster the same as new nodes joining the cluster with the "auto-join" property, since older nodes do not support staged joins.
The vnode rolling start built into the vnode manager causes a race condition in the `all_nodes` call where a vnode is started under the sup before the vnode manager sees it. Thus the children under the sup and the vnodes tracked by the manager do not agree.
Make the various precondition checks for repair on the owning node of the partition. This is better because the repair call could be made from any node but ideally you want to check the ring and node status from the owning node for the simple fact that nodes don't always agree on the ring or the state of other nodes.
The logging code in handoff sender was unnecessarily complex. Also, it's better not to use a helper function as it obfuscates _where_ the logging statement was actually made. I.e. lager will always print the name of the helper fun and not where the call was actually originated from.
The synchronous xfer call from vnode mgr to handoff mgr was very brittle. Firstly, the handoff sender needs the vnode pid and the pid comes from the vnode manager but if the src and target partitions are on the same node then a deadlock will occur when the handoff mgr tries to get the pid from the vnode mgr which is still waiting for the handoff mgr to return, thus a deadlock. Secondly, errors can occur during the call or max concurrency could be reached. Rather than code for these cases it's easier to make the xfer API idempotent and have the vnode manager constantly retry until it has confirmation of a completed xfer.