Split-brain handling

[gsvarovsky]

A clone is prompted to do the recovery dance when the remotes object revokes its 'live' status, for example if the network is down and the remotes object detects it is partitioned.

In the event of a split-brain, in which a set of clones partitions from another set, e.g. when using MQTT bridges, the remotes object may have no idea that this has occurred. The only clue is after the split heals and a message arrives for which the prev message is missing. All clones in receipt of such a message will then re-connect:

• Recovering from each other would mean not recovering, until at least one recovers by talking to someone from the other side of the split (see Rev-up and snapshot from best responder #6)
• This will flood the network with control messages (see Consider n^n cases #7)

Probably need a much smarter way to detect and recover from this case specifically.

[gsvarovsky]

Hypothesis: All of this is solvable inside the Remotes implementation (in JS, a PubSubRemotes or a plain MeldRemotes).

• Split-brain detection by e.g. ping messages, or anomaly detection, e.g. 5 clones suddenly disappear
• Recovery methods proactively seek out peers across a recovered split-brain divide