Multiple In-flight Broadcasts and Redundant Paths Break Broadcast Tree

[jrwest]

As found by Quviq in testing, the broadcast tree rooted at a node can be temporarily broken (it is healed by lazy gossip) if multiple writes are in-flight and there are redundant paths in the tree. Since the tree heals this is not so harmful but it does affect how fast updates converge across the cluster.

A few things to investigate here:

1. are nodes improperly pruning in this case
2. can we construct the initial tree w/o redundant links to simply avoid this case

2 is a bit of a punt, but sufficient in addition to healing. It would be good to look into 1 a bit before closing this out, however.

Original Message:

We looked into the broken spanning trees, and it happens when there are concurrent broadcasts
from the same node and there are multiple paths to a node. It's shouldn't be a problem in practice
since redundant paths are eventually removed and the broken tree is healed by the gossip protocol.
It might be worth thinking about how to generate the initial tree though.

[jrwest]

This needs to be re-investigated during the testing cycle. marked for 2.0-RC.

[reiddraper]

This got moved to 2.0. But it does not seem like a test or doc task. Should it get moved back to 2.0-RC?

[jrwest]

no it was purposefully moved to 2.0.1 yesterday. not sure when that became 2.0.

[reiddraper]

@jrwest ok, probably just a misunderstanding with the milestone mixup. Moving back to 2.0.1.