SIMD-0046: Optimistic cluster restart automation #46
Conversation
Co-authored-by: mvines <mvines@gmail.com>
Co-authored-by: mvines <mvines@gmail.com>
Co-authored-by: mvines <mvines@gmail.com>
…-improvement-documents into smart-restart-proposal
proposals/0024-repair-and-restart.md
Outdated
|
|
||
| So after a validator sees that 75% of the validators received 75% of the votes, | ||
| wait for 10 more minutes so that the message it sent out have propagated, then | ||
| restart from the Heaviest slot everyone agreed on. |
There was a problem hiding this comment.
From our last call I was thinking once each validator has figured out the heaviest fork and repaired up to the highest oc slot, the validator would:
- Issue a "hard fork" at the highest oc slot, which also changes the gossip shred version
- Execute the existing "--wait-for-supermajority" logic (ie, purge all slots above the highest oc slot, wait for 80%)
There was a problem hiding this comment.
Changed. I think we probably. should wait for 75% here because we assume 5% could be non-conforming.
proposals/0024-repair-and-restart.md
Outdated
|
|
||
| We calculate "enough" stake as follows. When there are 80% validators joining | ||
| the restart, assuming 5% restarted validators can make mistakes in voting, any | ||
| block with more than 67% - 5% - (100-80)% = 42% could potentially be |
There was a problem hiding this comment.
I don't understand this calculation.
What if the other 100% - 42% = 58% pick some other block?
Why should the minority 42% block be optimistically confirmed?
Why should ever a block with less than 67% vote be optimistically confirmed?
There was a problem hiding this comment.
The goal here is to prevent false negative (if a slot was oc'ed before the restart, you must pick it here), not to prevent false positive (it's okay if we pick a slot here which isn't oc'ed). Because when we select Heaviest later we should see the competing fork and count the votes accordingly.
There was a problem hiding this comment.
prolly add the motivation and justification for these values to the document
proposals/0024-repair-and-restart.md
Outdated
| 2.1 If vote_on_child + stake_on_validators_not_in_restart >= 62%, pick child. | ||
| For example, if 80% validators are in restart, child has 42% votes, then | ||
| 42 + (100-80) = 62%, pick child. 62% is chosen instead of 67% because 5% | ||
| could make the wrong votes. |
There was a problem hiding this comment.
similarly here, I am not sure why it is safe to go below 67%?!
There was a problem hiding this comment.
The goal here is to prevent false negative at all costs and it's okay to have false positive. Let's say X is the first block having only 62% but not 67%, we know if 75% of the validators decide to pick this fork, it will be instantly oc'ed and we won't kick another oc'ed slot out. Does that make sense?
There was a problem hiding this comment.
similarly, add the motivation and justification in the doc
Co-authored-by: mvines <mvines@gmail.com>
| sender's last voted fork. the most significant bit is always | ||
| `last_voted_slot`, least significant bit is `last_voted_slot-81000`. | ||
|
|
||
| The number of ancestor slots sent is hard coded at 81000, because that's |
There was a problem hiding this comment.
A quick justification for 81000 would be nice, to help the reader with context. I don't recall myself either! But I assume we're trying to limit the max size of the crds message?
There was a problem hiding this comment.
I remember that we set it at 9 hours because we assumed that most validator admins would wake up and join the choir in 9 hours.
In reality if the cluster gets stuck, new blocks would be confirmed and voted on very slowly, so very likely 81k slots is overkill.
There was a problem hiding this comment.
Added motivation to the description.
There was a problem hiding this comment.
Got it. So there's no underlying reason why 81k couldn't be larger beyond this? Like if we said 24 hours worth of slots is the very worst case, is there any downside to doing so if in reality all future restarts are handled in less than 9 hours?
There was a problem hiding this comment.
The main downside is that pulling from blockstore is slower and distributed RestartLastVotedForkSlots is larger. Right now when a validator receives RestartLastVotedForkSlots it will just discard anything older than its local root, because if its root is not on the RestartLastVotedForkSLots it can't participate in wen_restart anyway. And repairing anything before root slot doesn't make sense. So making this number larger mostly just wastes network bandwidth.
There was a problem hiding this comment.
Also, 81k is already larger than one UDP packet, so it needs to be chopped into chunks. So 24 hours is not much worse, just more packets.
Co-authored-by: mvines <mvines@gmail.com>
…-improvement-documents into smart-restart-proposal
RestartLastVotedForkSlots.
|
cc @ripatel-fd looks like this SIMD is getting close to consensus with some approval from the Labs-side. Could ya'll on Firedancer take a look at it? |
jacobcreech
added
standard
|
Can I pay (or donate to the charity of the contributors choice) for someone to get this merged? |
lheeger-jump
changed the title
| We set the line at 42% when 80% join the restart, it's possible that | ||
| different validators see different 80%, so their must-have blocks might | ||
| be different, but in reality this case should be rare. Whenever some block | ||
| gets to 42%, repair could be started, because when more validators join the | ||
| restart, this number will only go up but will never go down. |
There was a problem hiding this comment.
recommend explaining where 42% comes from here to before this, so it doesn't appear as a magic number
| For example, if 80% validators are in restart, the number would be | ||
| `67% - 5% - (100-80)% = 42%`. If 90% validators are in restart, the number | ||
| would be `67% - 5% - (100-90)% = 52%`. |
There was a problem hiding this comment.
similarly, recommend moving the explanation of 5% before this number appears
There was a problem hiding this comment.
Added explanation on line 158.
| The two added gossip messages `RestartLastVotedForkSlots` and | ||
| `RestartHeaviestFork` will only be sent and processed when the validator is | ||
| restarted in the new proposed optimistic `cluster restart` mode. They will also | ||
| be filtered out if a validator is not in this mode. So random validator\ | ||
| restarting in the new mode will not bring extra burden to the system. |
There was a problem hiding this comment.
could there be more explicit discussion of the "optimistic" aspect of this such that it won't violate any safety properties?
| combined stake would be `2 * (67% - 5% - X)`. Because we only allow one | ||
| RestartHeaviestFork per pubkey, even if the any validator can put both | ||
| children in its RestartHeaviestFork, the children's total stake should be | ||
| less than `100% - 5% - X`. We can caculate that if `134% - 2 * X < 95% - X`, |
| Even if the last block D on the list may not be optimistically confirmed, | ||
| it already has at least `42% - 5% = 37%` stake, with no competing sibling | ||
| E getting more than `42%` stake. This is equal to the case where `5%` stake | ||
| jumped ship from fork E to fork D, 80% of the cluster can switch to fork E |
There was a problem hiding this comment.
recommend precise terminology "switch proof" instead of "jumped ship"
|
|
||
| Even if the last block D on the list may not be optimistically confirmed, | ||
| it already has at least `42% - 5% = 37%` stake, with no competing sibling | ||
| E getting more than `42%` stake. This is equal to the case where `5%` stake |
There was a problem hiding this comment.
should this be a different letter? E was used for the parent above already
| it already has at least `42% - 5% = 37%` stake, with no competing sibling | ||
| E getting more than `42%` stake. This is equal to the case where `5%` stake | ||
| jumped ship from fork E to fork D, 80% of the cluster can switch to fork E | ||
| if that turns out to be the heavist fork. |
| on the list: | ||
|
|
||
| Assume block A is one such block, it would have `67%` stake, discounting | ||
| `5%` non-conforming and people not participating in wen_restart, it should |
There was a problem hiding this comment.
maybe clearer to call this dishonest nodes, who will vote on blocks they do not actually have
There was a problem hiding this comment.
Because validators can misbehave either due to being malicious or due to software bugs, we want to use non-conforming which doesn't automatically imply people are behaving poorly.
No description provided.