VoteListener doesn't retry votes even if they don't land in a bank

[carllin]

Problem

The vote listener will pull a vote off of gossip, record the timestamp of that vote, and then attempt to run the vote through banking stage. If the vote doesn't land, the vote listener will not reattempt to play that vote again.

[sagar-solana]

@aeyakovenko Can you help propose the right solution to this?

Carl describes the problem pretty well.

The simple fix is to do the following:

• Track vote arrival in Gossip per peer
• Keep retrying the latest vote of each peer through banking

The problem with this is that we should stop retrying the vote if it made it into a bank successfully. Otherwise we just end up with a bunch of duplicate signature errors (nbd?) until a new vote comes through or the validator that submitted it goes offline.

We could also cache these in banking_stage (ick) or track the results (since votes are usually all on the same thread) and keep retrying until it succeeds. But then there's no way to clean up the vote if it's genuinely bad.

Also I should add that we don't know just how much of a problem the current behavior is. But it doesn't seem correct.

[aeyakovenko]

@carllin is that the bug!? I think for now we can grab all the votes from gossip and retry. The duplicate signature errors are nbd.

[aeyakovenko]

How would the votes ever fail and succeed later?

[sagar-solana]

So we don't know if this is an issue. It's just how the code is behaving right now.
For this to ever happen, the leader has to lead on two forks. Where maybe they try all the votes on the wrong fork and then never retry when they lead on the other fork.
Is that possible @aeyakovenko ?

[aeyakovenko]

@sagar-solana @carllin

1. leader creates a bank, submit the votes
2. partition occurs
3. new fork that doesn't contain the previous votes is presented
4. next bank doesn't include the previous votes

I think that would cause it. not sure that is what we saw though.

[sagar-solana]

Agreed. Not sure this is exactly what happened but we should patch it.

[carllin]

yeah, just covering bases here. Root cause still pending more analysis.

[mvines]

Fixed by #6695