[Bug] BFT fails to increment the storage round when syncing and gets stuck at the old round

[feezybabee]

https://hackerone.com/reports/2287110

Summary

BFT fails to increment the storage round when syncing and gets stuck at the old round

Steps To Reproduce:

1. Run the ./devnet.sh

2. Klill the node-3 when BFT advance to round=10, wait serveral seconds and restart the node

3. Check the logs in validator-3.log

WARN BFT failed to increment to the next round from round 10 - Next round (11) is behind the current committee's starting round (14)

4. And the node is stuck

Proposed batch for round 10 is still valid
Resending batch proposal for round 10 to peer '127.0.0.1:5000'

Proof-of-Concept (PoC)

1. When the node sync_with_certificate_from_peer, it will probably produce a block when send_primary_certificate_to_bft

2.ledger.current_committee will be updated when advance_to_next_block

3. And there will be no way to increment_to_next_round because the check, see: https://github.com/AleoHQ/snarkOS/blob/testnet3/node/bft/src/helpers/storage.rs#L166

Supporting Material/References:

Logs: https://github.com/ghostant-1017/logs/blob/master/logs-20231215175639.tar.gz

Impact

The vulnerability will cause the node be stuck.

[ghostant-1017]

Still reproduced in the latest commit: 052457b

[raychu86]

This PR is not the final fix - https://github.com/AleoHQ/snarkOS/pull/3074, but should address this issue.

[howardwu]

@ghostant-1017 can you check if #3074 fixes the issue, and whether it introduces any new issues?

[ghostant-1017]

Yes! The pr seems has fixed this issue

@ghostant-1017 can you check if #3074 fixes the issue, and whether it introduces any new issues?

[raychu86]

We also added a new change on top of the previous fix and noticed more stability - https://github.com/AleoHQ/snarkOS/pull/3105. It should also fix the issue where the validator node would be stuck on an old proposal - "Proposed batch for round X is still valid.

Closing the issue now, but please feel free to reopen if you notice the fixes did not fully resolve the problem.