Investigate sc-consensus-grandpa tests::voter_persists_its_votes

[serban300]

test-linux-stable 1/3 fails very often because of this

[serban300]

Here is where it seems to go in an infinite waiting loop:

polkadot-sdk/substrate/client/consensus/grandpa/src/tests.rs

Lines 1209 to 1214 in 39b1f50

	interval
	.take_while(move |_| {
	future::ready(net2.lock().peer(1).client().info().best_number != 40)
	})
	.for_each(|_| future::ready(()))
	.await;

[serban300]

The problem is caused by the network sync when pushing some more blocks:

polkadot-sdk/substrate/client/consensus/grandpa/src/tests.rs

Line 1199 in 39b1f50

net.lock().peer(0).push_blocks(20, false);

Here are the logs:

2024-04-12T14:29:12.254881Z DEBUG sync: Pre-validating received block announcement 0xd7bc9a9c105e3a3681cd3bd37ba2469a8fc30a954d7d5510e99c4e7fe5bcd821 with number 36 from 12D3KooWH8EJhijrgEWWSoo1LxbFKQyuGMh3yVH7Sx82oaguJPDJ    
2024-04-12T14:29:12.254902Z  WARN sync: 💔 Ignored block (#36 -- 0xd7bc…d821) announcement from 12D3KooWH8EJhijrgEWWSoo1LxbFKQyuGMh3yVH7Sx82oaguJPDJ because all validation slots for this peer are occupied.    
2024-04-12T14:29:12.254969Z DEBUG sync: Pre-validating received block announcement 0xacf6e711af101b24170ecf35c3aafe275e502e55a3aa1e09d643af2f5f947147 with number 37 from 12D3KooWH8EJhijrgEWWSoo1LxbFKQyuGMh3yVH7Sx82oaguJPDJ    
2024-04-12T14:29:12.255006Z DEBUG sync: Pre-validating received block announcement 0xaacbe97d187a8a552cdb1290f27ac31c09cc0257ea0d0db3447725f0bb738467 with number 38 from 12D3KooWH8EJhijrgEWWSoo1LxbFKQyuGMh3yVH7Sx82oaguJPDJ    
2024-04-12T14:29:12.255027Z  WARN sync: 💔 Ignored block (#38 -- 0xaacb…8467) announcement from 12D3KooWH8EJhijrgEWWSoo1LxbFKQyuGMh3yVH7Sx82oaguJPDJ because all validation slots for this peer are occupied.    
2024-04-12T14:29:12.255057Z DEBUG sync: Pre-validating received block announcement 0x96a9cf78364e7cc9abadd8763d0f8c8348eb7cee1b8b66330255c934f214d32e with number 39 from 12D3KooWH8EJhijrgEWWSoo1LxbFKQyuGMh3yVH7Sx82oaguJPDJ    
2024-04-12T14:29:12.255077Z  WARN sync: 💔 Ignored block (#39 -- 0x96a9…d32e) announcement from 12D3KooWH8EJhijrgEWWSoo1LxbFKQyuGMh3yVH7Sx82oaguJPDJ because all validation slots for this peer are occupied.    
2024-04-12T14:29:12.255109Z DEBUG sync: Pre-validating received block announcement 0xbcedb0eadd5d8cd7f71d783bb6dba4070f13bd07a48c3f001155b6e2b7ebfa1c with number 40 from 12D3KooWH8EJhijrgEWWSoo1LxbFKQyuGMh3yVH7Sx82oaguJPDJ    
2024-04-12T14:29:12.255131Z  WARN sync: 💔 Ignored block (#40 -- 0xbced…fa1c) announcement from 12D3KooWH8EJhijrgEWWSoo1LxbFKQyuGMh3yVH7Sx82oaguJPDJ because all validation slots for this peer are occupied.    
2024-04-12T14:29:12.257610Z DEBUG sync: Handling block request from 12D3KooWQJuw6jVH3WaFFpbNjQ4ctDuDQwJrAy5Df8nSFae8PfKk: Starting at `BlockId::Hash(0xf8adcaf5c6e2f061ddf8d50b1205dc6c1134cfaa5c07bc1510141f50d4f7d882)` with maximum blocks of `1`, reputation_change: `None`, small_request `false`, direction `Descending` and attributes `HEADER | BODY | JUSTIFICATION`.    

When the blocks are imported we might run into errors. If such an error occurs on the latest blocks, including block 40, the client state is not updated anymore and net2.lock().peer(1).client().info().best_number will be < 40 => freeze

[dmitry-markin]

To add some details, voter_persists_its_votes freezes in 5-10% of runs at least back to version from 18 Dec 2023 on master. So I wonder why it became obvious only now.