Merge bug: besu producing zero transaction blocks

[garyschulte]

Description

besu/consensus client combinations on mainnet-shadowfork-5 are producing blocks with zero transactions.

metrics indicate that besu has stopped seeing new transactions, possibly since the merge.

Steps to Reproduce (Bug)

1. log into msnf5 node, observe transaction pool depth of 0

[garyschulte]

This appears to be a problem that occurs at merge transition that prevents remote transactions from getting added to besu's transaction pool. Short term, this has a very simple mitigation: just restart besu after the merge and remote transactions make it into the txpool just fine.

[jflo]

We know why Besu has been proposing empty blocks after hitting TTD. It turns out, that incoming transactions cannot be added to the transaction pool unless besu is synced within 100 blocks of its best peer. When one of those peers is a vanilla mainnet peer, and besu is on a shadowfork, that can be a problem.

[jflo]

It seems this functionality has caused about a 50% reduction in the number of peers that Besu can maintain on Ropsten. Reopening to investigate and address.

[jflo]

PR 3944 above which addressed this has been reverted.

[non-fungible-nelson]

Will test the combination of fixes with the Goerli shadow forks. @garyschulte and @jflo to confirm or close after.

[jflo]

Current case can be seen on Goerli Shadowfork 5. The Prysm(bonsai) and Teku (bonsai) instances are the only ones that are submitting proposals with transactions, and has txs in their pools. They both had been restarted at some point more than 13 hours after TTD.