consensus halt with no empty blocks

[ebuchman]

With create_empty_blocks = false, after some number of hours under load, tendermint was found to halt.

The consensus state at this point indicated that all precommits were received for the round, but progress was not being made.

This suggested that perhaps the triggeredTimeoutCommit was not being reset properly, and was preventing the timeoutPrecommit from being scheduled.

Turns out, when the mempool notifies the consensus that txs are available, its possible that the consensus would not have already entered the new round, and the enterNewRound method actually gets skipped, going right to the proposal instead:

tendermint/consensus/state.go

Line 735 in 2449bf7

cs.enterPropose(cs.Height, 0)

Since enterNewRound is where triggeredTimeoutPrecommit is reset, this explains the halt.

[ebuchman]

Should be fixed in #3197