go/consensus/tendermint: Implement {Prepare,Process}Proposal

[kostko]

This also makes the nodes execute the proposal in the prepare/process phase such that advanced modification (e.g. including meta transactions based on results) and validation (e.g. rejecting blocks with invalid transactions) becomes possible (although these are not implemented here).

[peternose]

// Update cache of consensus parameters (the only places where consensus parameters can be
// changed are InitChain and EndBlock, so we can safely update the cache here).

Off-topic: The upper comment and the code beneath are incorrect as I think that consensus parameters can also be changed in the BeginBlock, if we are doing an upgrade.

[abukosek]

Looks good to me!
The fixes made to the e2e tests should also hopefully reduce the flakiness.

I think Peter's extra logging idea in PrepareProposal in go/consensus/tendermint/abci/mux.go is good, but it might spam the logs too much, so it might be better to add it as a metric instead (but this can be done in a separate PR with perhaps a few other new metrics as well).

[kostko]

Ugh, fuzzing E2E tests indicate this stuff is somewhat racy as this situation is possible after node kill/restart:

• PrepareProposal is called for a height where the node was a proposer and it successfully executes generating proposal A.
• The node receives a previous proposal B via P2P for the same height (also proposed by the same node but during a previous run).
• ProcessProposal is not called for the newly prepared proposal A but is instead called for proposal B which results in corruption.

This seems to contradict the CometBFT spec which says:

For every round, if the local process is the proposer of the current round, CometBFT calls PrepareProposal, followed by ProcessProposal. These two always come together because they reflect the same proposal that the process also delivers to itself.

[peternose]

• PrepareProposal is called for a height where the node was a proposer and it successfully executes generating proposal A.

Are both proposals (A and B) for the same round?

[kostko]

Yes, both are for the same round, but B is a replay from the old node process (this happens when a node crashes or is killed and an old proposal is replayed). I believe it is as such because of how replay is implemented which first pushes a replay of the timeout event, triggering a new round and a new proposal being generated which fails on the signing step (as this would trigger double signing, but during replay the error is ignored), but then pushes the old proposal which is passed to ProcessProposal.

[kostko]

Ok, seems to be fixed now, continuing to run the e2e fuzzing tests though.

Also filed an issue upstream (cometbft/cometbft#1018).

[peternose]

Suggested change

	) error {
	) {