tx indexer: can lose data if crash after block committed before indexing finishes

[ebuchman]

Indexing of the txs in a block happens asynchronously so as to not block the next heights consensus. But if the block is fully committed and the process crashes before the indexing is complete, on restart, the old block will not be replayed, and so some txs will not have been properly indexed.

During the replay sequence on a restart, we must check to ensure that all txs from the previous block were indexed, and if not, re-index them.

Related to #3186, but that's talking about a block that didn't even finish committing. In that case, the block does get replayed, we just didn't start the eventBus ahead of time. Here, we need to check explicitly if txs didnt finish indexing.

[Stumble]

One question, is there setting or any ongoing proposal that can make tx_indexer runs synchronously?

[tac0turtle]

we recently implemented a re-indexing feature. Please reopen if you see this issue as still relevant.