mempool: Transactions getting stuck

[thanethomson]

In cases where an application does not correctly handle replay protection, transactions can get stuck in the mempool as per tendermint/tendermint#3865.

There have been discussions over the years about how to handle this sort of problem more intelligently. Our team cannot control how applications are built, but we can control whether we implement some kind of fallback mechanism to deal with such cases.

One suggestion has been to implement a TTL (time-to-live) parameter for transactions:

• A mempool configuration parameter that determines the lifespan of a particular transaction from the time that it is entered into the mempool.
• After that TTL period expires for that transaction, that transaction is removed from the mempool by Comet.
• The removal of that transaction is kept track of for some period of time in case another peer gossips it to our node again, in which case we opt not to add it to our mempool.

Definition of done

When we've implemented at least one mechanism by which we can eliminate the possibility of transactions getting stuck in the mempool.

[taariq]

Not sure if our ticket here is related, but adding for us to keep track: #1623

[faddat]

It's literally in my report

https://docs.google.com/document/d/1oCjsVYMaV77etxOEbDxh58vkAQaXf7RAkhXvF_8GYis/edit?usp=drivesdk

You guys wanted to ban me from contributing.

[hvanz]

Another reason to implement TTL is to get rid of the config keep-invalid-txs-in-cache. Quoting tendermint/tendermint#5813 (comment):

this is a temporary config option. The more correct solution would be to add a TTL to each transaction (i.e. CheckTx may return a TTL in ResponseCheckTx).

[faddat]

I really like that idea