[Bug]: Block Explorer API 'Multiple transactions found' error

[edd]

Problem encountered

Trying to view a transaction, the Block Explorer renders an error: https://explorer.vega.xyz/txs/0x4D50B40A4DB740C6FDDC405B46FD1DA65D633581BE5D2F02C3BBCAB5D9BBF82B

Looking at the network panel we can see it request:

curl 'https://be.vega.community/rest/transactions/4D50B40A4DB740C6FDDC405B46FD1DA65D633581BE5D2F02C3BBCAB5D9BBF82B' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:113.0) Gecko/20100101 Firefox/113.0' -H 'Accept: */*' -H 'Accept-Language: en-GB,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Origin: https://explorer.vega.xyz' -H 'Referer: https://explorer.vega.xyz/' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: cross-site'

And get back an error:

{
  "code": 9,
  "message": "FailedPrecondition error",
  "details": [
    {
      "@type": "type.googleapis.com/vega.ErrorDetail",
      "code": 10002,
      "message": "multiple transactions found",
      "inner": ""
    }
  ]
}

Observed behaviour

Transaction 4D50B40A4DB740C6FDDC405B46FD1DA65D633581BE5D2F02C3BBCAB5D9BBF82B cannoto be fetched from the API

Expected behaviour

A decoded transaction

Steps to reproduce

1. curl 'https://be.vega.community/rest/transactions/4D50B40A4DB740C6FDDC405B46FD1DA65D633581BE5D2F02C3BBCAB5D9BBF82B'

Software version

0.71.4

[zale144]

Discussed here https://vegaprotocol.slack.com/archives/CD4AKTFQT/p1685968032315849 and will likely be fixed with the newer version of Tendermint

[ze97286]

At end block we clean up the cache of out of scope blocks - i.e. we forget about txHash/tid of transactions from blocks that are no longer in scope.
For example, if spamPoWNumberOfPastBlocks is 100, then at the end of block 100 we forget about block 1's transactions.
When we verify block age we reject it if tx.BlockHeight()+params.spamPoWNumberOfPastBlocks < e.currentBlock
This opens up two scenarios which would illustrate what happens above in all cases, i.e. the cases where a replayed transaction is added to a block and executed and the ones it's added to a block and rejected on deliverTx.

Example

• spamPoWNumberOfPastBlocks = 100
• We have a transaction from block 1, executed in some random block 57
• When we reach the end of block 100 we forget about the hash of all transactions from block 1 (spamPoWNumberOfPastBlocks behind)
• Now we branch into the two cases:
  • case 1:
    • the transaction is resubmitted to the mempool between the endBlock of height 100 and the beginBlock for height 101 and is included in block 101. The checkTx at this point would pass because we're after endBlock for block 100, meaning we forgot about the hash and tid of the transaction from block 1, however the check for tx.BlockHeight()+params.spamPoWNumberOfPastBlocks < e.currentBlock would not reject the tx because (1 + 100 < 100) == false. So the transaction could potentially be included if lucky in block 101.
    • When evaluating the transaction in the block (now that block height is 101) we recheck the transaction, and still we don't know about its hash and 1+100 < 101 == false so we still don't reject it and it goes through and gets executed
    • In the case it stays in the mempool and not get executed in block 101, it gets rechecked at the end of block 101 and still passes checkTX for the same reasons, block height is still 101 (1+100 < 101 == false), if it gets included in block 102, we'll fall into the second case.
  • case 2:
    • the transaction is submitted to the mempool between the end of block 101 and begin block 102.
    • it has a potential to be included in block 102 for reasons explained in the first case, so lets say it gets included in block 102 - now the check for (1+100 < 102) == true therefore it will be rejected and will not be executed.
    • if it didn't get included in block 102 it will get rechecked and dropped from the mempool