Account sequence mismatch error during a tx raw transactions

[andynog]

Crate

ibc-relayer

Summary of Bug

While trying to execute a tx raw conn-try transaction between cosmoshub-4 and irishub-1 got a couple of "acount sequence mismatch" errors (see below).

Version

Branch: master
Commit: commit 1adb18d

Steps to Reproduce

When executing a conn-try connection, there was a timeout. When trying again a couple of times, got a account mismatch error.

hermes tx raw conn-try irishub-1 cosmoshub-4 07-tendermint-4 07-tendermint-157 -s connection-201
May 10 15:24:16.515  INFO ibc_relayer_cli::commands: Using default configuration from: '.hermes/config.toml'
May 10 15:24:16.571  INFO ibc_relayer::event::monitor: starting event monitor chain.id=cosmoshub-4
May 10 15:24:16.593  INFO ibc_relayer::event::monitor: starting event monitor chain.id=irishub-1
May 10 15:24:30.699  WARN ibc_relayer::event::rpc: error while building event send_packet.packet_data
Error: tx error: failed during a transaction submission step to chain id irishub-1 with underlying error: RPC error to endpoint http://iris.decentrox.com:26657/: RPC error to endpoint http://iris.decentrox.com:26657/: Internal error: timed out waiting for tx to be included in a block (code: -32603)
ubuntu@ip-172-31-36-51:~/relayer/ibc-rs$ hermes tx raw conn-try irishub-1 cosmoshub-4 07-tendermint-4 07-tendermint-157 -s connection-201
May 10 15:24:39.273  INFO ibc_relayer_cli::commands: Using default configuration from: '.hermes/config.toml'
May 10 15:24:39.283  INFO ibc_relayer::event::monitor: starting event monitor chain.id=cosmoshub-4
May 10 15:24:39.293  INFO ibc_relayer::event::monitor: starting event monitor chain.id=irishub-1
Error: tx error: failed with underlying cause: tx response error: check_tx reports error: log=Log("account sequence mismatch, expected 4, got 3: incorrect account sequence")
ubuntu@ip-172-31-36-51:~/relayer/ibc-rs$ hermes tx raw conn-try irishub-1 cosmoshub-4 07-tendermint-4 07-tendermint-157 -s connection-201
May 10 15:25:08.093  INFO ibc_relayer_cli::commands: Using default configuration from: '.hermes/config.toml'
May 10 15:25:08.103  INFO ibc_relayer::event::monitor: starting event monitor chain.id=cosmoshub-4
May 10 15:25:08.113  INFO ibc_relayer::event::monitor: starting event monitor chain.id=irishub-1
Error: tx error: failed with underlying cause: tx response error: check_tx reports error: log=Log("account sequence mismatch, expected 4, got 3: incorrect account sequence")
ubuntu@ip-172-31-36-51:~/relayer/ibc-rs$ hermes tx raw conn-try irishub-1 cosmoshub-4 07-tendermint-4 07-tendermint-157 -s connection-201
May 10 15:25:43.915  INFO ibc_relayer_cli::commands: Using default configuration from: '.hermes/config.toml'
May 10 15:25:43.949  INFO ibc_relayer::event::monitor: starting event monitor chain.id=cosmoshub-4
May 10 15:25:43.971  INFO ibc_relayer::event::monitor: starting event monitor chain.id=irishub-1
May 10 15:25:45.154  WARN ibc_relayer::event::rpc: error while building event send_packet.packet_data
May 10 15:25:52.767  WARN ibc_relayer::event::rpc: error while building event send_packet.packet_data
May 10 15:25:52.767  WARN ibc_relayer::event::rpc: error while building event send_packet.packet_data
Error: tx error: failed during a transaction submission step to chain id irishub-1 with underlying error: RPC error to endpoint http://iris.decentrox.com:26657/: RPC error to endpoint http://iris.decentrox.com:26657/: Internal error: timed out waiting for tx to be included in a block (code: -32603)

In theory the account sequence is retrieved before a transaction, so there might be some concurrency issue when getting the sequence and submitting the transaction ?

Acceptance Criteria

No account mismatch should appear.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[ancazamfir]

If the conn-try was the only command you were doing there shouldn't be any concurrency.
One possible scenario (as this is a checkTx error) is that the latest sequence number is queried from the full node before indexing is finished for the block that includes a previous Tx using the same sequence.
I am not sure we can do much for the tx raw commands other than adding retries to them which we don't want as these are exactly the kind of scenarios we want them to expose.
In the hermes create connection... command there are retries that should mitigate this problem.