Hermes incorrectly reports low max gas when gas price is too low

[romac]

Summary of Bug

When the gas price is too low, during relaying, when Hermes encounters the following error

out of gas in location: txSize; gasWanted: 130000, gasUsed: 392858: out of gas

it incorrectly reports that

the gas requirement is higher than the configured maximum gas! please check the Hermes config.toml

even when the gas requirement is lower than the configured max_gas.

See the log below for an example:

2023-08-10T12:32:46.427862Z ERROR ThreadId(08) worker.batch{chain=stride-1}:supervisor.handle_batch{chain=stride-1}:supervisor.process_batch{chain=stride-1}:worker.packet.cmd{src_chain=stride-1 src_port=transfer src_channel=channel-5 dst_chain=osmosis-1}:relay{odata=8b1142bf ->Destination @1-4938677; len=1}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=8b1142bf}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=1}: failed to broadcast tx with unrecoverable error response=Response { code: Err(11), data: b"", log: "out of gas in location: txSize; gasWanted: 130000, gasUsed: 392858: out of gas", hash: Hash::Sha256(3D9638619D6A7B5B1BF95A21BDEF67EE335A3EDB50D0A80FE425555D3F65879E) } diagnostic=the gas requirement is higher than the configured maximum gas! please check the Hermes config.toml

Interestingly, in that case the health check for the min gas price succeeded, so there might a discrepancy between the min gas price reported by the node and the one enforced.

Version

master

Steps to Reproduce

TBD

Acceptance Criteria

• Hermes should instead report that the gas price is likely too low and should be upped.
• We should understand why the discrepancy noted above can happen.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[romac]

Apparently, this could also be an issue when estimating the gas needed. Need to investigate further.

[taitruong]

Apparently, this could also be an issue when estimating the gas needed. Need to investigate further.

This may relate to tx simulation:

{
  "timestamp": "2023-10-23T06:42:45.859699Z",
  "level": "WARN",
  "fields": {
    "message": "failed to simulate tx, falling back on default gas because the error is potentially recoverable: gRPC call `send_tx_simulate` failed with status: status: Unknown, message: \"account sequence mismatch, expected 2572, got 2573: incorrect account sequence [cosmos/cosmos-sdk@v0.47.4/x/auth/ante/sigverify.go:269] With gas wanted: '100000000' and gas used: '81119' \", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"x-cosmos-block-height\": \"4534035\"} }"
  },
  "span": { "name": "estimate_gas" },
  "spans": [
    { "chain": "uni-6", "tracking_id": "ConnectionOpenTry", "name": "send_messages_and_wait_commit" },
    { "account.sequence": "2573", "chain": "uni-6", "name": "send_tx_with_account_sequence_retry" },
    { "name": "estimate_gas" }
  ],
  "threadId": "ThreadId(37)"
}
{
  "timestamp": "2023-10-23T06:42:45.982135Z",
  "level": "ERROR",
  "fields": {
    "message": "failed to broadcast tx with unrecoverable error",
    "response": "Response { code: Err(11), data: b\"\", log: \"out of gas in location: txSize; gasWanted: 6750, gasUsed: 57053: out of gas\", hash: Hash::Sha256(96DF490E0A6A127CAD3601556047158328BB63AA305F64E3395A237D965C5409) }",
    "diagnostic": "the gas requirement is higher than the configured maximum gas! please check the Hermes config.toml"
  },
  "span": { "account.sequence": "2573", "chain": "uni-6", "name": "send_tx_with_account_sequence_retry" },
  "spans": [
    { "chain": "uni-6", "tracking_id": "ConnectionOpenTry", "name": "send_messages_and_wait_commit" },
    { "account.sequence": "2573", "chain": "uni-6", "name": "send_tx_with_account_sequence_retry" }
  ],
  "threadId": "ThreadId(37)"
}

This error happens quite often. Rn only thing I can do is wait and give it another try - quite tedious :(. Please note the account sequence mismatch, expected 2572, got 2573 in the log above. It seems like multiple calls are done.

[gaia]

still happening on 1.8