Event monitor crashes when the RPC node goes away

[ancazamfir]

Crate

relayer

Summary of Bug

This was discovered by the DEX team during load testing and happens on hermes multi path startup.

Trace is:

The application panicked (crashed).
Message: called `Result::unwrap()` on an `Err` value: Error { code: ClientInternalError, message: "Client internal error", data: Some("failed to hear back from WebSocket driver") }
Location: relayer/src/chain/cosmos.rs:378
Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it. Run with RUST_BACKTRACE=full to include source snippets.

Version

Steps to Reproduce

Acceptance Criteria

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[nodebreaker0-0]

Hello, I'm bug informant.

To reproduce this crashe, see

1. Set up at least 3 chains

2. Assuming chain-a, chain-b, chain-c,

3. connect a and b , connect a and c
   hermes -c config.toml create channel chain-a chain-b --port-a transfer --port-b transfer -o unordered
hermes -c config.toml create channel chain-a chain-c --port-a transfer --port-b transfer -o unordered

4. Run Hermes with start-multi.

5. Exit the chain-c

Then the crashe will be reproduced.

[adizere]

Thank you for the feedback, @nodebreaker0-0.

I would like to know what would be the required behavior in your opinion, if running hermes start-multi and one of the chains crashes. When that chain crashes, then Hermes cannot continue relaying packets for all paths involving that chain. Now I can imagine two options:

1. The hermes relayer process could exit with an error.
2. The hermes relayer process could continue operating if there are other paths that are alive (paths which still have the chains at both ends alive).

Any thoughts?

[ancazamfir]

I would say 2.
Also, it is possible that the chain is fine (continues to produce blocks) but the full node "crashes". In this case, once the full node is operational, would hermes in case 2. (and with your fix) be able to resume relaying over the paths spanning that chain?

[romac]

We could perhaps introduce a thread to monitor the state of all configured chains and trigger a reload/restart when a crashed node comes back online.

[ancazamfir]

As an initial fix, maybe we do 2 without resuming relaying for the affected chain?

[gamarin2]

I confirm that 2. is much preferred.

[nodebreaker0-0]

In my opinion, number two is appropriate.

It is necessary to separate the rpc connection for each chain independently.

For example

Fundamental Problem Solving

1. connect a and b , connect a and c

2. Chain-c is dead

3. Terminates all relay processes associated with chain-c.

4. But the relay process of a and b must be alive.

If you think this is very complicated development.

Temporary Measure

Multiple rpc nodes in config.toml are required.

Then, if node 1 is unable to communicate rpc, you can try to connect to node 2 (or up to number 3 and 4).

In conclusion, however, the fundamental problem must be solved.

[ancazamfir]

What are the minimum requirements for recovery when the full node comes back? Is manual restart ok, similar to when a new chain is added? Or need something like @romac proposes here #863 (comment)