🐞 [Bug]: the RMB peer losing connections periodically

[muhamadazmy]

What happened?

The rmb peer in the sdk loses connection periodically. It reconnects again but we need to figure out what causes those disconnects (can be a mis-configured connection timeout or similar

which network/s did you face the problem on?

Dev

Twin ID/s

No response

Node ID/s

No response

Farm ID/s

No response

Contract ID/s

No response

Relevant log output

debug failed to read message error="read tcp 192.168.123.44:44778->176.9.62.68:443: use of closed network connection"

[Eslam-Nawara]

After investigation, I've found that rmb sends a ping over the connection every pingInterval = 20 Seconds
and wait for response for pongWait = 40 Seconds

If it didn't receive pong, it returns with error connection stalling, which then closes the local context and the websocket connection, so the reader function fails to read the message with the error specified in the issue and restart the connection.

failed to read message error="read tcp 192.168.123.44:44778->176.9.62.68:443: use of closed network connection

This is the scenario happened to me while investigating the error message, but still not sure why the pong message is not sent back in this case, so still investigating.

[Eslam-Nawara]

Work Completed:

• The problem is if there are a lot of requests sent to a process and there is always a new message is received by the process for 40 seconds then the process wouldn't have the chance to send ping in the first place, but still it will expect to have a pong every pongWait so it would report failure and restart the connection.
• I updated the process to update the latest pong time whenever there's a new message received, as receiving a new message indicates that the connection is not stalling and now the connection is not restarting periodically.

[Omarabdul3ziz]

verified by sending multiple requests to a node with the same connection the connection was held open until all the requests were finished, it takes about 2.5min