Node hangs after losing access to RPC nodes

[roman-khimov]

Expected Behavior

Node should either retry RPC connections forever or should shutdown properly (to be restarted by systemd) when it can't connect.

Current Behavior

neofs-node[314962]: 2023-04-14T19:40:11.191Z        error        client/multi.go:113        could not establish connection to any RPC node
...
neofs-node[314962]: 2023-04-14T19:40:41.687Z        warn        subscriber/subscriber.go:136        remote notification channel has been closed
...
neofs-node[314962]: 2023-04-14T19:40:42.779Z        info        event/listener.go:228        stop event listener by context        {"reason": "context canceled"}
...
neofs-node[314962]: 2023-04-14T19:40:41.926Z        warn        neofs-node/main.go:137        internal application error        {"message": "morph connection has been lost"}
...
neofs-node[314962]: 2023-04-14T19:40:54.140Z        info        neofs-node/config.go:819        closing components of the storage engine...
...
neofs-node[314962]: 2023-04-14T19:42:28.638Z        info        neofs-node/config.go:827        all components of the storage engine closed successfully
neofs-node[314962]: 2023-04-14T19:42:28.641Z        info        neofs-node/grpc.go:110        stopping gRPC server...        {"name": "NeoFS Public API"}
neofs-node[314962]: 2023-04-14T19:42:29.074Z        info        neofs-node/grpc.go:89        stop listening gRPC endpoint        {"endpoint": "65.111.111.111:8080"}
neofs-node[314962]: 2023-04-14T19:42:29.074Z        info        neofs-node/grpc.go:126        gRPC server stopped successfully        {"name": "NeoFS Public API"}
neofs-node[314962]: 2023-04-14T19:42:29.084Z        info        neofs-node/grpc.go:110        stopping gRPC server...        {"name": "NeoFS Public API"}
neofs-node[314962]: 2023-04-14T19:42:29.118Z        info        neofs-node/grpc.go:126        gRPC server stopped successfully        {"name": "NeoFS Public API"}
neofs-node[314962]: 2023-04-14T19:42:29.118Z        info        neofs-node/grpc.go:89        stop listening gRPC endpoint        {"endpoint": "65.111.111.111:8082"}

And it hangs forever afterwards.

Context

Happens on testnet, just happens. Both with 0.35.0 and 0.36.0.

Regression

Seems like 0.34.0 behaved better.

Your Environment

• Version used: 0.36.0
• Server setup and configuration: testnet storage node

[roman-khimov]

Can be reproduced in dev-env with a simple morph_chain container restart.