RPC reconnection failure leading to infinite request retries

[roman-khimov]

Expected Behavior

When RPC connection fails (and it can), storage node reconnects to some other node and continues to operate.

Current Behavior

neofs-node[381288]: 2023-05-01T06:06:22.386Z#011info#011subscriber/subscriber.go:228#011RPC connection lost, attempting reconnect
neofs-node[381288]: 2023-05-01T06:06:22.392Z#011error#011policer/check.go:87#011could not get container#011{"component": "Object Policer", "cid": "2cVbPMzKoaLd54Yv5XsWmBTa9qFJ63G8VDZjKkxCxov3", "error": "could not perform test invocation (get): connection lost while waiting for the response"}
// this repeats like forever for various containers
neofs-node[381288]: 2023-05-01T06:06:27.402Z#011warn#011client/multi.go:42#011could not establish connection to the switched RPC node#011{"endpoint": "wss://rpc3.morph.t5.fs.neo.org:51331/ws", "error": "WS client creation: read tcp 65.21.148.207:41294->65.108.90.74:51331: i/o timeout"}
neofs-node[381288]: 2023-05-01T06:06:30.362Z#011info#011client/multi.go:52#011connection to the new RPC node has been established#011{"endpoint": "wss://rpc6.morph.t5.fs.neo.org:51331/ws"}
neofs-node[381288]: 2023-05-01T06:06:33.972Z#011error#011policer/check.go:87#011could not get container#011{"component": "Object Policer", "cid": "7Jpq6TrARwb13PcgQg66G8xatXX3RamjgcjqVR5Y2BDg", "error": "could not perform test invocation (get): connection lost while waiting for the response"}
// but it still loops like crazy with this
neofs-node[381288]: 2023-05-01T06:21:33.667Z#011info#011neofs-node/morph.go:215#011new epoch event from sidechain#011{"number": 5936}
neofs-node[381288]: 2023-05-01T07:24:59.575Z#011info#011neofs-node/morph.go:215#011new epoch event from sidechain#011{"number": 5937}
neofs-node[381288]: 2023-05-01T07:24:59.577Z#011info#011neofs-node/config.go:903#011bootstrapping with online state#011{"previous": "ONLINE"}
neofs-node[381288]: 2023-05-01T08:28:25.935Z#011info#011neofs-node/morph.go:215#011new epoch event from sidechain#011{"number": 5938}
neofs-node[381288]: 2023-05-01T09:31:50.731Z#011info#011neofs-node/morph.go:215#011new epoch event from sidechain#011{"number": 5939}
neofs-node[381288]: 2023-05-01T09:31:50.741Z#011info#011neofs-node/config.go:903#011bootstrapping with online state#011{"previous": "ONLINE"}
neofs-node[381288]: 2023-05-01T10:35:22.285Z#011info#011neofs-node/morph.go:215#011new epoch event from sidechain#011{"number": 5940}
neofs-node[381288]: 2023-05-01T11:38:54.169Z#011info#011neofs-node/morph.go:215#011new epoch event from sidechain#011{"number": 5941}
neofs-node[381288]: 2023-05-01T11:38:54.175Z#011info#011neofs-node/config.go:903#011bootstrapping with online state#011{"previous": "ONLINE"}
neofs-node[381288]: 2023-05-01T11:40:48.690Z#011error#011policer/check.go:87#011could not get container#011{"component": "Object Policer", "cid": "CmTo5i5DDZR12sVPgdPM1RneBgVj1rKbUwpnEue69DRp", "error": "could not perform test invocation (get): connection lost while waiting for the response"}
// seems to be better for some time, but then continues looping with error
neofs-node[381288]: 2023-05-01T11:41:01.381Z#011error#011policer/check.go:87#011could not get container#011{"component": "Object Policer", "cid": "DdRBUdCzwVg4431cJYmrdBKLMuqCqEH8q9GwRtbpPacV", "error": "could not perform test invocation (get): connection lost before registering response channel"}
// the switches to this message and never ends

Context

Testnet, 0.36.1.

[roman-khimov]

The first reconnection happens normally, the second one never happens.

sudo nsenter -t $S01_PID -n ss -K dst 192.168.130.90 dport 30333

to test.