panic after neofs-storage and neofs-ir binary update

[anatoly-bogatyrev]

After replacing the binaries with the latest master, there was a panic:

окт 19 19:39:57 az neofs-node[37515]: 2022-10-19T19:39:57.548Z        info        neofs-node/main.go:114        notary service started successfully
окт 19 19:39:57 az neofs-node[37515]: 2022-10-19T19:39:57.548Z        info        neofs-node/config.go:866        bootstrapping with online state        {"previous": "ONLINE"}
окт 19 19:39:57 az neofs-node[37515]: 2022-10-19T19:39:57.559Z        info        neofs-node/main.go:128        application started        {"version": "v0.33.0-15-g0d14ef69"}
окт 19 19:39:57 az neofs-node[37515]: 2022-10-19T19:39:57.559Z        info        neofs-node/main.go:110        starting metrics service...
окт 19 19:39:57 az neofs-node[37515]: 2022-10-19T19:39:57.559Z        info        neofs-node/tree.go:71        synchronizing trees...
окт 19 19:39:57 az neofs-node[37515]: 2022-10-19T19:39:57.559Z        info        neofs-node/main.go:110        starting profiler service...
окт 19 19:39:57 az neofs-node[37515]: 2022-10-19T19:39:57.560Z        info        neofs-node/main.go:110        starting control service...
окт 19 19:39:57 az neofs-node[37515]: 2022-10-19T19:39:57.560Z        info        neofs-node/main.go:110        starting morph notification service...
окт 19 19:39:59 az neofs-node[37515]: 2022/10/19 19:39:59 worker exits from a panic: open gRPC connection: gRPC dial: context deadline exceeded
окт 19 19:39:59 az neofs-node[37515]: 2022/10/19 19:39:59 worker exits from panic: goroutine 3369 [running]:
окт 19 19:39:59 az neofs-node[37515]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
окт 19 19:39:59 az neofs-node[37515]:         github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
окт 19 19:39:59 az neofs-node[37515]: panic({0xf540e0, 0xc006dfd2c0})
окт 19 19:39:59 az neofs-node[37515]:         runtime/panic.go:838 +0x207
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).createForAddress(0xc004372f00, {{0x127cba0?, 0xc00073ac18?}})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:62 +0x2d0
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).client(0xc004372f00, {{0x127cba0?, 0xc00073ac18?}})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:249 +0x125
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).iterateClients.func1({{0x127cba0?, 0xc00073ac18?}})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:109 +0xba
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network.AddressGroup.IterateAddresses(...)
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/group.go:37
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).iterateClients(0x0?, {0x1274c58?, 0xc004372ea0?}, 0xb8ced5?)
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:99 +0xbb
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).ObjectHead(0x0?, {0x1274c58?, 0xc004372ea0?}, {{{0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0x0, ...}, ...}, ...})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:171 +0x87
окт 19 19:39:59 az neofs-node[37515]: main.(*reputationClient).ObjectHead(0xc0056fb600, {0x1274c58?, 0xc004372ea0?}, {{{0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0x0, ...}, ...}, ...})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/object.go:496 +0x7a
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/services/object/internal/client.HeadObject({{{{0x127c8a8, 0xc0056fb600}, {0x1274c58, 0xc004372ea0}, 0xc000316120, 0x0, 0x0, 0x1, {0x0, 0x0, ...}}}, ...})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/services/object/internal/client/client.go:253 +0x215
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/services/object/head.(*RemoteHeader).Head(0xc001d449d8, {0x1274c58, 0xc004372ea0}, 0xc0048d1820)
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/services/object/head/remote.go:93 +0x47c
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).processNodes(0xc001d44a20, 0xc00073abb8, {{0x1, 0x63, 0x6a, 0x41, 0xcc, 0x25, 0x3, 0xd5, ...}, ...}, ...)
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/services/policer/check.go:186 +0x605
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).processObject(0xc001d44a20, {0x12759b0, 0xc0000fb840}, {{0x1, 0x63, 0x6a, 0x41, 0xcc, 0x25, 0x3, ...}, ...})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/services/policer/check.go:125 +0xbed
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).shardPolicyWorker.func1()
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/services/policer/process.go:65 +0x1a5
окт 19 19:39:59 az neofs-node[37515]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
окт 19 19:39:59 az neofs-node[37515]:         github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
окт 19 19:39:59 az neofs-node[37515]: created by github.com/panjf2000/ants/v2.(*goWorker).run
окт 19 19:39:59 az neofs-node[37515]:         github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65
окт 19 19:39:59 az neofs-node[37515]: 2022/10/19 19:39:59 worker exits from a panic: open gRPC connection: gRPC dial: context deadline exceeded
окт 19 19:39:59 az neofs-node[37515]: 2022/10/19 19:39:59 worker exits from panic: goroutine 3341 [running]:
окт 19 19:39:59 az neofs-node[37515]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
окт 19 19:39:59 az neofs-node[37515]:         github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
окт 19 19:39:59 az neofs-node[37515]: panic({0xf540e0, 0xc0069143c0})
окт 19 19:39:59 az neofs-node[37515]:         runtime/panic.go:838 +0x207
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).createForAddress(0xc0027767e0, {{0x127cba0?, 0xc00346f170?}})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:62 +0x2d0
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).client(0xc0027767e0, {{0x127cba0?, 0xc00346f170?}})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:249 +0x125
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).iterateClients.func1({{0x127cba0?, 0xc00346f170?}})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:109 +0xba
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network.AddressGroup.IterateAddresses(...)
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/group.go:37
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).iterateClients(0x0?, {0x1274c58?, 0xc002776780?}, 0xb8ced5?)
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:99 +0xbb
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).ObjectHead(0x203001?, {0x1274c58?, 0xc002776780?}, {{{0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0x0, ...}, ...}, ...})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:171 +0x87
окт 19 19:39:59 az neofs-node[37515]: main.(*reputationClient).ObjectHead(0xc004c1ba40, {0x1274c58?, 0xc002776780?}, {{{0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0x0, ...}, ...}, ...})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/object.go:496 +0x7a
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/services/object/internal/client.HeadObject({{{{0x127c8a8, 0xc004c1ba40}, {0x1274c58, 0xc002776780}, 0xc000316120, 0x0, 0x0, 0x1, {0x0, 0x0, ...}}}, ...})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/services/object/internal/client/client.go:253 +0x215
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/services/object/head.(*RemoteHeader).Head(0xc001d449d8, {0x1274c58, 0xc002776780}, 0xc002e61820)
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/services/object/head/remote.go:93 +0x47c
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).processNodes(0xc001d44a20, 0xc00346f110, {{0x1, 0x63, 0x6a, 0x41, 0xcc, 0x25, 0x3, 0xd5, ...}, ...}, ...)
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/services/policer/check.go:186 +0x605
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).processObject(0xc001d44a20, {0x12759b0, 0xc0000fb840}, {{0x1, 0x63, 0x6a, 0x41, 0xcc, 0x25, 0x3, ...}, ...})
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/services/policer/check.go:125 +0xbed
окт 19 19:39:59 az neofs-node[37515]: github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).shardPolicyWorker.func1()
окт 19 19:39:59 az neofs-node[37515]:         github.com/nspcc-dev/neofs-node/pkg/services/policer/process.go:65 +0x1a5
окт 19 19:39:59 az neofs-node[37515]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
окт 19 19:39:59 az neofs-node[37515]:         github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
окт 19 19:39:59 az neofs-node[37515]: created by github.com/panjf2000/ants/v2.(*goWorker).run
окт 19 19:39:59 az neofs-node[37515]:         github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65
окт 19 19:41:20 az neofs-node[37515]: 2022-10-19T19:41:20.731Z        info        event/listener.go:237        stop event listener by context        {"reason": "context canceled"}

And the same issue for neofs-ir:

окт 19 20:03:16 buky neofs-ir[62041]: 2022-10-19T20:03:16.292Z        info        audit/calculate.go:65        calculate audit settlements        {"current epoch": 2204}
окт 19 20:03:16 buky neofs-ir[62041]: 2022-10-19T20:03:16.342Z        info        settlement/handlers.go:18        audit processing finished        {"epoch": 2204}
окт 19 20:03:16 buky neofs-ir[62041]: 2022-10-19T20:03:16.376Z        info        audit/process.go:39        select containers for audit        {"epoch": 2204, "amount": 171}
окт 19 20:03:21 buky neofs-ir[62041]: 2022/10/19 20:03:21 worker exits from a panic: open gRPC connection: gRPC dial: context deadline exceeded
окт 19 20:03:21 buky neofs-ir[62041]: 2022/10/19 20:03:21 worker exits from panic: goroutine 127 [running]:
окт 19 20:03:21 buky neofs-ir[62041]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
окт 19 20:03:21 buky neofs-ir[62041]: panic({0xd64d80, 0xc0015ac5c0})
окт 19 20:03:21 buky neofs-ir[62041]:         runtime/panic.go:838 +0x207
окт 19 20:03:21 buky neofs-ir[62041]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).createForAddress(0xc001a55140, {{0x10078c0?, 0xc001a277a0?}})
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:62 +0x2d0
окт 19 20:03:21 buky neofs-ir[62041]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).client(0xc001a55140, {{0x10078c0?, 0xc001a277a0?}})
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:249 +0x125
окт 19 20:03:21 buky neofs-ir[62041]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).iterateClients.func1({{0x10078c0?, 0xc001a277a0?}})
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:109 +0xba
окт 19 20:03:21 buky neofs-ir[62041]: github.com/nspcc-dev/neofs-node/pkg/network.AddressGroup.IterateAddresses(...)
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/nspcc-dev/neofs-node/pkg/network/group.go:37
окт 19 20:03:21 buky neofs-ir[62041]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).iterateClients(0x7f57f9e30bc0?, {0x10017f0?, 0xc001a550e0?}, 0x30?)
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:99 +0xbb
окт 19 20:03:21 buky neofs-ir[62041]: github.com/nspcc-dev/neofs-node/pkg/network/cache.(*multiClient).ObjectSearchInit(0xc001600001?, {0x10017f0?, 0xc001a550e0?}, {{0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, 0x0, .>
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/nspcc-dev/neofs-node/pkg/network/cache/multi.go:189 +0x87
окт 19 20:03:21 buky neofs-ir[62041]: github.com/nspcc-dev/neofs-node/pkg/innerring/internal/client.Client.SearchSG({0xc0003260f0?, {0x1006ec0?, 0xc001a55140?}}, {{{0x10017f0, 0xc001a550e0}}, {0x0, 0x75, >
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/nspcc-dev/neofs-node/pkg/innerring/internal/client/client.go:69 +0x19e
окт 19 20:03:21 buky neofs-ir[62041]: github.com/nspcc-dev/neofs-node/pkg/innerring.ClientCache.ListSG({0xc00039db60, {0xffe3e8, 0xc001548680}, 0xc0003260f0, 0x12a05f200, 0x12a05f200, 0x12a05f200}, 0xc001>
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/nspcc-dev/neofs-node/pkg/innerring/rpc.go:229 +0x125
окт 19 20:03:21 buky neofs-ir[62041]: github.com/nspcc-dev/neofs-node/pkg/innerring/processors/audit.(*Processor).findStorageGroups(0xc00154cc00, {0x0, 0x75, 0xd0, 0x7b, 0x19, 0x12, 0x46, 0x7a, 0x7, ...},>
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/nspcc-dev/neofs-node/pkg/innerring/processors/audit/process.go:155 +0x762
окт 19 20:03:21 buky neofs-ir[62041]: github.com/nspcc-dev/neofs-node/pkg/innerring/processors/audit.(*Processor).processStartAudit(0xc00154cc00, 0x89c)
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/nspcc-dev/neofs-node/pkg/innerring/processors/audit/process.go:84 +0xdec
окт 19 20:03:21 buky neofs-ir[62041]: github.com/nspcc-dev/neofs-node/pkg/innerring/processors/audit.(*Processor).handleNewAuditRound.func1()
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/nspcc-dev/neofs-node/pkg/innerring/processors/audit/handlers.go:17 +0x25
окт 19 20:03:21 buky neofs-ir[62041]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
окт 19 20:03:21 buky neofs-ir[62041]: created by github.com/panjf2000/ants/v2.(*goWorker).run
окт 19 20:03:21 buky neofs-ir[62041]:         github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65
окт 19 20:08:04 buky neofs-ir[62041]: 2022-10-19T20:08:04.476Z        info        alphabet/handlers.go:11        tick        {"type": "alphabet gas emit"}