neofs-node paniced while flushing write-cache after node rebooted

[anikeev-yadro]

Test ran on cluster prefilled to 1%.

Steps to Reproduce (for bugs)

1. Preset nodes with k6

./scenarios/preset/preset_grpc.py --size 1000 --containers 40 --out /var/data/tmp/grpc_030123-1000.json --endpoint 10.78.70.118:8080 --preload_obj 20  --wallet scenarios/files/wallet.json --config scenarios/files/empty_pass.yaml

2. Start k6 load

./k6 run -e DURATION=600 -e WRITERS=50 -e READERS=50 -e DELETERS=0 -e DELETE_AGE=10 -e REGISTRY_FILE=/var/data/tmp/grpc_reg_030123-1000-1.json -e WRITE_OBJ_SIZE=1000  -e GRPC_ENDPOINTS=10.78.70.118:8080,10.78.70.119:8080,10.78.70.121:8080 -e PREGEN_JSON=/var/data/tmp/grpc_030123-1000.json -e STREAM_TIMEOUT=30  -e SLEEP_READ=0.2  -e SLEEP_WRITE=0.2 scenarios/grpc.js

3. Reboot node VEDI

anikeev@NB-1670:~/neofs/xk6-neofs$ date;ipmitool -I lanplus -H 10.78.68.120 -U admin -P admin power reset
Fri Feb  3 09:30:47 MSK 2023
Chassis Power Control: Reset

4. After VEDI restarted we had panic while neofs-node flushing writre-cache:

Feb 03 09:54:33 vedi neofs-node[8534]: 2023-02-03T09:54:33.891Z        info        log/log.go:13        local object storage ope
ration        {"shard_id": "EaE3hhKvbuDyRbzJrQhET4", "address": "J1qUxZ43tDzKEwi3kPayotCBwEtTDtusGMaGFDrUBxnp/EMSVuanjASK46m9gbr
Y2pfTERpsxzmqNUuAz13gNPvsw", "type": "write-cache", "op": "db DELETE"}
Feb 03 09:54:33 vedi neofs-node[8534]: panic: runtime error: index out of range [435] with length 205
Feb 03 09:54:33 vedi neofs-node[8534]: goroutine 275 [running]:
Feb 03 09:54:33 vedi neofs-node[8534]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/writecache.(*cache).initFlushMarks.func3(0xc000610d00?)
Feb 03 09:54:33 vedi neofs-node[8534]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/writecache/init.go:82 +0x105
Feb 03 09:54:33 vedi neofs-node[8534]: go.etcd.io/bbolt.(*DB).Update(0xfd1bc0?, 0xc078df8870)
Feb 03 09:54:33 vedi neofs-node[8534]:         go.etcd.io/bbolt@v1.3.6/db.go:741 +0x82
Feb 03 09:54:33 vedi neofs-node[8534]: go.etcd.io/bbolt.(*DB).Batch(0xc000638480, 0xc078df8870)
Feb 03 09:54:33 vedi neofs-node[8534]:         go.etcd.io/bbolt@v1.3.6/db.go:819 +0x325
Feb 03 09:54:33 vedi neofs-node[8534]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/writecache.(*cache).initFlushMarks(0xc0001211e0)
Feb 03 09:54:33 vedi neofs-node[8534]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/writecache/init.go:79 +0x29f
Feb 03 09:54:33 vedi neofs-node[8534]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/writecache.(*cache).Init(0xc0002c64d0?)
Feb 03 09:54:33 vedi neofs-node[8534]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/writecache/writecache.go:147 +0x1e
Feb 03 09:54:33 vedi neofs-node[8534]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*Shard).Init(0xc0000ef950)
Feb 03 09:54:33 vedi neofs-node[8534]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/control.go:117 +0x64f
Feb 03 09:54:33 vedi neofs-node[8534]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Init.func1({0xc000816810, 0x16}, 0x0?)
Feb 03 09:54:33 vedi neofs-node[8534]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/control.go:65 +0x6e
Feb 03 09:54:33 vedi neofs-node[8534]: created by github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Init
Feb 03 09:54:33 vedi neofs-node[8534]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/control.go:63 +0x12e
Feb 03 09:54:36 vedi systemd[1]: neofs-storage.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 03 09:54:36 vedi systemd[1]: neofs-storage.service: Failed with result 'exit-code'.

Version

NeoFS Storage node
Version: v0.35.0-24-ga96dabf7-dirty
GoVersion: go1.18.4

Logs

node_panic_vedi.log.tar.gz

Your Environment

Server setup and configuration:
HW, 4 servers, 4 SN, 4 http qw, 4 s3 gw

Operating System and version (uname -a):
linux vedi 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

[fyrchik]

Closed via #2257 .