Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bunch of panics with shards in degraded-read-only mode #1974

Closed
cthulhu-rider opened this issue Oct 26, 2022 · 1 comment · Fixed by #1975
Closed

Bunch of panics with shards in degraded-read-only mode #1974

cthulhu-rider opened this issue Oct 26, 2022 · 1 comment · Fixed by #1975
Labels
bug Something isn't working neofs-storage Storage node application issues U3 Regular
Milestone
v0.34.0

Comments

@cthulhu-rider
Copy link
Contributor

Swithcing shards to the degraded-read-only mode causes panics.

Expected Behavior

Node goes normally.

Current Behavior

GC

2022/10/26 05:58:50 worker exits from a panic: runtime error: invalid memory address or nil pointer dereference
2022/10/26 05:58:50 worker exits from panic: goroutine 917 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0xf4c660, 0x1a62d70})
	runtime/panic.go:838 +0x207
go.etcd.io/bbolt.(*DB).beginTx(0x0)
	go.etcd.io/bbolt@v1.3.6/db.go:598 +0x27
go.etcd.io/bbolt.(*DB).Begin(0x50?, 0x30?)
	go.etcd.io/bbolt@v1.3.6/db.go:591 +0x2c
go.etcd.io/bbolt.(*DB).View(0xc000300f60?, 0xc001fc2df8)
	go.etcd.io/bbolt@v1.3.6/db.go:756 +0x3b
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/metabase.(*DB).IterateExpired(0x7f5af866b2f?, 0x68c61714000?, 0xc000118870?)
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/metabase/iterators.go:46 +0x49
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*Shard).getExpiredObjects(0x414445?, {0x126ee78, 0xc000148940}, 0x172188d3ee5b4981?, 0x0?)
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/gc.go:332 +0x79
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*Shard).collectExpiredLocks(0xc000118c30, {0x126ee78, 0xc000148940}, {0x1264660?, 0x1a29998?})
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/gc.go:316 +0x70
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*gc).listenEvents.func1()
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/gc.go:139 +0x3a
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65

Container load announcer

2022/10/26 05:58:51 worker exits from a panic: runtime error: invalid memory address or nil pointer dereference
2022/10/26 05:58:51 worker exits from panic: goroutine 911 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0xf4c660, 0x1a62d70})
	runtime/panic.go:838 +0x207
go.etcd.io/bbolt.(*DB).beginTx(0x0)
	go.etcd.io/bbolt@v1.3.6/db.go:598 +0x27
go.etcd.io/bbolt.(*DB).Begin(0x4b5d47?, 0x10?)
	go.etcd.io/bbolt@v1.3.6/db.go:591 +0x2c
go.etcd.io/bbolt.(*DB).View(0x40db85?, 0xc0003c35c8)
	go.etcd.io/bbolt@v1.3.6/db.go:756 +0x3b
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/metabase.(*DB).Containers(0xc00012a240?)
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/metabase/containers.go:14 +0xe5
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*Shard).ListContainers(0xc000492630?, {})
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/list.go:96 +0x1d
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).listContainers.func1({0xc000271330?, 0xc000118b90?})
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/container.go:125 +0x66
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).iterateOverUnsortedShards(0xc001bca018?, 0xc0003c3830)
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:224 +0x5f
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).listContainers(0xc000118aa0)
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/container.go:124 +0x1a5
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).ListContainers.func1()
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/container.go:98 +0x33
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).execIfNotBlocked(0x0?, 0x10fbc04?)
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/control.go:142 +0xcf
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).ListContainers(0x0?, {})
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/container.go:97 +0x68
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.ListContainers(0x1a66ce0?)
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/container.go:109 +0x19
main.(*localStorageLoad).Iterate(0xc001a81990, 0x112baf8, 0xc00150d200)
	github.com/nspcc-dev/neofs-node/cmd/neofs-node/container.go:447 +0x4d
github.com/nspcc-dev/neofs-node/pkg/services/container/announcement/load/controller.(*announceContext).announce(0xc000492570)
	github.com/nspcc-dev/neofs-node/pkg/services/container/announcement/load/controller/calls.go:84 +0x34f
github.com/nspcc-dev/neofs-node/pkg/services/container/announcement/load/controller.(*Controller).Start(0xc00183d610?, {0xc00183d5fc?})
	github.com/nspcc-dev/neofs-node/pkg/services/container/announcement/load/controller/calls.go:52 +0x5e
main.initContainerService.func3({0x1264760?, 0x1a29990?})
	github.com/nspcc-dev/neofs-node/cmd/neofs-node/container.go:185 +0x39
github.com/nspcc-dev/neofs-node/pkg/morph/event.WorkerPoolHandler.func1.1()
	github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:88 +0x25
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65

Notificator

2022/10/26 05:58:50 worker exits from a panic: runtime error: invalid memory address or nil pointer dereference
2022/10/26 05:58:50 worker exits from panic: goroutine 914 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0xf4c660, 0x1a62d70})
	runtime/panic.go:838 +0x207
go.etcd.io/bbolt.(*DB).beginTx(0x0)
	go.etcd.io/bbolt@v1.3.6/db.go:598 +0x27
go.etcd.io/bbolt.(*DB).Begin(0xc001414200?, 0x10?)
	go.etcd.io/bbolt@v1.3.6/db.go:591 +0x2c
go.etcd.io/bbolt.(*DB).View(0x40db85?, 0xc00144f478)
	go.etcd.io/bbolt@v1.3.6/db.go:756 +0x3b
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/metabase.(*DB).Containers(0xc00012a240?)
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/metabase/containers.go:14 +0xe5
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*Shard).ListContainers(0xc00001bd40?, {})
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/list.go:96 +0x1d
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).listContainers.func1({0xc000271330?, 0xc000118b90?})
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/container.go:125 +0x66
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).iterateOverUnsortedShards(0xc00000d380?, 0xc00144f6e0)
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:224 +0x5f
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).listContainers(0xc000118aa0)
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/container.go:124 +0x1a5
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).ListContainers.func1()
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/container.go:98 +0x33
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).execIfNotBlocked(0x1?, 0x60?)
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/control.go:142 +0xcf
github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).ListContainers(0xc000114180?, {})
	github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/container.go:97 +0x68
main.(*notificationSource).Iterate(0xc00198af60, 0x3, 0x1?)
	github.com/nspcc-dev/neofs-node/cmd/neofs-node/notificator.go:28 +0x13a
github.com/nspcc-dev/neofs-node/pkg/services/notificator.(*Notificator).ProcessEpoch(0xc001ad2ab0, 0x3)
	github.com/nspcc-dev/neofs-node/pkg/services/notificator/service.go:78 +0x14b
main.initNotifications.func1({0x12647a0?, 0xc0003b2570?})
	github.com/nspcc-dev/neofs-node/cmd/neofs-node/notificator.go:153 +0xc5
github.com/nspcc-dev/neofs-node/pkg/morph/event.WorkerPoolHandler.func1.1()
	github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:88 +0x25
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65

Steps to Reproduce (for bugs)

  1. Up NeoFS DevEnv make up/basic
  2. Create container neofs-cli -c cli.yaml container create --policy 'REP 1' --await -v
  3. Put single object neofs-cli -c cli.yml object put --cid <cnr> --file Makefile -v
  4. Look up for the node which holds the object
  5. Swtich the mode of all shards neofs-cli control shards set-mode --mode degraded-read-only --endpoint s0X.neofs.devenv:8081 -w ${DEVENV_PATH}/services/storage/wallet0X.json --all
    6.. Tick epoch make tick.epoch

Regression

Maybe.

Your Environment

  • Version used: 0.33.0-73-g3c6daa29
@cthulhu-rider cthulhu-rider added bug Something isn't working triage neofs-storage Storage node application issues labels Oct 26, 2022
@alexchetaev alexchetaev added the U3 Regular label Oct 26, 2022
@cthulhu-rider
Copy link
Contributor Author

This is caused by metabase.DB.boltDB == nil.

@fyrchik fyrchik mentioned this issue Oct 26, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working neofs-storage Storage node application issues U3 Regular
Projects
None yet
Milestone
v0.34.0
Development

Successfully merging a pull request may close this issue.

Make metabase.ContainerSize actually read-only fyrchik/neofs-node
3 participants
@fyrchik @alexchetaev @cthulhu-rider