Cant open leveldb file

[ysyyork]

Describe the bug
I'm using the official helm deployment and set the volume server replica to 2. It was working well with memory index for volume server but when I change it to leveldb, some pods randomly failed and keep the failure status forever. The log I got is below:

I0307 09:01:46     1 config.go:29] Reading security.toml from 
I0307 09:01:46     1 config.go:33] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /usr/local/etc/seaweedfs /etc/seaweedfs]"
I0307 09:01:46     1 file_util.go:23] Folder /data Permission: -rwxrwxr-x
I0307 09:01:46     1 volume_info.go:22] maybeLoadVolumeInfo checks /data/4.vif
I0307 09:01:46     1 volume_info.go:34] maybeLoadVolumeInfo reads /data/4.vif
I0307 09:01:46     1 volume_info.go:41] maybeLoadVolumeInfo Unmarshal volume info /data/4.vif
I0307 09:01:46     1 volume_loading.go:106] open to write file /data/4.idx
I0307 09:01:46     1 volume_loading.go:128] loading leveldb /data/4.ldb
I0307 09:01:46     1 needle_map_leveldb.go:34] Opening /data/4.ldb...
I0307 09:01:46     1 volume_loading.go:135] loading leveldb /data/4.ldb error: resource temporarily unavailable
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x188 pc=0xe5c69e]

goroutine 146 [running]:
github.com/syndtr/goleveldb/leveldb.(*DB).setClosed(...)
	/root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:225
github.com/syndtr/goleveldb/leveldb.(*DB).Close(0x0, 0x0, 0x0)
	/root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:1117 +0x3e
github.com/chrislusf/seaweedfs/weed/storage.(*LevelDbNeedleMap).Close(0xc000193c20)
	/go/src/github.com/chrislusf/seaweedfs/weed/storage/needle_map_leveldb.go:150 +0xb6
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).load.func1(0xc000cb1a5d, 0xc0004f8ea0)
	/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_loading.go:32 +0x95
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).load(0xc0004f8ea0, 0xc0004f0101, 0x1, 0x0, 0x26d4d00, 0x357a738)
	/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_loading.go:172 +0x3be
github.com/chrislusf/seaweedfs/weed/storage.NewVolume(0x7ffd03b27cb4, 0x5, 0x7ffd03b27cb4, 0x5, 0x0, 0x0, 0x4, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume.go:59 +0x156
github.com/chrislusf/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc000172980, 0x271a660, 0xc00011d450, 0x1, 0x0)
	/go/src/github.com/chrislusf/seaweedfs/weed/storage/disk_location.go:122 +0x38e
github.com/chrislusf/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2(0xc000626070, 0xc0004d44e0, 0xc000172980, 0x1)
	/go/src/github.com/chrislusf/seaweedfs/weed/storage/disk_location.go:162 +0x87
created by github.com/chrislusf/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
	/go/src/github.com/chrislusf/seaweedfs/weed/storage/disk_location.go:159 +0xeb

But some pods are working fine. Screenshot as below:

System Setup

• List the command line to start "weed master", "weed volume", "weed filer", "weed s3", "weed mount".
  I'm using the helm charts

• OS version
  Ubuntu 18.04 with k8s 1.20.0

• output of weed version
  2.23

• if using filer, show the content of filer.toml
  IDK where is this file.

Expected behavior
It should work smoothly

[chrislusf]

Added a fix. The nil happened when volume loading failed.

But it is not clear why the volume loading failed. The "resource temporarily unavailable" is not very clear.

[PeterCxy]

resource temporarily unavailable is normally related to limits on open file handles or nprocs. This would also explain why the failure seems random.

[ysyyork]

ok sweet!

…

On Sun, Mar 7, 2021 at 8:44 PM Peter Cai ***@***.***> wrote: resource temporarily unavailable is normally related to limits on open file handles or nprocs. This would also explain why the failure seems random. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1869 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTCFZID7TUUY4MSJGONAWLTCNYLHANCNFSM4YXUX6VA> .