seaweedfs crashed

[guol-fnst]

version:2.65 linux amd64
Command:
nohup weed server -ip=90.90.90.12 -dir=/mnt/vol/object_444/data_1 -s3 -s3.port=29000 -s3.allowEmptyFolder -volume.port=27000 -filer.port=28000 -volume.max=4294967295 &
I use cosbench to test performance of weed. When running the test , I quickly encountered two serious problems：
1） weed crashed

I0928 22:04:43 61566 volume_loading.go:133] loading index /mnt/vol/object_444/data_1/abc203726_55138.idx to memory
I0928 22:04:43 61566 store.go:135] add volume 55138
I0928 22:04:44 61566 topology.go:208] removing volume info: Id:55135, Size:0, ReplicaPlacement:000, Collection:abc203726, Version:3, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false
I0928 22:04:44 61566 volume_layout.go:349] Volume 55135 becomes unwritable
I0928 22:04:44 61566 topology.go:208] removing volume info: Id:55137, Size:0, ReplicaPlacement:000, Collection:abc203726, Version:3, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false
I0928 22:04:44 61566 volume_layout.go:349] Volume 55137 becomes unwritable
I0928 22:04:44 61566 topology.go:208] removing volume info: Id:55136, Size:0, ReplicaPlacement:000, Collection:abc203726, Version:3, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false
I0928 22:04:44 61566 volume_layout.go:349] Volume 55136 becomes unwritable
I0928 22:04:44 61566 master_grpc_server.go:113] master see deleted volume 55135 from 90.90.90.12:27000
I0928 22:04:44 61566 master_grpc_server.go:113] master see deleted volume 55137 from 90.90.90.12:27000
I0928 22:04:44 61566 master_grpc_server.go:113] master see deleted volume 55136 from 90.90.90.12:27000
I0928 22:04:44 61566 master_grpc_server.go:154] master send to filer@90.90.90.12:28000: url:"90.90.90.12:27000"  public_url:"90.90.90.12:27000"  deleted_vids:55135  deleted_vids:55137  deleted_vids:55136  data_center:"DefaultDataCenter"
panic: runtime error: index out of range [0] with length 0

goroutine 8037320 [running]:
github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLocationList).Head(0xd53fc12f48)
	/root/original_weed/seaweedfs/weed/topology/volume_location_list.go:31 +0x4a
github.com/chrislusf/seaweedfs/weed/topology.(*Topology).PickForWrite(0xc0003c81e0, 0x1, 0xd54ff8fe30)
	/root/original_weed/seaweedfs/weed/topology/topology.go:148 +0x774
github.com/chrislusf/seaweedfs/weed/server.(*MasterServer).Assign(0xc000ed6f00, {0x31bea78, 0xd5529cd950}, 0xd5529de6e0)
	/root/original_weed/seaweedfs/weed/server/master_grpc_server_volume.go:150 +0x7d2
github.com/chrislusf/seaweedfs/weed/pb/master_pb._Seaweed_Assign_Handler({0x2e726a0, 0xc000ed6f00}, {0x31bea78, 0xd5529cd950}, 0xd5529a5e60, 0x0)
	/root/original_weed/seaweedfs/weed/pb/master_pb/master.pb.go:4460 +0x385
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00068ad00, {0x31e7e90, 0xc000282480}, 0xd5529bf100, 0xc001069b00, 0x431edd8, 0x0)
	/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1082 +0x14c9
google.golang.org/grpc.(*Server).handleStream(0xc00068ad00, {0x31e7e90, 0xc000282480}, 0xd5529bf100, 0x0)
	/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1405 +0x852
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0x11d
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0x125
I0928 22:04:47 02813 master.go:165] current: 90.90.90.12:9333 peers:
I0928 22:04:47 02813 file_util.go:23] Folder /mnt/vol/object_444/data_1 Permission: -rwxr-xr-x
I0928 22:04:47 02813 file_util.go:23] Folder /mnt/vol/object_444/data_1 Permission: -rwxr-xr-x
I0928 22:04:47 02813 master.go:165] current: 90.90.90.12:9333 peers:90.90.90.12:9333
I0928 22:04:47 02813 master_server.go:116] Volume Size Limit is 30000 MB
I0928 22:04:47 02813 master_server.go:210] adminScripts:
I0928 22:04:47 02813 master.go:120] Start Seaweed Master 30GB 2.65  at :9333
F0928 22:04:47 02813 master.go:123] Master startup error: listen tcp :9333: bind: address already in use
goroutine 44 [running]:

2) weed show no free volume left ,but --volume.max was set to 4294967295 . After restarting weed, the error disapperd, but a few minutes later ,it shows again. I used weed shell to list volumes, and found that the number of volume was increasing

Here is th weed log.
weed.log

[chrislusf]

1. Thanks for the details. Added a fix.
2. Seems you need to wait a bit before throwing traffic. Seems there are many volumes and the volume server may take a longer time to start. The master did not receive the volumes information from the volume servers yet.