Volumes not created on pristine fast/slow drive setup

[chymian]

Describe the bug
no active volumes are generated on a pristine fast/slow disk setup.

System Setup
Master:

usr/local/sbin/weed master -ip.bind 0.0.0.0 -ip 10.11.11.182 \
  -defaultReplication 000 \
  -volumeSizeLimitMB 1024 \
  -mdir /srv/data_fast/weed.dei.privat/master  \
  -volumePreallocate

Volume:

/usr/local/sbin/weed volume -port 8080 -publicUrl http://weed.dei.privat -max 100 \
 -ip.bind 0.0.0.0 -ip 10.11.11.182  \
 -mserver 10.11.11.182:9333 \
 -disk hdd,ssd -dir /srv/data/weed.dei.privat/volume,/srv/data_fast/weed.dei.privat/volume  \
 -compactionMBps 30  -minFreeSpace 1 -dataCenter DefaultDataCenter -rack DefaultRack

Filer:

/usr/local/sbin/weed filer -port 8888 -ip.bind 0.0.0.0 -ip 10.11.11.182 \
 -master 10.11.11.182:9333 \
 -dataCenter DefaultDataCenter -rack DefaultRack \
 -s3 -s3.config=/srv/data/weed.dei.privat/.seaweedfs/s3.json 

Status-Data:

> cluster.ps
the cluster has 1 filers
  * 10.11.11.182:8888 (30GB 3.07 fbd99d53c1e45ef12c49ad1ec043c918f4da94e8)
    signature: 411477689
> cluster.check
Topology volumeSizeLimit:1024 MB hdd(volume:0/100 active:0 free:100 remote:0) ssd(volume:0/100 active:0 free:100 remote:0)
error: Need to a hdd disk type!

> fs.configure
{

}

> volume.list
Topology volumeSizeLimit:1024 MB hdd(volume:0/100 active:0 free:100 remote:0) ssd(volume:0/100 active:0 free:100 remote:0)
  DataCenter DefaultDataCenter hdd(volume:0/100 active:0 free:100 remote:0) ssd(volume:0/100 active:0 free:100 remote:0)
    Rack DefaultRack hdd(volume:0/100 active:0 free:100 remote:0) ssd(volume:0/100 active:0 free:100 remote:0)
      DataNode 10.11.11.182:8080 hdd(volume:0/100 active:0 free:100 remote:0) ssd(volume:0/100 active:0 free:100 remote:0)
        Disk hdd(volume:0/100 active:0 free:100 remote:0)
        Disk hdd total size:0 file_count:0 
        Disk ssd(volume:0/100 active:0 free:100 remote:0)
        Disk ssd total size:0 file_count:0 
      DataNode 10.11.11.182:8080 total size:0 file_count:0 
    Rack DefaultRack total size:0 file_count:0 
  DataCenter DefaultDataCenter total size:0 file_count:0 
total size:0 file_count:0 
> 

• OS version
  Debian 11 bullseye

• output of weed version
  version 30GB 3.07 fbd99d5 linux amd64
  also on latest 3.06, 30GB

• if using filer, show the content of filer.toml
  none

Expected behavior
like in singe volume setup before, Volumes get generated and used.

Additional context
I'm modifying an ansible role to setup a one-host weed-cluster, running in a LXContainer (@proxmox). the filer with 2 mounted host-zfs-drives, with two diff. TAGs.
Always removing the old setup before rerun.

While using only on drive, everything works as expected.
when using two drives, no volumes get created.

in the one volume setup, only /srv/data/weed.dei.privat/volume, was used. no -disk option.

two disk setup:
first try:
-disk ssd,nvme -dir /srv/data/weed.dei.privat/volume,/srv/data_fast/weed.dei.privat/volume

according to my misinterpretation of the error message (s.u.) from 'cluster.check': error: Need to a hdd disk type!, I changed to:

-disk hdd,ssd -dir /srv/data/weed.dei.privat/volume,/srv/data_fast/weed.dei.privat/volume
with out effect…

Dir-structure on /srv/

tree */weed.dei.* -L 3
data_fast/weed.dei.privat
├── master
│   ├── conf
│   ├── log
│   └── snapshot
└── volume
    └── vol_dir.uuid
data/weed.dei.privat
├── filer
│   └── filerldb2
│       ├── 00
│       ├── 01
│       ├── 02
│       ├── 03
│       ├── 04
│       ├── 05
│       ├── 06
│       └── 07
└── volume
    └── vol_dir.uuid

[chrislusf]

did you upload some files?

[chymian]

it seems to be a workflow problem.
when I just create a bucket & upload via s3 (mcli), everything is fine.

testing for the ansible role involved as next step, setting up dir & path settings, with out uploading a file in between.

the following would be next, which did not work entirely:

$ BUCKET_NAME=backup                                         
# All S3 Buckets to warm/ssd
echo fs.configure -locationPrefix=/buckets/   -disk=ssd  -volumeGrowthCount 1 -apply | weed shell
# create default bucket     
echo s3.bucket.create --name=$BUCKET_NAME | weed shell
master: localhost:9333 filers: [10.11.11.182:8888]
{                                                            
  "locations": [        
    {                       
      "locationPrefix": "/buckets/",
      "diskType": "ssd",
      "volumeGrowthCount": 1
    }         
  ]         
}                                                            
master: localhost:9333 filers: [10.11.11.182:8888]
create bucket under /buckets
created bucket backup
root@weed /srv
$ # create dirs                                                                                                            
for i in /scratch /srv/files; do                             
  echo fs.mkdir ${i} -apply | weed shell                     
done     
                              
# fast scratch-drive with 7d TTL                             
echo fs.configure -locationPrefix=/scratch/   -disk=nvme -volumeGrowthCount 1 -ttl 7d -apply | weed shell                  
master: localhost:9333 filers: [10.11.11.182:8888]
master: localhost:9333 filers: [10.11.11.182:8888]         
master: localhost:9333 filers: [10.11.11.182:8888]           
{         
  "locations": [    
    {               
      "locationPrefix": "/buckets/",                         
      "diskType": "ssd",
      "volumeGrowthCount": 1
    },             
    {    
      "locationPrefix": "/scratch/",                         
      "ttl": "7d",
      "diskType": "nvme",
      "volumeGrowthCount": 1
    }        
  ]                                                          
}                                                            
root@weed /srv
$ # archive to slower disk
echo fs.configure -locationPrefix=/srv/files/ -disk=ssd  -volumeGrowthCount 1 -apply | weed shell                          
master: localhost:9333 filers: [10.11.11.182:8888]           
{                         
  "locations": [             
    {                     
      "locationPrefix": "/buckets/",                         
      "diskType": "ssd",
      "volumeGrowthCount": 1
    },
    {         
      "locationPrefix": "/scratch/",                         
      "ttl": "7d",          
      "diskType": "nvme",                                                                                                  
      "volumeGrowthCount": 1
    },                                                       
    {                                                        
      "locationPrefix": "/srv/files/",                       
      "diskType": "ssd",
      "volumeGrowthCount": 1
    }                                                        
  ]                     
}                           
root@weed /srv
$ weed shell
master: localhost:9333 filers: [10.11.11.182:8888]           
> ls                                                         
buckets
> ls buckets
backup                     
> 

the dirs in step 2 # create dirs don't get created.
instead I get the following in the logs, last line repeatedly spamming:

Mai 31 19:47:41 weed weed-master[829574]: I0531 19:47:41 29574 master_grpc_server.go:332] + client .adminShell@            
Mai 31 19:47:42 weed weed-master[829574]: I0531 19:47:42 29574 master_grpc_server.go:348] - client .adminShell@            
Mai 31 19:47:42 weed weed-filer[829908]: I0531 19:47:42 29908 filer_on_meta_event.go:50] procesing directory:"/etc/seaweedf
s"  event_notification:{new_entry:{name:"filer.conf"  attributes:{file_size:127  mtime:1654019262  file_mode:420  crtime:16
54019262}  content:"{\n  \"locations\": [\n    {\n      \"locationPrefix\": \"/buckets/\",\n      \"diskType\": \"ssd\",\n      \"volumeGrowthCount\": 1\n    }\n  ]\n}"}  delete_chunks:true  new_parent_path:"/etc/seaweedfs"  signatures:561898289}
  ts_ns:1654019262128485006                                                                                                
Mai 31 19:47:42 weed weed-master[829574]: I0531 19:47:42 29574 master_grpc_server.go:332] + client .adminShell@            Mai 31 19:47:42 weed weed-master[829574]: I0531 19:47:42 29574 master_grpc_server.go:348] - client .adminShell@            
Mai 31 19:48:13 weed weed-master[829574]: I0531 19:48:13 29574 master_grpc_server.go:332] + client .adminShell@            
Mai 31 19:48:13 weed weed-master[829574]: I0531 19:48:13 29574 master_grpc_server.go:348] - client .adminShell@            Mai 31 19:48:13 weed weed-master[829574]: I0531 19:48:13 29574 master_grpc_server.go:332] + client .adminShell@            
Mai 31 19:48:13 weed weed-master[829574]: I0531 19:48:13 29574 master_grpc_server.go:348] - client .adminShell@            
Mai 31 19:48:14 weed weed-filer[829908]: I0531 19:48:14 29908 filer_notify.go:103] metadata log write failed /topics/.system/log/2022-05-31/17-47.3c879de8: AssignVolume: rpc error: code = Unknown desc = no free volumes left for {"replication":{},
"ttl":{"Count":0,"Unit":0},"preallocate":1073741824}                                                                       

[chymian]

additionally, I realized that I never did the next steps to setup dir & path on a single volume cluster, cause it was later in the ansible role…
so I checked the same as above, but single volume setup & took out the -disk=xxx from the cmd list above and it worked., volumes get created.
the dir /topics get created.
but on the first run the mkdir-cmd in step 2, did not create the dirs.
but I could then create them manually.
turned out, that the -apply behind the fs.mkdir hinders the cmd to work.

[chrislusf]

There is a bug to report error that is fixed in 89948a3

Is this issue resolved? Just need to ensure there are hdd volumes available.

[chymian]

Just need to ensure there are hdd volumes available.

that does mean, one must use the hdd-tag! yes?
i.e -disk ssd,hdd is valid
and -disk nvme,ssd is not valid?

IMHO: on a multi-tiered setup with dual-drive laptop, on has to name the new PCI3 & PCI4 drives as hdd,ssd.
doesn't that must have give away flexibility?

on 3.08:
its working with -disk hdd,sdd.
its not working with -disk sdd,nvme.

maybe related to that:
I'm tackling to tell filer to put his db2/3 files on the fast path (also $HOME), but it insist on putting them on the hdd-tagged drive.

if there is no ./filer/-dir in the same path as the hdd ./volume/-dir is located, filer fails to start.

even a precreated, distinct path in filer.toml like here fails:

recursive_delete = true

[leveldb2]
# local on disk, mostly for simple single-machine setup, fairly scalable
# faster than previous leveldb, recommended.
enabled = false
dir = "./filerldb2"                    # directory to store level db files

[leveldb3]
# similar to leveldb2.
# each bucket has its own meta store.
enabled = true
dir = "/srv/data_fast/weed.dei.privat/filer/filerldb3"                    # directory to store level db files

what I'm missing?