preseed storage filtering not clear / failing to discover

[wideawakening]

i'm not able to filter the two available disks on each node.
ex.: on pc7b-ra8-n2: nvme1n1, nvme3n1

microcloud version: 2/stable, 2.1.0-3e8b183, rev 1144

ubuntu@pc7b-ra8-n2:~$ lsblk -d -o NAME,ROTA,TYPE,MODEL
NAME    ROTA TYPE MODEL
... loops
nvme2n1    0 disk INTEL SSDPF2KE064T1O
nvme1n1    0 disk INTEL SSDPF2KE064T1O
nvme0n1    0 disk INTEL SSDPF2KE064T1O
nvme3n1    0 disk INTEL SSDPF2KE064T1O

ubuntu@pc7b-ra8-n2:~$ lsblk
NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
...
nvme2n1               259:0    0   5.8T  0 disk  
├─nvme2n1p1           259:1    0   488M  0 part  /boot/efi
└─nvme2n1p2           259:2    0   5.8T  0 part  
  └─md1                 9:1    0   5.8T  0 raid1 
    ├─vg0-lvephemeral 252:0    0   5.3T  0 lvm   
    ├─vg0-lvlog       252:1    0  93.1G  0 lvm   /var/log
    └─vg0-lvroot      252:2    0 465.7G  0 lvm   /
nvme1n1               259:3    0   5.8T  0 disk  
nvme0n1               259:4    0   5.8T  0 disk  
├─nvme0n1p1           259:6    0   488M  0 part  
└─nvme0n1p2           259:7    0   5.8T  0 part  
  └─md1                 9:1    0   5.8T  0 raid1 
    ├─vg0-lvephemeral 252:0    0   5.3T  0 lvm   
    ├─vg0-lvlog       252:1    0  93.1G  0 lvm   /var/log
    └─vg0-lvroot      252:2    0 465.7G  0 lvm   /
nvme3n1               259:5    0   5.8T  0 disk  

i tried several approaches

1 - partitioned/partitions attribute not usable
✅ (removed from documentation; filter not available)

seems partitioned attribute is not there anymore
partitions does exist though https://github.com/canonical/lxd/blob/c86603236167a43836c2766647e2fac97d79f899/shared/api/resource.go#L591
seems this attributed should not be used eitherway, since the partitioned disks are ignored https://github.com/canonical/microcloud/blob/main/service/system_information.go#L105

i also tried partritions == nil or len(partitions) == 0

##  Error: Remote storage filter cannot be defined with find_min less than 1                                                           
storage:
  ceph:
    - find:  partitioned == false 
      find_min: 1
      find_max: 2
      wipe: true

2 - should find filter be mandatory? i just want two available disks

## Error: Received empty remote disk filter
storage:
          ceph:
            - find_min: 1
              find_max: 2
              wipe: true
              encrypt: true

3 - does not detect SSD disks

## Failed to find at least 1 disks for filter "type == ssd"
        storage:
          ceph:
            - find: type == ssd
              find_min: 1
              find_max: 2
              wipe: true
              encrypt: true

4 - more than two available disks, when only 2 are unpartitioned

## Error: Found more than 2 disks for filter "size > 10GiB"
        storage:
          ceph:
            - find: size > 10GiB
              find_min: 1
              find_max: 2
              wipe: true
              encrypt: true

maybe i'm misunderstanding how this filtering is used, but i would request

1. make the preseed disk filtering easier to understand, by having a microcloud generated log of the available disks, with the ResourcesStorageDisk attribute values. i'm using cloud-init right now, in case that gives more context

2. the current format of pre-seed explanation in the docs https://documentation.ubuntu.com/microcloud/latest/microcloud/how-to/initialize/#minimal-preseed-using-multicast-discovery
   is not very easy to read due to the horizontal scroll.
   maybe having a non-code, per-attribute explanation would be better

[roosterfish]

Hi, thanks for reporting this.

Being able to filter for non-partitioned disks clearly is a bug in the docs as those disks are already filtered out before entering the selection stage.

The error from 2) seems valid as it seems you have to provide at least one filter criteria using find:. But we could discuss if its absence will translate to all disks being accepted at this stage.

Regarding 3), if you want to search for an SSD, rather use type == nvme.

4. looks odd, can you please show the full output of lsblk on this system? It looks there are more than two disks which are applicable for selection.

[wideawakening]

hey julian! thanks for triaging the issue.

i have no access to the cluster anymore (it was canonicals pc7b, as part of a validation pulse), but i'm pretty sure the remaining items of the lsblk from the first message were just snaps, so that should be all, 2 un-partitioned disks, not fitting the 4th. use case criteria.

not all nodes had same disk specs (not homogeneous), that's why i thought it fit perfectly the use-case of the pressed disk discovery.

as for 3. i will try it out next time, but regarding filtering, in general, it would help a lot having request 1 (availble disk attributes log); that way we could see the ResourcesStorageDisk log of what microcloud detected and filter disks based on it.

[roosterfish]

but regarding filtering, in general, it would help a lot having request 1 (availble disk attributes log)

I like this idea. Could be part of the debug logging. Unfortunately right now there is no easy way to enable debug logging for the MicroCloud daemon but I'll create a separate issue so this can be enabled in the snap, same as we have it in LXD.

[roosterfish]

I'll create a separate issue so this can be enabled in the snap, same as we have it in LXD.

#806

[wideawakening]

another try/context
in this case i'm relying on maas and despite having a single bucket i'm not getting deterministic on where OS partition gets assigned, so i wanted to rely again on storage filtering.

root@mc-01:~# lsblk | grep -v loop
NAME                 MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                    8:0    0   3.5T  0 disk 
├─sda1                 8:1    0   952M  0 part /boot/efi
└─sda2                 8:2    0   3.2T  0 part 
  ├─vol--data-vartmp 252:0    0  93.1G  0 lvm  /var/tmp
  ├─vol--data-varlog 252:1    0 465.7G  0 lvm  /var/log
  ├─vol--data-var    252:2    0 651.9G  0 lvm  /var
  ├─vol--data-home   252:3    0 279.4G  0 lvm  /home
  └─vol--data-root   252:4    0   1.5T  0 lvm  /
sdb                    8:16   0   3.5T  0 disk 
sdc                    8:32   0   3.5T  0 disk 
sdd                    8:48   0   3.5T  0 disk 
sde                    8:64   0   3.5T  0 disk 
sdf                    8:80   0   3.5T  0 disk 
sdg                    8:96   0   3.5T  0 disk 



root@mc-01:~# cat foo.yaml | microcloud preseed 
Waiting for services to start ...
Searching for joining systems (mc-02, mc-03)
Detected joining system "mc-02" at "10.15.40.12"
Detected joining system "mc-03" at "10.15.40.13"
Warning: OSD host count is less than 3. Distributed storage is not fault-tolerant
Error: Failed to find at least 6 disks for filter "type == nvme"


root@mc-01:~# cat foo.yaml | microcloud preseed
Waiting for services to start ...       
Searching for joining systems (mc-02, mc-03) 
Detected joining system "mc-03" at "10.15.40.13"    
Detected joining system "mc-02" at "10.15.40.12"  
Warning: OSD host count is less than 3. Distributed storage is not fault-tolerant  
Error: Failed to find at least 6 disks for filter "type == ssd"    

# these were with find_min: filter
Error: Found more than 6 disks for filter "size > 10GiB"
Error: Found more than 7 disks for filter "size > 10GiB"

despite having only 6 non-partitioned disks, at some point it only worked with find_min:9 (did not try 8) when i used the size filter.

i was intrigued by the type, and found the tip i wanted here

root@mc-01:~# sudo microceph disk list --json
{"ConfiguredDisks":[],"AvailableDisks":[
{"Model":"Dell DC NVMe CD8"
,"Size":"3.49TiB",
"Type":"scsi",
"Path":"/dev/disk/by-id/scsi-SNVMe_Dell_DC_NVMe_CD8_Z440A16JTSRJ"},...}

so, scsi seemed to be the way.
this is the filter that ended up working (did not try 8 yet, but may give it a try later on)

storage:
  ceph:
  - find: type == scsi
    find_min: 6