[BUG] disks are visible in cli but not in UI in Harvester 1.4.0

[juergen-koller]

Describe the bug
Disks installed in Harvester host visible (and accessible) in the cli as sda-sdc, but not visible in the UI

To Reproduce

1. Install disks on Host
2. Check if they ar visible in cli -> ok
3. not shown in UI
4. wipefs the disk, mkfs.ext4 the disk, reboot host --> nothing changes

Expected behavior
disks visible in UI under Host->Storage->Add Disk

Environment

• Harvester ISO version: 1.4.0
• Underlying Infrastructure:
  Baremetal with Dell PowerEdge R760
  3 NVMe Disks

Additional context
in the log files i just find:
level=warning msg="Skip adding duplicated WWN device unknown, device path: /dev/sda"
level=warning msg="Skip adding duplicated WWN device unknown, device path: /dev/sdb"
level=warning msg="Skip adding duplicated WWN device unknown, device path: /dev/sdc"

[bk201]

@juergen-koller as the message suggests, the disks share the same wwn. What are these disks? you mention they are NVME, but the device names don't look like so cc @Vicente-Cheng @tserong

[juergen-koller]

@bk201 This NVME-Disks are Dell U2-NVMes in shelfs. The controller is a PERC H755N. In an other system all disks are normal visible to the os and the UI. I have unplugged all but one NVMes -> no changes.
As mentions all disks are normal accessible through the os/cli. but not visible in the UI.
Yes I know normally NVMes are /dev/nvme[XXXXX] but not in this Dell R760.
Changing slots on the Dell does not change anything.

[Vicente-Cheng]

@bk201 This NVME-Disks are Dell U2-NVMes in shelfs. The controller is a PERC H755N. In an other system all disks are normal visible to the os and the UI. I have unplugged all but one NVMes -> no changes. As mentions all disks are normal accessible through the os/cli. but not visible in the UI. Yes I know normally NVMes are /dev/nvme[XXXXX] but not in this Dell R760. Changing slots on the Dell does not change anything.

It seems the PERC H755N is the controller with the RAID capability.
Could you check the configuration on this node? Did it only passthrough the NVMEs?
Or did any raid be configured?

So, on other nodes, those U.2 NVMEs were also present as /dev/sdx?
Thanks!

[juergen-koller]

@Vicente-Cheng yes this controller has RAID capability. All disks are in the NON-RAID mode and as single disks visible to the system (twice checked and done in the BIOS). On an other node (with PERC 755 (NO N!!)) with 6 NVMe disks the first 3 are visible as sdX and the last 6 as nvmX. The system also is a Dell R760 can handle 8 disks the first 3 and the last 3 slots are equiped.
On the all nodes there was previously Harvester 1.3.1 installed. in this configuration all disks were visible in the UI. The installation with version 1.2. had a similar problem which could be "fixed" by placing the disk in an other slot. This did not help in this case.

[juergen-koller]

@Vicente-Cheng I have look a little bit further. Harvester OS did not create a WWN for this NVMes no the 755 on the 755N there are a WWNs. The "normal" NVMes are having no wwn link in /dev/disk/by-id. Also there is no visible WWN using lsblk -o WWN /dev/sdX

[juergen-koller]

@Vicente-Cheng @bk201 @tserong have you any hint I can look at? I am a litte desperate that I can see (and use) the disks on OS-level but I cant see them in the GUI (and therefore I cannot us them for data storage)
Thank you very much

[tserong]

Harvester requires disks to either have a WWN, or a filesystem created on them as a workaround in case they don't have a WWN. I have a hunch though... If you've got several disks with no WWN, and the WWN is thus stored as "unknown" in the blockdevice CR, the code that checks for duplicate WWNs may activate which would probably cause those other disks to be skipped even if they've got a filesystem on them.

@juergen-koller can you please provide the output of kubectl -n longhorn-system get bds -o yaml and also udevadm info for each device (e.g. udevadm info /dev/sda ; udevadm info /dev/sdb etc.)?

[juergen-koller]

@tserong

kubectl:
bds.txt

the udevadm's for iharvester01 (all disks are identical NVMEs):
sdX.txt

I tried to wipefs the NVMs and to create en ext4 filesystem on them. But all disks will not be seen in the GUI. I think I made some thing wrong.

Thank you very much

Jürgen

[tserong]

I don't think it's you doing anything wrong @juergen-koller, it's a problem with the way Harvester deals with disks that don't have a WWN. I'm sorry to say I don't have an immediate solution for you, unless there's some way you can configure that RAID controller to present WWNs for each attached NVMe.

After reviewing the outputs you provided, I tried a test here in a virtual environment, with three additional SCSI disks with no WWNs. Initially, node-disk-manager refused to add them because:

time="2025-01-14T08:14:05Z" level=info msg="Skip adding non-identifiable block device /dev/sda"
time="2025-01-14T08:14:05Z" level=info msg="Skip adding non-identifiable block device /dev/sdb"
time="2025-01-14T08:14:05Z" level=info msg="Skip adding non-identifiable block device /dev/sdc"

This is because on a completely blank disk with no WWN, GenerateDiskGUID() can't find anything to make an ID from - there's no WWN and no filesystem UUID:

https://github.com/harvester/node-disk-manager/blob/dad8d98b8f90a90583902fa806d2482cf09f847d/pkg/block/block_device.go#L555C6-L569

If we try to work around that, by creating a filesystem on each disk, GenerateDiskGUID() will subsequently succeed (because there is now a filesystem UUID), but then later we run into the duplicate WWN checker:

time="2025-01-14T08:19:55Z" level=warning msg="Skip adding duplicated WWN device unknown, device path: /dev/sda"
time="2025-01-14T08:19:55Z" level=warning msg="Skip adding duplicated WWN device unknown, device path: /dev/sdb"
time="2025-01-14T08:19:55Z" level=warning msg="Skip adding duplicated WWN device unknown, device path: /dev/sdc"

I suspect we may want to change the duplicate WWN checking code so that it allows devices whose WWN is "unknown", i.e. doesn't take that as a duplicate, but will just let the device through. I'm not sure if that will have any other negative effects though. WDYT @Vicente-Cheng ?

(We really need to come up with a better way to handle disks without WWNs... The mkfs workaround is no good for longhorn V2 volumes and probably not for LVM either, because in both those cases we don't want a filesystem and if there is one it will be wiped during provisioning, which messes up the UUID. See also #6034 (comment) and #6261.)