shared db/wal device

[Ornias1993]

It's not completely clear from the manual, but can a DB/WAL device be shared automatically by giving multiple OSDs the same device to use for DB/WAL?

[travisn]

Yes, if you add this config, all new OSDs on a node will be created with the same db/wal device. Existing OSDs cannot be updated to add this later, nor can additional OSDs be added at a later time with the same device.

config:
      metadataDevice: <deviceName>

[Ornias1993]

nor can additional OSDs be added at a later time with the same device.

How does that work?
On initial incusion I can add multiple OSDs but one cannot be added later?
DOes it detect how many OSDs are on a node on init or something?

[travisn]

On initial incusion I can add multiple OSDs but one cannot be added later?

Correct, more OSDs cannot be added later on that node. If you want more flexibility, it's recommended to use the cluster-on-pvc mode, which would only have a single metadata device per OSD.

Does it detect how many OSDs are on a node on init or something?

Correct, it will find all available devices on the node, assuming you are using useAllNodes: true and useAllDevices: true. There are options to filter down the list or provide a specific list of devices if desired.

[Ornias1993]

I was indeed looking at cluster-on-pvc mode, are there any negative performance implications, assuming a simple low-overhead Local PV provisioner?

[travisn]

Performance during creation of the OSDs will be negligible, what is more important for storage performance is the underlying devices.

[Ornias1993]

Thanks! :)

[Ornias1993]

Assuming I set useAllNodes: true and useAllDevices: true and define PVC metadata claims.
How do I then assign those metadata claims to actual blockdevices?

[rlanore]

Hi, i'm surprised of this limitations. We use this metadataDevice per host and on previous week we lost a drive. A team member has process to delete the failing OSD. Now we received the new device and operator says:

2024-04-22 12:46:29.525796 I | cephosd: configuring new LVM device sdj
2024-04-22 12:46:29.537456 C | rookcmd: failed to configure devices: failed to initialize osd: metadata device /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 is not found

We deleted the metadata lvm volume by hand.

What is your advice in this case @travisn

[travisn]

You are trying to replace just a single OSD for the shared metadata device? This is issue #13240 that we hope to fix soon.

[rlanore]

Thank you for mentioning this issue. I will update on it