osd: handle device name change and device removal correctly

[satoru-takeuchi]

Description of your changes:

If a kernel device name change happens and a block device file in the OSD directory becomes dangling link, this OSD fails to start continuously. This problem can be resolved by confirming the validity of the device file and recreating it if necessary.

Which issue is resolved by this Pull Request:
Resolves #10860

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide).
• Skip Tests for Docs: If this is only a documentation change, add the label skip-ci on the PR.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[satoru-takeuchi]

This PR will be ready to review when I finish verifying this PR covers the following matrix.

• device name change and device removal
• device is specified as kernel device name and symlink such as udev persistent device

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

[satoru-takeuchi]

I finished testing. So this PR is ready to review.

test detail

• There are two extra disks B(sdb) and C(sdc).
• test matrix:
  • device name
    • kernel name(i.e. sdb, sdc, and so on)
    • udev persistent name(i.e. /dev/disks/by-id/XXX)
  • the type of device name change
    • device becomes missing: remove disk B. Then disk C becomes sdb
    • flip device names: exchange the slot of disk B and C. Then disk C becomes sdb and disk B becomes sdc

environment

• HW
  • A Hyper-V guests
  • two extra disks
    • device B: inserted to slot2: sdb in guest
    • device C: inserted to slot3: sdc in guest
• SW
  • ubuntu 20.04
  • k8s: 1.25.6
  • rook: master(d7f84c9) + my PR
  • ceph: v17.2.5

device name: kernel name

device becomes missing

1. shutdown the VM
2. remove device B as follows and restart the VM

• slot2: missing
• slot3: device C: sdb in guest

4. verify OSD on device C is running

=> OK

flip device names

1. shutdown the VM
2. exchange the slot of device B and C as follows and restart the VM.

• slot2: device C: sdb in guest
• slot3: device B: sdc in guest

3. verify OSD on device C is running

=> OK

NOTE
When restarting, the new OSD is created on top of new /dev/sdc points to scratch1.img.
However, it's by design and this behavior can be avoided by specifying the udev persistent name.

device name: persistent device name (/dev/disk/by-id/wwn-0x60022480e20a2701d91006b946030922)

device becomes missing

1. shutdown the VM
2. remove device B as follows and restart the VM

• slot2: missing
• slot3: device C: sdb in guest

3. verify OSD on device C is running

=> OK

flip device names

1. shutdown
2. exchange the slot of device B and C as follows.

• slot2: scratch0.img: sdb in guest
• slot3: scratch1.img: sdc in guest

3. Then sdc points to device B and vice versa.
4. verify OSD is running

=> OK

[satoru-takeuchi]

The original problem happened in host-based clusters. A similar problem exists in PVC-based clusters. In this case, if a PV, corresponds to an existing OSD, and points to a missing block device file, the OSD pod fails to consume this PV. Although this behavior is undesirable, I don't think this problem should be handled in Rook. If doing so, we must re-create the existing PV. Rook shouldn't do such work.

[satoru-takeuchi]

multi-cluster-mirroring test fails consistently as described in #11742

[travisn]

The re-creating of the path is done by ceph-volume, correct?

[satoru-takeuchi]

It's correct.