direct-csi format wrong drives

[romqn1999]

Hello team, I have a machine with 2 nvme disks

$ lsblk
nvme1n1     259:0    0 465.8G  0 disk 
├─nvme1n1p1 259:2    0 402.4G  0 part /
├─nvme1n1p2 259:3    0    48M  0 part /boot/efi
└─nvme1n1p3 259:4    0  63.3G  0 part 
nvme0n1     259:1    0 465.8G  0 disk 
├─nvme0n1p1 259:7    0    50G  0 part 
├─nvme0n1p2 259:8    0   200G  0 part 
└─nvme0n1p3 259:10   0 215.8G  0 part

$ kubectl direct-csi drives ls -a
 DRIVE          CAPACITY  ALLOCATED  FILESYSTEM  VOLUMES  NODE                ACCESS-TIER  STATUS
 /dev/nvme0n11  50 GiB    50 GiB     -           -        machine-38          -            Unavailable
 /dev/nvme0n12  200 GiB   200 GiB    -           -        machine-38          -            Available
 /dev/nvme0n13  216 GiB   216 GiB    -           -        machine-38          -            Available
 /dev/nvme1n11  402 GiB   156 GiB    ext4        -        machine-38          -            Available
 /dev/nvme1n12  48 MiB    7.3 MiB    fat32       -        machine-38          -            Unavailable
 /dev/nvme1n13  63 GiB    63 GiB     -           -        machine-38          -            Available

And I run direct-csi format drives using command kubectl direct-csi drives format --nodes=machine-38 --drives='/dev/nvme0n12,/dev/nvme0n13' --force then I check the drives status

$ kubectl direct-csi drives ls -a
 DRIVE          CAPACITY  ALLOCATED  FILESYSTEM  VOLUMES  NODE                ACCESS-TIER  STATUS                                                              
 /dev/nvme0n11  50 GiB    50 GiB     -           -        machine-38          -            Unavailable                                                         
 /dev/nvme0n12  200 GiB   200 GiB    -           -        machine-38          -            Available*    failed to unmount drive ; no such file or directory   
 /dev/nvme0n13  216 GiB   153 GiB    xfs         -        machine-38          -            Ready                                                               
 /dev/nvme1n11  402 GiB   156 GiB    ext4        -        machine-38          -            Available                                                           
 /dev/nvme1n12  48 MiB    7.3 MiB    fat32       -        machine-38          -            Unavailable                                                         
 /dev/nvme1n13  63 GiB    63 GiB     -           -        machine-38          -            Available

$ lsblk
nvme1n1     259:0    0 465.8G  0 disk 
├─nvme1n1p1 259:2    0 402.4G  0 part /
├─nvme1n1p2 259:3    0    48M  0 part /boot/efi
└─nvme1n1p3 259:4    0  63.3G  0 part /var/lib/direct-csi/mnt/f322df92-f86c-473f-895a-2bd2cbdb1538
nvme0n1     259:1    0 465.8G  0 disk 
├─nvme0n1p1 259:7    0    50G  0 part 
├─nvme0n1p2 259:8    0   200G  0 part 
└─nvme0n1p3 259:10   0 215.8G  0 part

I found that I run direct-csi format for 2 partiotions nvme0n1p2 & nvme0n1p3 of disk nvme0n1 but it's formated 2 partitions nvme1n1p2 & nvme1n1p3 of disk nvme1n1. I think direct-csi formatted wrong drive.

[Praveenrajmani]

can you do kubectl direct-csi drives ls --nodes=machine-38 --drives='/dev/nvme0n12,/dev/nvme0n13' -o yaml and paste the output here?

The drive order could've changed here, can you restart the direct-csi pod running in machine-38 node once?

[romqn1999]

can you do kubectl direct-csi drives ls --nodes=machine-38 --drives='/dev/nvme0n12,/dev/nvme0n13' -o yaml and paste the output here?

I have restart the direct-csi pod running in machine-38 node

$ kubectl direct-csi drives ls --nodes=machine-38 --drives='/dev/nvme0n12,/dev/nvme0n13' -o yaml
apiVersion: direct.csi.min.io/v1beta2
items:
- apiVersion: direct.csi.min.io/v1beta2
  kind: DirectCSIDrive
  metadata:
    creationTimestamp: "2021-11-08T08:20:01Z"
    generation: 8
    labels:
      direct.csi.min.io/access-tier: Unknown
      direct.csi.min.io/created-by: directcsi-driver
      direct.csi.min.io/node: machine-38
      direct.csi.min.io/path: nvme0n-part-1-part-2
      direct.csi.min.io/version: v1beta2
    managedFields:
    - apiVersion: direct.csi.min.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            .: {}
            f:direct.csi.min.io/access-tier: {}
            f:direct.csi.min.io/created-by: {}
            f:direct.csi.min.io/node: {}
            f:direct.csi.min.io/path: {}
            f:direct.csi.min.io/version: {}
        f:spec: {}
        f:status:
          .: {}
          f:accessTier: {}
          f:allocatedCapacity: {}
          f:conditions:
            .: {}
            k:{"type":"Formatted"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Initialized"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Mounted"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Owned"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
          f:driveStatus: {}
          f:filesystemUUID: {}
          f:logicalBlockSize: {}
          f:majorNumber: {}
          f:minorNumber: {}
          f:mountOptions: {}
          f:mountpoint: {}
          f:nodeName: {}
          f:partitionNum: {}
          f:partitionUUID: {}
          f:path: {}
          f:physicalBlockSize: {}
          f:rootPartition: {}
          f:serialNumber: {}
          f:topology:
            .: {}
            f:direct.csi.min.io/identity: {}
            f:direct.csi.min.io/node: {}
            f:direct.csi.min.io/rack: {}
            f:direct.csi.min.io/region: {}
            f:direct.csi.min.io/zone: {}
          f:totalCapacity: {}
      manager: direct-csi
      operation: Update
      time: "2021-11-08T08:21:46Z"
    - apiVersion: direct.csi.min.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:directCSIOwned: {}
          f:requestedFormat:
            .: {}
            f:filesystem: {}
            f:force: {}
      manager: kubectl-direct_csi
      operation: Update
      time: "2021-11-08T08:21:46Z"
    name: fa77a3ae-174b-4201-9962-2b5efa0d329a
    resourceVersion: "2337899"
    uid: be5e356f-0b16-41d1-b658-81ac0148f5ca
  spec:
    directCSIOwned: true
    requestedFormat:
      filesystem: xfs
      force: true
  status:
    accessTier: Unknown
    allocatedCapacity: 214748364288
    conditions:
    - lastTransitionTime: "2021-11-08T08:22:55Z"
      message: failed to unmount drive fa77a3ae-174b-4201-9962-2b5efa0d329a; no such
        file or directory
      reason: Added
      status: "False"
      type: Owned
    - lastTransitionTime: "2021-11-08T08:22:55Z"
      message: Mounted
      reason: Added
      status: "True"
      type: Mounted
    - lastTransitionTime: "2021-11-08T08:22:55Z"
      message: NotFormatted
      reason: Added
      status: "False"
      type: Formatted
    - lastTransitionTime: "2021-11-08T08:20:01Z"
      message: ""
      reason: Initialized
      status: "True"
      type: Initialized
    driveStatus: Available
    filesystemUUID: 7d2e9324-adb2-4113-896b-06f9195206b1
    logicalBlockSize: 512
    majorNumber: 259
    minorNumber: 3
    mountOptions:
    - rw
    - relatime
    mountpoint: /boot/efi
    nodeName: machine-38
    partitionNum: 3
    partitionUUID: D1E47DF8-5743-2E4A-9CF7-11D347625659
    path: /var/lib/direct-csi/devices/nvme0n-part-1-part-2
    physicalBlockSize: 4096
    rootPartition: nvme0n1
    serialNumber: 'S466NX0KB99361V     '
    topology:
      direct.csi.min.io/identity: direct-csi-min-io
      direct.csi.min.io/node: machine-38
      direct.csi.min.io/rack: default
      direct.csi.min.io/region: default
      direct.csi.min.io/zone: default
    totalCapacity: 214748364288
- apiVersion: direct.csi.min.io/v1beta2
  kind: DirectCSIDrive
  metadata:
    creationTimestamp: "2021-11-08T08:20:01Z"
    finalizers:
    - direct.csi.min.io/data-protection
    generation: 3
    labels:
      direct.csi.min.io/access-tier: Unknown
      direct.csi.min.io/created-by: directcsi-driver
      direct.csi.min.io/node: machine-38
      direct.csi.min.io/path: nvme0n-part-1-part-3
      direct.csi.min.io/version: v1beta2
    managedFields:
    - apiVersion: direct.csi.min.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:directCSIOwned: {}
      manager: kubectl-direct_csi
      operation: Update
      time: "2021-11-08T08:21:46Z"
    - apiVersion: direct.csi.min.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            .: {}
            v:"direct.csi.min.io/data-protection": {}
          f:labels:
            .: {}
            f:direct.csi.min.io/access-tier: {}
            f:direct.csi.min.io/created-by: {}
            f:direct.csi.min.io/node: {}
            f:direct.csi.min.io/path: {}
            f:direct.csi.min.io/version: {}
        f:spec: {}
        f:status:
          .: {}
          f:accessTier: {}
          f:allocatedCapacity: {}
          f:conditions:
            .: {}
            k:{"type":"Formatted"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Initialized"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Mounted"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Owned"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
          f:driveStatus: {}
          f:filesystem: {}
          f:filesystemUUID: {}
          f:freeCapacity: {}
          f:logicalBlockSize: {}
          f:majorNumber: {}
          f:minorNumber: {}
          f:mountpoint: {}
          f:nodeName: {}
          f:partitionNum: {}
          f:partitionUUID: {}
          f:path: {}
          f:physicalBlockSize: {}
          f:rootPartition: {}
          f:serialNumber: {}
          f:topology:
            .: {}
            f:direct.csi.min.io/identity: {}
            f:direct.csi.min.io/node: {}
            f:direct.csi.min.io/rack: {}
            f:direct.csi.min.io/region: {}
            f:direct.csi.min.io/zone: {}
          f:totalCapacity: {}
      manager: direct-csi
      operation: Update
      time: "2021-11-08T08:21:47Z"
    name: 0c75868f-2e19-4ea2-955e-bd9cec6c0828
    resourceVersion: "2337659"
    uid: eca9e45f-7957-49c1-b8ec-c9d714f1b78b
  spec:
    directCSIOwned: true
  status:
    accessTier: Unknown
    allocatedCapacity: 164262288384
    conditions:
    - lastTransitionTime: "2021-11-08T08:21:47Z"
      message: ""
      reason: Added
      status: "True"
      type: Owned
    - lastTransitionTime: "2021-11-08T08:21:47Z"
      message: Mounted
      reason: Added
      status: "True"
      type: Mounted
    - lastTransitionTime: "2021-11-08T08:21:47Z"
      message: Formatted
      reason: Added
      status: "True"
      type: Formatted
    - lastTransitionTime: "2021-11-08T08:20:01Z"
      message: ""
      reason: Initialized
      status: "True"
      type: Initialized
    driveStatus: Ready
    filesystem: xfs
    filesystemUUID: f2834f8f-46bc-422b-9fa0-9c228f68e772
    freeCapacity: 67409051648
    logicalBlockSize: 512
    majorNumber: 259
    minorNumber: 4
    mountpoint: /var/lib/direct-csi/mnt/f2834f8f-46bc-422b-9fa0-9c228f68e772
    nodeName: machine-38
    partitionNum: 4
    partitionUUID: BB5B4848-B7F5-BD47-BD23-96D486957580
    path: /var/lib/direct-csi/devices/nvme0n-part-1-part-3
    physicalBlockSize: 4096
    rootPartition: nvme0n1
    serialNumber: 'S466NX0KB99361V     '
    topology:
      direct.csi.min.io/identity: direct-csi-min-io
      direct.csi.min.io/node: machine-38
      direct.csi.min.io/rack: default
      direct.csi.min.io/region: default
      direct.csi.min.io/zone: default
    totalCapacity: 231671340032
kind: List
metadata: {}

[Praveenrajmani]

Thanks @romqn1999

Can you also get us the outputs of

cat /sys/block/nvme0n1/nvme0n1p2/uevent
cat /sys/block/nvme0n1/nvme0n1p3/uevent

[romqn1999]

Can you also get us the outputs of

cat /sys/block/nvme0n1/nvme0n1p2/uevent
cat /sys/block/nvme0n1/nvme0n1p3/uevent

$ cat /sys/block/nvme0n1/nvme0n1p2/uevent
MAJOR=259
MINOR=6
DEVNAME=nvme0n1p2
DEVTYPE=partition
PARTN=2

$ cat /sys/block/nvme0n1/nvme0n1p3/uevent
MAJOR=259
MINOR=7
DEVNAME=nvme0n1p3
DEVTYPE=partition
PARTN=3

[Praveenrajmani]

the lsblk output does not match with the /sys/block information. Looks like, there are some active hot-swaps happening. We would need some more information about the disk setup here.

$ lsblk
nvme1n1     259:0    0 465.8G  0 disk 
├─nvme1n1p1 259:2    0 402.4G  0 part /
├─nvme1n1p2 259:3    0    48M  0 part /boot/efi
└─nvme1n1p3 259:4    0  63.3G  0 part /var/lib/direct-csi/mnt/f322df92-f86c-473f-895a-2bd2cbdb1538
nvme0n1     259:1    0 465.8G  0 disk 
├─nvme0n1p1 259:7    0    50G  0 part 
├─nvme0n1p2 259:8    0   200G  0 part 
└─nvme0n1p3 259:10   0 215.8G  0 part

[romqn1999]

What other information do we need for this matter?

[romqn1999]

$ lsblk
nvme1n1 259:0 0 465.8G 0 disk
├─nvme1n1p1 259:2 0 402.4G 0 part /
├─nvme1n1p2 259:3 0 48M 0 part /boot/efi
└─nvme1n1p3 259:4 0 63.3G 0 part /var/lib/direct-csi/mnt/f322df92-f86c-473f-895a-2bd2cbdb1538
nvme0n1 259:1 0 465.8G 0 disk
├─nvme0n1p1 259:7 0 50G 0 part
├─nvme0n1p2 259:8 0 200G 0 part
└─nvme0n1p3 259:10 0 215.8G 0 part

I have tried to partitioning disk many times today, I think the difference is because the output I passed here at difference time. For now, I think the output of lsblk and /sys/block are same.

$ lsblk
nvme1n1     259:0    0 465.8G  0 disk 
├─nvme1n1p1 259:2    0 402.4G  0 part /
├─nvme1n1p2 259:3    0    48M  0 part /boot/efi
└─nvme1n1p3 259:4    0  63.3G  0 part /var/lib/direct-csi/mnt/f2834f8f-46bc-422b-9fa0-9c228f68e772
nvme0n1     259:1    0 465.8G  0 disk 
├─nvme0n1p1 259:5    0    50G  0 part 
├─nvme0n1p2 259:6    0   200G  0 part 
└─nvme0n1p3 259:7    0 215.8G  0 part 

Sorry if I bother you.

[Praveenrajmani]

@romqn1999 Dynamic discovery is a feature that is currently been worked on, this will detect any dynamic order-changes, hot-swaps, addition and deletion of drives. This will be available in the next release.

For now, you can restart the direct-csi pod running on that node to sync the drives on order changes (Assuming direct-csi version 1.4.3 is being used here).

[romqn1999]

I'm using direct-csi version 1.4.3 and I have restart the direct-csi pod running on that node every time I make any change on drives.

I have partitioning the drive many times to confirm that the problem format wrong drive is reproducible on my machine.

[Praveenrajmani]

Can you try to reproduce in a fresh setup just to clarify few things here

kubectl direct-csi uninstall --crd --force (not recommended when there are PVs in use)
kubeclt direct-csi install

And then format the drives with necessary selectors by,

kubectl direct-csi format --drives XXX --nodes XXX --force

Then list the drives to see if it is formatted correctly, Else, paste us the following information

kubectl direct-csi drives ls --drives XXX --nodes XXX -o yaml

and also the uevent info for these drives in /sys/block (as asked before)

direct-csi would probe the device information from /sys/block. So, we need to check if there is any inconsistency here.

[Praveenrajmani]

@romqn1999 ^^

[romqn1999]

kubectl direct-csi uninstall --crd --force (not recommended when there are PVs in use)
kubeclt direct-csi install

I just uninstall and reinstall direct-csi

$ kubectl direct-csi uninstall --crd --force
I1109 03:12:34.980896 3711457 request.go:668] Waited for 1.006863492s due to client-side throttling, not priority and fairness, request: DELETE:https://192.168.1.19:6443/apis/direct.csi.min.io/v1beta2/directcsidrives/fa77a3ae-174b-4201-9962-2b5efa0d329a
I1109 03:12:34.989962 3711457 uninstall.go:168] 'direct.csi.min.io' CRD resources deleted
I1109 03:12:35.010229 3711457 uninstall.go:174] 'direct.csi.min.io' crds deleted
I1109 03:12:35.042718 3711457 uninstall.go:180] 'direct.csi.min.io' csidriver deleted
I1109 03:12:38.118668 3711457 uninstall.go:185] 'direct.csi.min.io' storageclass deleted
I1109 03:12:38.131980 3711457 uninstall.go:190] 'direct.csi.min.io' service deleted
I1109 03:12:38.144663 3711457 uninstall.go:195] 'direct.csi.min.io' rbac roles deleted
I1109 03:12:38.148426 3711457 uninstall.go:200] 'direct.csi.min.io' daemonset deleted
I1109 03:12:38.155141 3711457 uninstall.go:209] 'direct.csi.min.io' drive validation rules removed
I1109 03:12:38.182323 3711457 uninstall.go:214] 'direct.csi.min.io' controller deployment deleted
I1109 03:12:38.208336 3711457 uninstall.go:228] 'direct.csi.min.io' conversion deployment deleted
I1109 03:12:38.212281 3711457 uninstall.go:233] 'direct.csi.min.io' namespace deleted

$ kubectl direct-csi install
I1109 03:14:14.515145 3712743 install.go:99] 'direct.csi.min.io' namespace created
W1109 03:14:14.531821 3712743 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
I1109 03:14:14.544175 3712743 install.go:119] 'direct.csi.min.io' rbac roles created
I1109 03:14:14.591311 3712743 install.go:137] crds successfully registered
I1109 03:14:19.141895 3712743 install.go:146] 'direct.csi.min.io' conversion deployment created
I1109 03:14:22.187069 3712743 register.go:124] successfully updated conversion webhook settings in 'directcsidrives.direct.csi.min.io' CRD
I1109 03:14:22.212009 3712743 register.go:124] successfully updated conversion webhook settings in 'directcsivolumes.direct.csi.min.io' CRD
I1109 03:14:22.229708 3712743 install.go:159] 'direct.csi.min.io' csidriver created
I1109 03:14:23.262009 3712743 request.go:668] Waited for 1.029516975s due to client-side throttling, not priority and fairness, request: GET:https://192.168.1.19:6443/apis/policy/v1beta1?timeout=32s
I1109 03:14:23.722300 3712743 install.go:168] 'direct.csi.min.io' storageclass created
I1109 03:14:23.732528 3712743 install.go:177] 'direct.csi.min.io' service created
I1109 03:14:23.750062 3712743 install.go:186] 'direct.csi.min.io' daemonset created
I1109 03:14:30.422744 3712743 install.go:195] 'direct.csi.min.io' deployment created

Then I format the drives:

$ kubectl direct-csi drives format --nodes=machine-38 --drives='/dev/nvme0n12,/dev/nvme0n13' --force

And the yaml output of list drives are:

$ kubectl direct-csi drives ls --nodes=machine-38 --drives='/dev/nvme0n12,/dev/nvme0n13' -o yaml    
apiVersion: direct.csi.min.io/v1beta2
items:
- apiVersion: direct.csi.min.io/v1beta2
  kind: DirectCSIDrive
  metadata:
    creationTimestamp: "2021-11-09T03:14:26Z"
    generation: 9
    labels:
      direct.csi.min.io/access-tier: Unknown
      direct.csi.min.io/created-by: directcsi-driver
      direct.csi.min.io/node: machine-38
      direct.csi.min.io/path: nvme0n-part-1-part-2
      direct.csi.min.io/version: v1beta2
    managedFields:
    - apiVersion: direct.csi.min.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            .: {}
            f:direct.csi.min.io/access-tier: {}
            f:direct.csi.min.io/created-by: {}
            f:direct.csi.min.io/node: {}
            f:direct.csi.min.io/path: {}
            f:direct.csi.min.io/version: {}
        f:spec: {}
        f:status:
          .: {}
          f:accessTier: {}
          f:allocatedCapacity: {}
          f:conditions:
            .: {}
            k:{"type":"Formatted"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Initialized"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Mounted"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Owned"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
          f:driveStatus: {}
          f:filesystemUUID: {}
          f:logicalBlockSize: {}
          f:majorNumber: {}
          f:minorNumber: {}
          f:mountOptions: {}
          f:mountpoint: {}
          f:nodeName: {}
          f:partitionNum: {}
          f:partitionUUID: {}
          f:path: {}
          f:physicalBlockSize: {}
          f:rootPartition: {}
          f:serialNumber: {}
          f:topology:
            .: {}
            f:direct.csi.min.io/identity: {}
            f:direct.csi.min.io/node: {}
            f:direct.csi.min.io/rack: {}
            f:direct.csi.min.io/region: {}
            f:direct.csi.min.io/zone: {}
          f:totalCapacity: {}
      manager: direct-csi
      operation: Update
      time: "2021-11-09T03:19:25Z"
    - apiVersion: direct.csi.min.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:directCSIOwned: {}
          f:requestedFormat:
            .: {}
            f:filesystem: {}
            f:force: {}
      manager: kubectl-direct_csi
      operation: Update
      time: "2021-11-09T03:19:25Z"
    name: 0729bf63-bb27-4889-b05a-2f930dc2465b
    resourceVersion: "2578195"
    uid: aab9c649-7c07-4c0b-8e53-57fee2069195
  spec:
    directCSIOwned: true
    requestedFormat:
      filesystem: xfs
      force: true
  status:
    accessTier: Unknown
    allocatedCapacity: 214748364288
    conditions:
    - lastTransitionTime: "2021-11-09T03:23:59Z"
      message: failed to unmount drive 0729bf63-bb27-4889-b05a-2f930dc2465b; no such
        file or directory
      reason: Added
      status: "False"
      type: Owned
    - lastTransitionTime: "2021-11-09T03:23:59Z"
      message: Mounted
      reason: Added
      status: "True"
      type: Mounted
    - lastTransitionTime: "2021-11-09T03:23:59Z"
      message: NotFormatted
      reason: Added
      status: "False"
      type: Formatted
    - lastTransitionTime: "2021-11-09T03:14:25Z"
      message: ""
      reason: Initialized
      status: "True"
      type: Initialized
    driveStatus: Available
    filesystemUUID: 9a09dd8c-8c10-4d3c-bd8e-d5fac143e441
    logicalBlockSize: 512
    majorNumber: 259
    minorNumber: 3
    mountOptions:
    - rw
    - relatime
    mountpoint: /boot/efi
    nodeName: machine-38
    partitionNum: 3
    partitionUUID: D1E47DF8-5743-2E4A-9CF7-11D347625659
    path: /var/lib/direct-csi/devices/nvme0n-part-1-part-2
    physicalBlockSize: 4096
    rootPartition: nvme0n1
    serialNumber: 'S466NX0KB99361V     '
    topology:
      direct.csi.min.io/identity: direct-csi-min-io
      direct.csi.min.io/node: machine-38
      direct.csi.min.io/rack: default
      direct.csi.min.io/region: default
      direct.csi.min.io/zone: default
    totalCapacity: 214748364288
- apiVersion: direct.csi.min.io/v1beta2
  kind: DirectCSIDrive
  metadata:
    creationTimestamp: "2021-11-09T03:14:26Z"
    finalizers:
    - direct.csi.min.io/data-protection
    generation: 3
    labels:
      direct.csi.min.io/access-tier: Unknown
      direct.csi.min.io/created-by: directcsi-driver
      direct.csi.min.io/node: machine-38
      direct.csi.min.io/path: nvme0n-part-1-part-3
      direct.csi.min.io/version: v1beta2
    managedFields:
    - apiVersion: direct.csi.min.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            .: {}
            v:"direct.csi.min.io/data-protection": {}
          f:labels:
            .: {}
            f:direct.csi.min.io/access-tier: {}
            f:direct.csi.min.io/created-by: {}
            f:direct.csi.min.io/node: {}
            f:direct.csi.min.io/path: {}
            f:direct.csi.min.io/version: {}
        f:spec: {}
        f:status:
          .: {}
          f:accessTier: {}
          f:allocatedCapacity: {}
          f:conditions:
            .: {}
            k:{"type":"Formatted"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Initialized"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Mounted"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Owned"}:
              .: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
          f:driveStatus: {}
          f:filesystem: {}
          f:filesystemUUID: {}
          f:freeCapacity: {}
          f:logicalBlockSize: {}
          f:majorNumber: {}
          f:minorNumber: {}
          f:mountpoint: {}
          f:nodeName: {}
          f:partitionNum: {}
          f:partitionUUID: {}
          f:path: {}
          f:physicalBlockSize: {}
          f:rootPartition: {}
          f:serialNumber: {}
          f:topology:
            .: {}
            f:direct.csi.min.io/identity: {}
            f:direct.csi.min.io/node: {}
            f:direct.csi.min.io/rack: {}
            f:direct.csi.min.io/region: {}
            f:direct.csi.min.io/zone: {}
          f:totalCapacity: {}
      manager: direct-csi
      operation: Update
      time: "2021-11-09T03:19:25Z"
    - apiVersion: direct.csi.min.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:directCSIOwned: {}
      manager: kubectl-direct_csi
      operation: Update
      time: "2021-11-09T03:19:25Z"
    name: 5a91d07d-1f97-4395-960d-5cbec61a8f6e
    resourceVersion: "2577230"
    uid: ad7de0db-8c97-4f07-97c2-beb90a0ffcae
  spec:
    directCSIOwned: true
  status:
    accessTier: Unknown
    allocatedCapacity: 164262288384
    conditions:
    - lastTransitionTime: "2021-11-09T03:19:25Z"
      message: ""
      reason: Added
      status: "True"
      type: Owned
    - lastTransitionTime: "2021-11-09T03:19:25Z"
      message: Mounted
      reason: Added
      status: "True"
      type: Mounted
    - lastTransitionTime: "2021-11-09T03:19:25Z"
      message: Formatted
      reason: Added
      status: "True"
      type: Formatted
    - lastTransitionTime: "2021-11-09T03:14:25Z"
      message: ""
      reason: Initialized
      status: "True"
      type: Initialized
    driveStatus: Ready
    filesystem: xfs
    filesystemUUID: 1ff3559c-d812-4759-a79d-79e2c6bbca72
    freeCapacity: 67409051648
    logicalBlockSize: 512
    majorNumber: 259
    minorNumber: 4
    mountpoint: /var/lib/direct-csi/mnt/1ff3559c-d812-4759-a79d-79e2c6bbca72
    nodeName: machine-38
    partitionNum: 4
    partitionUUID: BB5B4848-B7F5-BD47-BD23-96D486957580
    path: /var/lib/direct-csi/devices/nvme0n-part-1-part-3
    physicalBlockSize: 4096
    rootPartition: nvme0n1
    serialNumber: 'S466NX0KB99361V     '
    topology:
      direct.csi.min.io/identity: direct-csi-min-io
      direct.csi.min.io/node: machine-38
      direct.csi.min.io/rack: default
      direct.csi.min.io/region: default
      direct.csi.min.io/zone: default
    totalCapacity: 231671340032
kind: List
metadata: {}

and also the uevent info for these drives in /sys/block are:

$ cat /sys/block/nvme0n1/nvme0n1p2/uevent
MAJOR=259
MINOR=6
DEVNAME=nvme0n1p2
DEVTYPE=partition
PARTN=2

$ cat /sys/block/nvme0n1/nvme0n1p3/uevent
MAJOR=259
MINOR=7
DEVNAME=nvme0n1p3
DEVTYPE=partition
PARTN=3

and output of lsblk:

nvme1n1     259:0    0 465.8G  0 disk 
├─nvme1n1p1 259:2    0 402.4G  0 part /
├─nvme1n1p2 259:3    0    48M  0 part /boot/efi
└─nvme1n1p3 259:4    0  63.3G  0 part /var/lib/direct-csi/mnt/1ff3559c-d812-4759-a79d-79e2c6bbca72
nvme0n1     259:1    0 465.8G  0 disk 
├─nvme0n1p1 259:5    0    50G  0 part 
├─nvme0n1p2 259:6    0   200G  0 part 
└─nvme0n1p3 259:7    0 215.8G  0 part 

[Praveenrajmani]

@romqn1999 just to re-confirm, there weren't any drive order changes during the run? ie, i suppose the drives weren't re-partitioned during this test?

[romqn1999]

@romqn1999 just to re-confirm, there weren't any drive order changes during the run? ie, i suppose the drives weren't re-partitioned during this test?

Yes, I did nothing to the drives. There is no other command in above run.

[wlan0]

@romqn1999 Thanks for reporting this issue. We are working on a patch right away. We will update on this issue as soon as the patch is released.