Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to find the FC disks starts with fc prefix #95450

Closed
mkumatag opened this issue Oct 9, 2020 · 2 comments · Fixed by #95451
Closed

failed to find the FC disks starts with fc prefix #95450

mkumatag opened this issue Oct 9, 2020 · 2 comments · Fixed by #95451
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/storage Categorizes an issue or PR as relevant to SIG Storage. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@mkumatag
Copy link
Member

mkumatag commented Oct 9, 2020

What happened:

Currently, we have logic here to look for the disk names with -fc but this fails when we have devices which are starting with fc

fcPath := "-fc-0x" + wwn + "-lun-" + lun

e.g: on ppc64le architecture, NPIV based fc disks are discovered as follows:

[root@dhcp-test-1 kubernetes]# ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root  9 Oct  9 08:32 fc-0x5005076810213b32-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root 10 Oct  9 08:32 fc-0x5005076810213b32-lun-0-part1 -> ../../sdf1
lrwxrwxrwx. 1 root root 10 Oct  9 08:32 fc-0x5005076810213b32-lun-0-part2 -> ../../sdf2
lrwxrwxrwx. 1 root root  9 Oct  9 16:47 fc-0x5005076810213b32-lun-1 -> ../../sds
lrwxrwxrwx. 1 root root  9 Oct  9 16:35 fc-0x5005076810213b32-lun-2 -> ../../sdt
lrwxrwxrwx. 1 root root  9 Oct  9 14:37 fc-0x5005076810213b4f-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root 10 Oct  9 08:32 fc-0x5005076810213b4f-lun-0-part1 -> ../../sde1
lrwxrwxrwx. 1 root root 10 Oct  9 08:32 fc-0x5005076810213b4f-lun-0-part2 -> ../../sde2
lrwxrwxrwx. 1 root root  9 Oct  9 16:35 fc-0x5005076810213b4f-lun-1 -> ../../sdq
lrwxrwxrwx. 1 root root  9 Oct  9 16:35 fc-0x5005076810213b4f-lun-2 -> ../../sdr
lrwxrwxrwx. 1 root root  9 Oct  9 08:32 fc-0x5005076810223b32-lun-0 -> ../../sdd
lrwxrwxrwx. 1 root root 10 Oct  9 08:32 fc-0x5005076810223b32-lun-0-part1 -> ../../sdd1
lrwxrwxrwx. 1 root root 10 Oct  9 08:32 fc-0x5005076810223b32-lun-0-part2 -> ../../sdd2
lrwxrwxrwx. 1 root root  9 Oct  9 16:35 fc-0x5005076810223b32-lun-1 -> ../../sdo
lrwxrwxrwx. 1 root root  9 Oct  9 16:35 fc-0x5005076810223b32-lun-2 -> ../../sdp

and here is the udev information for one of the disk:

[root@dhcp-test-1 kubernetes]# udevadm info -q all -n /dev/sds
P: /devices/vio/30000008/host2/rport-2:0-1/target2:0:1/2:0:1:1/block/sds
N: sds
S: disk/by-id/scsi-360050768108181da7800000000001b07
S: disk/by-id/scsi-SIBM_2145_02042060769eXX00
S: disk/by-id/wwn-0x60050768108181da7800000000001b07
S: disk/by-path/fc-0x5005076810213b32-lun-1
S: disk/by-path/fc-0xc050760b725f0568-0x5005076810213b32-lun-1
S: disk/by-uuid/314f3dcf-1b83-4571-9371-73d538dcaeb8
E: DEVLINKS=/dev/disk/by-id/wwn-0x60050768108181da7800000000001b07 /dev/disk/by-uuid/314f3dcf-1b83-4571-9371-73d538dcaeb8 /dev/disk/by-id/scsi-SIBM_2145_02042060769eXX00 /dev/disk/by-id/scsi-360050768108181da7800000000001b07 /dev/disk/by-path/fc-0xc050760b725f0568-0x5005076810213b32-lun-1 /dev/disk/by-path/fc-0x5005076810213b32-lun-1
E: DEVNAME=/dev/sds
E: DEVPATH=/devices/vio/30000008/host2/rport-2:0-1/target2:0:1/2:0:1:1/block/sds
E: DEVTYPE=disk
E: FC_INITIATOR_WWPN=0xc050760b725f0568
E: FC_TARGET_LUN=1
E: FC_TARGET_WWPN=0x5005076810213b32
E: ID_BUS=scsi
E: ID_FS_TYPE=ext4
E: ID_FS_USAGE=filesystem
E: ID_FS_UUID=314f3dcf-1b83-4571-9371-73d538dcaeb8
E: ID_FS_UUID_ENC=314f3dcf-1b83-4571-9371-73d538dcaeb8
E: ID_FS_VERSION=1.0
E: ID_MODEL=2145
E: ID_MODEL_ENC=2145\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_PATH=fc-0x5005076810213b32-lun-1
E: ID_PATH_TAG=fc-0x5005076810213b32-lun-1
E: ID_REVISION=0000
E: ID_SCSI=1
E: ID_SCSI_INQUIRY=1
E: ID_SCSI_SERIAL=02042060769eXX00
E: ID_SERIAL=360050768108181da7800000000001b07
E: ID_SERIAL_SHORT=60050768108181da7800000000001b07
E: ID_TARGET_PORT=0
E: ID_TYPE=disk
E: ID_VENDOR=IBM
E: ID_VENDOR_ENC=IBM\x20\x20\x20\x20\x20
E: ID_WWN=0x60050768108181da
E: ID_WWN_VENDOR_EXTENSION=0x7800000000001b07
E: ID_WWN_WITH_EXTENSION=0x60050768108181da7800000000001b07
E: MAJOR=65
E: MINOR=32
E: SCSI_IDENT_LUN_NAA_REGEXT=60050768108181da7800000000001b07
E: SCSI_IDENT_PORT_RELATIVE=2180
E: SCSI_IDENT_PORT_TARGET_PORT_GROUP=0x0
E: SCSI_IDENT_SERIAL=02042060769eXX00
E: SCSI_MODEL=2145
E: SCSI_MODEL_ENC=2145\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: SCSI_REVISION=0000
E: SCSI_TPGS=1
E: SCSI_TYPE=disk
E: SCSI_VENDOR=IBM
E: SCSI_VENDOR_ENC=IBM\x20\x20\x20\x20\x20
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=29384079103

[root@dhcp-test-1 kubernetes]#

Because of the existing logic k8s fails to detect the disk because it does not start with -fc-0x instead starts with fc-0x, so pod creation fails with the following error:

Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Normal   Scheduled               36m                default-scheduler        Successfully assigned default/mkumatag-fc-pod to dhcp-test-1
  Normal   SuccessfulAttachVolume  36m                attachdetach-controller  AttachVolume.Attach succeeded for volume "fc-vol"
  Warning  FailedMount             33m                kubelet                  Unable to attach or mount volumes: unmounted volumes=[fc-vol], unattached volumes=[default-token-k6pl4 fc-vol]: timed out waiting for the condition
  Warning  FailedMount             33m (x5 over 35m)  kubelet                  MountVolume.WaitForAttach failed for volume "fc-vol" : no fc disk found

What you expected to happen:
FC disk should get attached properly to the pod

How to reproduce it (as minimally and precisely as possible):
sample pod yaml on a machine with path names starts with fc

apiVersion: v1
kind: Pod
metadata:
 name: mkumatag-fc-pod
spec:
 containers:
   - image: ubuntu:latest
     command: ["/bin/sh", "-ec", "while :; do echo '.'; sleep 5 ; done"]
     name: fc
     volumeMounts:
       - name: fc-vol
         mountPath: /mnt/fc
 volumes:
   - name: fc-vol
     fc:
       targetWWNs:
         - 5005076810213b32
       lun: 1
       fsType: ext4

Anything else we need to know?:

Environment:
N/A

@mkumatag mkumatag added the kind/bug Categorizes issue or PR as related to a bug. label Oct 9, 2020
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 9, 2020
@mkumatag
Copy link
Member Author

mkumatag commented Oct 9, 2020

/sig storage

@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 9, 2020
@mkumatag mkumatag mentioned this issue Oct 9, 2020
@mkumatag
Copy link
Member Author

mkumatag commented Oct 9, 2020

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/storage Categorizes an issue or PR as relevant to SIG Storage. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants