[BUG]: PowerStore CSI driver 2.5 create volumes successfully but unable to map volumes to hosts on PowerStore

[LeoHu1985]

Bug Description

I have encountered an issue with PowerStore CSI driver 2.5,
CSI driver installed successfully & K8s hosts registered on PowerStore via iSCSI during the process,
however when deploying pvc & application, I can see volumes created automatically & successfully on PowerStore by CSI driver as expected, however those volumes were NOT mapped to k8s hosts on PowerStore (CSI driver for some reason didn't perform this step), as a results, apparently all pods are stuck in container creating status due to unable to mount volumes, although kubectl get pvcs shows pvcs have been created successfully.

I have tried re-install CSI driver multiple times, still no go.
I've deployed PowerStore CSI driver many times previously, all worked without issue, this is the first time encounter this issue.

Not sure if this is a bug introduced in the latest 2.5 driver. Any advice would be appreciated!!

My versions are: PowerStore CSI Driver version 2.5, PowerStore version 2.1.1.0 Ubuntu 22.04.1 K8S version 1.24 (edited)

Logs

service@k8s-00-a-1:~/csi-lab/csi-powerstore$ kubectl get all -n yb-demo
NAME READY STATUS RESTARTS AGE
pod/yb-master-0 0/2 ContainerCreating 0 60m
pod/yb-master-1 0/2 ContainerCreating 0 60m
pod/yb-master-2 0/2 ContainerCreating 0 60m
pod/yb-tserver-0 0/2 ContainerCreating 0 60m
pod/yb-tserver-1 0/2 ContainerCreating 0 60m
pod/yb-tserver-2 0/2 ContainerCreating 0 60m

service@k8s-00-a-1:~/csi-lab/csi-powerstore$ kubectl get pvc -n yb-demo
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir0-yb-master-0 Bound csivol-932530fb43 10Gi RWO powerstore-local 70m
datadir0-yb-master-1 Bound csivol-58fa9f9a21 10Gi RWO powerstore-local 70m
datadir0-yb-master-2 Bound csivol-ed5ac3e2ce 10Gi RWO powerstore-local 70m
datadir0-yb-tserver-0 Bound csivol-d2045acf5f 10Gi RWO powerstore-local 70m
datadir0-yb-tserver-1 Bound csivol-faefa302bf 10Gi RWO powerstore-local 70m
datadir0-yb-tserver-2 Bound csivol-57f71da69e 10Gi RWO powerstore-local 70m

service@k8s-00-a-1:~/csi-lab/csi-powerstore$ kubectl describe pod -n yb-demo
Type Reason Age From Message

---

Warning FailedMount 7m12s (x4 over 50m) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir0], unattached volumes=[kube-api-access-mcd7b datadir0]: timed out waiting for the condition
Warning FailedAttachVolume 72s (x43 over 72m) attachdetach-controller AttachVolume.Attach failed for volume "csivol-57f71da69e" : rpc error: code = Internal desc = failure checking volume status for volume publishing: Unknown property volume.logical_used requested.
Warning FailedMount 22s (x28 over 70m) kubelet Unable to attach or mount volumes: unmounted volumes=

csi-pod1.log.txt
csi-pod2.log
csi-pod3.log

Screenshots

Additional Environment Information

No response

Steps to Reproduce

I have an environment that can reproduce this issue at any time, this environment can be accessed through zoom screen share.

Expected Behavior

Volumes created by CSI driver on PowerStore should be automatically mapped to related hosts on PowerStore by CSI driver as well

CSM Driver(s)

CSI PowerStore 2.5

Installation Type

Helm 3

Container Storage Modules Enabled

Replication enabled

Container Orchestrator

k8s 1.24

Operating System

Ubuntu 22.04

[csmbot]

@LeoHu1985: Thank you for submitting this issue!

The issue is currently awaiting triage. Please make sure you have given us as much context as possible.

If the maintainers determine this is a relevant issue, they will remove the needs-triage label and assign an appropriate priority label.

---

We want your feedback! If you have any questions or suggestions regarding our contributing process/workflow, please reach out to us at karavi@dell.com.

[LeoHu1985]

I've reproduced issue and this is the latest pod-log from powerstore controller pods
csi_controller_b76vk_log.txt
csi_controller_dsvpr_log.txt

[cemiscen]

Hi there,

I'm experiencing a similar issue with NFS mount. CSI driver creates the volume on the powerstore and maps to one of the nodes. I can see the "bound" messages for both pv and pvc however container/pod is getting "access denied by server" message and is stuck in "containercreating" state. I'm running simple.yaml to do this test.

I've found a workaround for this issue though! As soon as, the "file system/nas export" default access is changed to "Read/Wright, allow Root" (didn't test just read/wright option) on the PowerStore gui, pod/container creation works fine.

thanks,

[suryagupta4]

@LeoHu1985: This issue has been fixed and will be available in the next release that is CSI-PowerStore v2.6

@cemiscen: The issue which you reported seems to be a different one and not related to this. We will be working on it. You may create a new issue if needed.

Closing this issue.

Thanks.

[fabiodmf]

Hello,

Is there any workaround for this issue? We have the same problem here.

• CSI Driver version 2.5
• Power Store version 2.0.1.1
• Ubuntu 22.04.1 LTS
• Kubernetes v1.24.9

Does the driver work with any Kubernetes/OS/PowerStore version combination?

Do we have an estimated time for the CSI-PowerStore v2.6 release?

Thank you,
Fabio