Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The created SC cannot be used #735

Closed
jieshiyeskey opened this issue Mar 27, 2023 · 34 comments
Closed

The created SC cannot be used #735

jieshiyeskey opened this issue Mar 27, 2023 · 34 comments

Comments

@jieshiyeskey
Copy link

Basic information
kubernetes-v1.26.3
directpv version v4.0.1

#kubectl get sc
NAME              PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
directpv-min-io   directpv-min-io   Delete          WaitForFirstConsumer   true                   13h

Test example

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-m
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  serviceName: redis-m
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:4-alpine
        ports:
        - containerPort: 6379
          name: redisport
        volumeMounts:
        - mountPath: /data
          name: redisdata
  volumeClaimTemplates:
    - metadata:
        name: redisdata
      spec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: "1Gi"
        storageClassName: directpv-min-io

** Result**

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS      REASON   AGE
pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507   1Gi        RWO            Delete           Bound    default/redisdata-redis-m-0   directpv-min-io            12m		


# kubectl get pvc
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
redisdata-redis-m-0   Bound    pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507   1Gi        RWO            directpv-min-io   12m	

# kubectl get pod
NAME                        READY   STATUS              RESTARTS      AGE
redis-m-0                   0/1     ContainerCreating   0             13m

Logs

# kubectl describe  pod redis-m-0
Name:             redis-m-0
Namespace:        default
Priority:         0
Service Account:  default
Node:             cn-cp-1/172.16.0.11
Start Time:       Sun, 26 Mar 2023 22:18:47 +0800
Labels:           app=redis
                  controller-revision-hash=redis-m-57cc8c6b88
                  statefulset.kubernetes.io/pod-name=redis-m-0
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Controlled By:    StatefulSet/redis-m
Containers:
  redis:
    Container ID:
    Image:          redis:4-alpine
    Image ID:
    Port:           6379/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data from redisdata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4tfxt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  redisdata:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  redisdata-redis-m-0
    ReadOnly:   false
  kube-api-access-4tfxt:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    13m                  default-scheduler  Successfully assigned default/redis-m-0 to cn-cp-1
  Warning  FailedMount  5m1s (x4 over 11m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[redisdata], unattached volumes=[redisdata kube-api-access-4tfxt]: timed out waiting for the condition
  Warning  FailedMount  89s (x14 over 13m)   kubelet            MountVolume.SetUp failed for volume "pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507" : rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted
  Warning  FailedMount  32s (x2 over 2m46s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[redisdata], unattached volumes=[kube-api-access-4tfxt redisdata]: timed out waiting for the condition

Problem
The preceding information is displayed to keep the pod in the ContainerCreating state. Previous directpv version has also been saved this problem, please master to check, thanks.
@balamurugana
Copy link
Member

If you were upgraded from v3.x, there is a known issue which fixed by #734.
If you haven't added any drives to DirectPV, you would need to add drives to succeed volume provisioning.

@jieshiyeskey
Copy link
Author

Deployment

  # wget https://github.com/minio/directpv/releases/download/v4.0.1/kubectl-directpv_4.0.1_linux_amd64
  # chmod 755 kubectl-directpv_4.0.1_linux_amd64
  # mv kubectl-directpv_4.0.1_linux_amd64  /usr/bin/kubectl-directpv
  # kubectl directpv --version
  directpv version v4.0.1

  # kubectl directpv install

# kubectl get pod -n directpv -o wide
  NAME                         READY   STATUS    RESTARTS      AGE   IP             NODE      NOMINATED NODE   READINESS GATES
  controller-7ddb59ffb-c8r7m   3/3     Running   0             15h   10.244.0.245   cn-cp-1   <none>           <none>
  controller-7ddb59ffb-cqp75   3/3     Running   0             15h   10.244.2.50    cn-cp-3   <none>           <none>
  controller-7ddb59ffb-pqdh8   3/3     Running   0             15h   10.244.1.23    cn-cp-2   <none>           <none>
  node-server-c64c4            4/4     Running   0             15h   10.244.1.183   cn-cp-2   <none>           <none>
  node-server-m9zfm            4/4     Running   0             15h   10.244.0.209   cn-cp-1   <none>           <none>
  node-server-wzzzd            4/4     Running   0             15h   10.244.2.205   cn-cp-3   <none>           <none>

  # kubectl get crd  |grep directpv
  directpvdrives.directpv.min.io               2023-03-26T13:48:56Z
  directpvinitrequests.directpv.min.io         2023-03-26T13:48:57Z
  directpvnodes.directpv.min.io                2023-03-26T13:48:56Z
  directpvvolumes.directpv.min.io              2023-03-26T13:48:56Z

# kubectl directpv discover --nodes=cn-cp{1...3} --drives=/dev/sdb 

# cat drives.yaml
  version: v1
  nodes:
      - name: cn-cp-1
        drives:
          - id: 8:16$hqz1YtDKvgdTonmA25V/47SDOjgPn892MXyIMl5LWjY=
            name: sdb
            size: 8589934592
            make: ATA cn-cp-1-0_SSD
            select: "yes"
      - name: cn-cp-3
        drives:
          - id: 8:16$Pd91k17Q8zELOSdaYAeKwzhZeoW5iERLKW0BXDHBISI=
            name: sdb
            size: 8589934592
            make: ATA cn-cp-3-0_SSD
            select: "yes"
      - name: cn-cp-2
        drives:
          - id: 8:16$FCOd79DphtO8iQ49ZrGXfIA1cDJLEiSZaZK6tAbrKBQ=
            name: sdb
            size: 8589934592
            make: ATA cn-cp-2-1_SSD
            select: "yes"

kubectl directpv init drives.yaml
# kubectl directpv info
┌───────────┬──────────┬───────────┬─────────┬────────┐
│ NODE      │ CAPACITY │ ALLOCATED │ VOLUMES │ DRIVES │
├───────────┼──────────┼───────────┼─────────┼────────┤
│ • cn-cp-1 │ 8.0 GiB  │ 0 B       │ 0       │ 1      │
│ • cn-cp-2 │ 8.0 GiB  │ 0 B       │ 0       │ 1      │
│ • cn-cp-3 │ 8.0 GiB  │ 0 B       │ 0       │ 1      │
└───────────┴──────────┴───────────┴─────────┴────────┘

# kubectl get sc
NAME              PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
directpv-min-io   directpv-min-io   Delete          WaitForFirstConsumer   true                   15h

@jieshiyeskey
Copy link
Author 

# kubectl directpv  list volumes
┌──────────────────────────────────────────┬──────────┬─────────┬───────┬─────────┬──────────────┬────────┐
│ VOLUME                                   │ CAPACITY │ NODE    │ DRIVE │ PODNAME │ PODNAMESPACE │ STATUS │
├──────────────────────────────────────────┼──────────┼─────────┼───────┼─────────┼──────────────┼────────┤
│ pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 │ 1.0 GiB  │ cn-cp-1 │ sdb   │ -       │ -            │ Ready  │
│ pvc-9b7aad8b-9b1b-4159-bdc1-f22fbce2313a │ 6.0 GiB  │ cn-cp-1 │ sdb   │ -       │ -            │ Ready  │
└──────────────────────────────────────────┴──────────┴─────────┴───────┴─────────┴──────────────┴────────┘ 

# kubectl directpv list drives
┌─────────┬──────┬───────────────────┬─────────┬──────────┬─────────┬────────┐
│ NODE    │ NAME │ MAKE              │ SIZE    │ FREE     │ VOLUMES │ STATUS │
├─────────┼──────┼───────────────────┼─────────┼──────────┼─────────┼────────┤
│ cn-cp-1 │ sdb  │ ATA cn-cp-1-0_SSD │ 8.0 GiB │ 1014 MiB │ 2       │ Ready  │
│ cn-cp-2 │ sdb  │ ATA cn-cp-2-1_SSD │ 8.0 GiB │ 8.0 GiB  │ -       │ Ready  │
│ cn-cp-3 │ sdb  │ ATA cn-cp-3-0_SSD │ 8.0 GiB │ 8.0 GiB  │ -       │ Ready  │
└─────────┴──────┴───────────────────┴─────────┴──────────┴─────────┴────────┘ 

# df -h |grep directpv
/dev/sdb             8.0G   90M  8.0G   2% /var/lib/directpv/mnt/6f005d73-e0d3-4640-9525-cbfaf6e5160d

Pod message

Unable to attach or mount volumes: unmounted volumes=[redisdata], unattached volumes=[redisdata kube-api-access-4tfxt]: timed out waiting for the condition 

Unable to attach or mount volumes: unmounted volumes=[redisdata], unattached volumes=[kube-api-access-4tfxt redisdata]: timed out waiting for the condition

MountVolume.SetUp failed for volume "pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507" : rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted

@jieshiyeskey
Copy link
Author

If you were upgraded from v3.x, there is a known issue which fixed by #734. If you haven't added any drives to DirectPV, you would need to add drives to succeed volume provisioning.

It's not a problem with #734
Can you tell me how to deal with it? Give me an example. Thank you very much

@balamurugana
Copy link
Member

Let me test locally and get back to you

@balamurugana balamurugana reopened this Mar 27, 2023
@Praveenrajmani
Copy link
Collaborator

Can you describe the PVC? @jieshiyeskey

Also the logs from directpv pods running in this node cn-cp-1?

@jieshiyeskey
Copy link
Author 

# kubectl describe pvc redisdata-redis-m-0
Name:          redisdata-redis-m-0
Namespace:     default
StorageClass:  directpv-min-io
Status:        Bound
Volume:        pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507
Labels:        app=redis
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
               volume.kubernetes.io/selected-node: cn-cp-1
               volume.kubernetes.io/storage-provisioner: directpv-min-io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       redis-m-0
Events:        <none>

@jieshiyeskey
Copy link
Author 

# kubectl logs controller-7ddb59ffb-c8r7m  -n directpv
Defaulted container "csi-provisioner" out of: csi-provisioner, csi-resizer, controller
W0326 13:49:07.277121       1 feature_gate.go:241] Setting GA feature gate Topology=true. It will be removed in a future release.
I0326 13:49:07.277168       1 feature_gate.go:249] feature gates: &{map[Topology:true]}
I0326 13:49:07.277181       1 csi-provisioner.go:154] Version: v3.4.0
I0326 13:49:07.277185       1 csi-provisioner.go:177] Building kube configs for running in cluster...
W0326 13:49:17.278619       1 connection.go:173] Still connecting to unix:///csi/csi.sock
I0326 13:49:22.537110       1 common.go:111] Probing CSI driver for readiness
I0326 13:49:22.539727       1 csi-provisioner.go:230] Detected CSI driver directpv-min-io
I0326 13:49:22.540757       1 csi-provisioner.go:302] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I0326 13:49:22.541236       1 controller.go:732] Using saving PVs to API server in background
I0326 13:49:22.541528       1 leaderelection.go:248] attempting to acquire leader lease directpv/directpv-min-io...
I0326 13:49:22.553682       1 leader_election.go:185] new leader detected, current leader: 1679838561515-8081-directpv-min-io

# kubectl logs node-server-m9zfm  -n directpv
Defaulted container "node-driver-registrar" out of: node-driver-registrar, node-server, node-controller, liveness-probe
I0326 13:49:01.846415    3377 main.go:166] Version: v2.6.3
I0326 13:49:01.846446    3377 main.go:167] Running node-driver-registrar in mode=registration
I0326 13:49:01.847260    3377 main.go:191] Attempting to open a gRPC connection with: "unix:///csi/csi.sock"
W0326 13:49:11.848031    3377 connection.go:173] Still connecting to unix:///csi/csi.sock
I0326 13:49:18.020648    3377 main.go:198] Calling CSI driver to discover driver name
I0326 13:49:18.026522    3377 main.go:208] CSI driver name: "directpv-min-io"
I0326 13:49:18.026649    3377 node_register.go:53] Starting Registration Server at: /registration/directpv-min-io-reg.sock
I0326 13:49:18.028102    3377 node_register.go:62] Registration Server started at: /registration/directpv-min-io-reg.sock
I0326 13:49:18.028375    3377 node_register.go:92] Skipping HTTP server because endpoint is set to: ""
I0326 13:49:18.319684    3377 main.go:102] Received GetInfo call: &InfoRequest{}
I0326 13:49:18.320056    3377 main.go:109] "Kubelet registration probe created" path="/var/lib/kubelet/plugins/directpv-min-io/registration"
I0326 13:49:18.360806    3377 main.go:120] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

@Praveenrajmani
Copy link
Collaborator

we need the logs from node-server container @jieshiyeskey

kubectl logs node-server-m9zfm -n directpv -c node-server

@jieshiyeskey
Copy link
Author 

E0327 06:16:46.780677    3478 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/591a997e2347f1ddfd81473958f57258537d8c3b5defac0aaaa7ef86adbf55c8/globalmount is not mounted
E0327 06:16:46.780701    3478 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/591a997e2347f1ddfd81473958f57258537d8c3b5defac0aaaa7ef86adbf55c8/globalmount is not mounted"
I0327 06:17:29.502383    3478 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount" targetPath="/var/lib/kubelet/pods/af9a0de5-007f-4dfb-a414-b36bdfd48ed9/volumes/kubernetes.io~csi/pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507/mount"
E0327 06:17:29.511351    3478 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted
E0327 06:17:29.511392    3478 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted"
I0327 06:18:48.828598    3478 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-9b7aad8b-9b1b-4159-bdc1-f22fbce2313a" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/591a997e2347f1ddfd81473958f57258537d8c3b5defac0aaaa7ef86adbf55c8/globalmount" targetPath="/var/lib/kubelet/pods/760506a7-6f83-4e22-a3d1-d871050b9cc0/volumes/kubernetes.io~csi/pvc-9b7aad8b-9b1b-4159-bdc1-f22fbce2313a/mount"
E0327 06:18:48.838347    3478 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/591a997e2347f1ddfd81473958f57258537d8c3b5defac0aaaa7ef86adbf55c8/globalmount is not mounted
E0327 06:18:48.838388    3478 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/591a997e2347f1ddfd81473958f57258537d8c3b5defac0aaaa7ef86adbf55c8/globalmount is not mounted"
I0327 06:19:31.526212    3478 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount" targetPath="/var/lib/kubelet/pods/af9a0de5-007f-4dfb-a414-b36bdfd48ed9/volumes/kubernetes.io~csi/pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507/mount"
E0327 06:19:31.535931    3478 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted
E0327 06:19:31.535952    3478 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted"
I0327 06:20:50.912156    3478 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-9b7aad8b-9b1b-4159-bdc1-f22fbce2313a" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/591a997e2347f1ddfd81473958f57258537d8c3b5defac0aaaa7ef86adbf55c8/globalmount" targetPath="/var/lib/kubelet/pods/760506a7-6f83-4e22-a3d1-d871050b9cc0/volumes/kubernetes.io~csi/pvc-9b7aad8b-9b1b-4159-bdc1-f22fbce2313a/mount"
E0327 06:20:50.922738    3478 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/591a997e2347f1ddfd81473958f57258537d8c3b5defac0aaaa7ef86adbf55c8/globalmount is not mounted
E0327 06:20:50.922793    3478 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/591a997e2347f1ddfd81473958f57258537d8c3b5defac0aaaa7ef86adbf55c8/globalmount is not mounted"
I0327 06:21:33.597531    3478 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount" targetPath="/var/lib/kubelet/pods/af9a0de5-007f-4dfb-a414-b36bdfd48ed9/volumes/kubernetes.io~csi/pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507/mount"
E0327 06:21:33.606952    3478 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted
E0327 06:21:33.607074    3478 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted"
[root@cn-cp-1 ~]#

@Praveenrajmani
Copy link
Collaborator

Is this the complete log? Can you restart the workload pod once - kubectl delete and then check the directpv pod? @jieshiyeskey

@Praveenrajmani
Copy link
Collaborator

Praveenrajmani commented Mar 27, 2023 •
edited

I don't see stage volume call made by k8s. Lets check once more by deleting the pods.

@jieshiyeskey
Copy link
Author

Just a second. I'm gonna clean up the environment

@jieshiyeskey
Copy link
Author 

[root@cn-cp-1 ~]# kubectl -n directpv get pod -o wide
NAME                         READY   STATUS    RESTARTS   AGE   IP             NODE      NOMINATED NODE   READINESS GATES
controller-7ddb59ffb-4g827   2/3     Running   0          57s   10.244.0.67    cn-cp-1   <none>           <none>
controller-7ddb59ffb-gthcr   2/3     Running   0          46s   10.244.2.202   cn-cp-3   <none>           <none>
controller-7ddb59ffb-sh2vs   2/3     Running   0          37s   10.244.1.186   cn-cp-2   <none>           <none>
node-server-66ph8            3/4     Running   0          17s   10.244.0.56    cn-cp-1   <none>           <none>
node-server-k8gj7            3/4     Running   0          26s   10.244.1.187   cn-cp-2   <none>           <none>
node-server-ththf            3/4     Running   0          10s   10.244.2.253   cn-cp-3   <none>           <none>
[root@cn-cp-1 ~]# kubectl logs node-server-66ph8  -n directpv -c node-server
I0327 06:42:59.074026    6845 node-server.go:72] Identity server started
I0327 06:42:59.074103    6845 node-server.go:95] Node server started
I0327 06:42:59.074303    6845 ready.go:42] Serving readiness endpoint at :30443
I0327 06:42:59.075612    6845 metrics.go:63] Starting metrics exporter at port 10443
I0327 06:42:59.075883    6845 reflector.go:221] Starting reflector *v1beta1.DirectPVDrive (10m0s) from k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169
I0327 06:42:59.075905    6845 reflector.go:257] Listing and watching *v1beta1.DirectPVDrive from k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169
I0327 06:42:59.076539    6845 reflector.go:221] Starting reflector *v1beta1.DirectPVVolume (10m0s) from k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169
I0327 06:42:59.076550    6845 reflector.go:257] Listing and watching *v1beta1.DirectPVVolume from k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169
I0327 06:42:59.176200    6845 controller.go:141] drive controller synced and ready
I0327 06:42:59.176483    6845 controller.go:141] volume controller synced and ready

@Praveenrajmani
Copy link
Collaborator 

➜  directpv git:(devel) ✗ kubectl apply -f test.yaml 
statefulset.apps/redis-m created
➜  directpv git:(devel) ✗ kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
redis-m-0   1/1     Running   0          39s
➜  directpv git:(devel) ✗ kubectl get pvc 
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
redisdata-redis-m-0   Bound    pvc-ab5ae7c0-484b-41ba-93e5-72dff7bab0ea   500Mi      RWO            directpv-min-io   40s
➜  directpv git:(devel) ✗ kubectl get pv 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS      REASON   AGE
pvc-ab5ae7c0-484b-41ba-93e5-72dff7bab0ea   500Mi      RWO            Delete           Bound    default/redisdata-redis-m-0   directpv-min-io            41s
➜  directpv git:(devel) ✗ kubectl directpv list drives
┌────────────────────────────────┬──────┬───────────┬─────────┬─────────┬─────────┬────────┐
│ NODE                           │ NAME │ MAKE      │ SIZE    │ FREE    │ VOLUMES │ STATUS │
├────────────────────────────────┼──────┼───────────┼─────────┼─────────┼─────────┼────────┤
│ praveen-thinkpad-x1-carbon-6th │ dm-1 │ vg0-lv--1 │ 800 MiB │ 294 MiB │ 1       │ Ready  │
│ praveen-thinkpad-x1-carbon-6th │ dm-0 │ vg0-lv--0 │ 800 MiB │ 794 MiB │ -       │ Ready  │
│ praveen-thinkpad-x1-carbon-6th │ dm-2 │ vg0-lv--2 │ 800 MiB │ 794 MiB │ -       │ Ready  │
│ praveen-thinkpad-x1-carbon-6th │ dm-3 │ vg0-lv--3 │ 800 MiB │ 794 MiB │ -       │ Ready  │
└────────────────────────────────┴──────┴───────────┴─────────┴─────────┴─────────┴────────┘
➜  directpv git:(devel) ✗ kubectl directpv list volumes 
┌──────────────────────────────────────────┬──────────┬────────────────────────────────┬───────┬───────────┬──────────────┬─────────┐
│ VOLUME                                   │ CAPACITY │ NODE                           │ DRIVE │ PODNAME   │ PODNAMESPACE │ STATUS  │
├──────────────────────────────────────────┼──────────┼────────────────────────────────┼───────┼───────────┼──────────────┼─────────┤
│ pvc-ab5ae7c0-484b-41ba-93e5-72dff7bab0ea │ 500 MiB  │ praveen-thinkpad-x1-carbon-6th │ dm-1  │ redis-m-0 │ default      │ Bounded │
└──────────────────────────────────────────┴──────────┴────────────────────────────────┴───────┴───────────┴──────────────┴─────────┘
➜  directpv git:(devel) ✗ kubectl directpv info        
┌──────────────────────────────────┬──────────┬───────────┬─────────┬────────┐
│ NODE                             │ CAPACITY │ ALLOCATED │ VOLUMES │ DRIVES │
├──────────────────────────────────┼──────────┼───────────┼─────────┼────────┤
│ • praveen-thinkpad-x1-carbon-6th │ 3.1 GiB  │ 500 MiB   │ 1       │ 4      │
└──────────────────────────────────┴──────────┴───────────┴─────────┴────────┘

500 MiB/3.1 GiB used, 1 volumes, 4 drives
➜  directpv git:(devel) ✗ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:58:16Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:12Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
➜  directpv git:(devel) ✗ kubectl directpv --version
directpv version v4.0.1
➜  directpv git:(devel) ✗

I couldn't reproduce this locally @jieshiyeskey

@Praveenrajmani
Copy link
Collaborator

can you delete the redis pod once? @jieshiyeskey

@balamurugana
Copy link
Member

It looks like you have local issue.

@jieshiyeskey
Copy link
Author

ok.

# kubectl delete -f redis-sc.yaml
# kubectl delete  pvc redisdata-redis-m-0 
# kubectl apply -f redis-sc.yaml

[root@cn-cp-1 ~]# kubectl get pod |grep redis
redis-m-0                   0/1     ContainerCreating   0             115s

[root@cn-cp-1 ~]# kubectl describe  pod redis-m-0
Name:             redis-m-0
Namespace:        default
Priority:         0
Service Account:  default
Node:             cn-cp-1/172.16.0.11
Start Time:       Mon, 27 Mar 2023 14:47:44 +0800
Labels:           app=redis
                  controller-revision-hash=redis-m-57cc8c6b88
                  statefulset.kubernetes.io/pod-name=redis-m-0
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Controlled By:    StatefulSet/redis-m
Containers:
  redis:
    Container ID:
    Image:          redis:4-alpine
    Image ID:
    Port:           6379/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data from redisdata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mg6v6 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  redisdata:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  redisdata-redis-m-0
    ReadOnly:   false
  kube-api-access-mg6v6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    2m30s                default-scheduler  Successfully assigned default/redis-m-0 to cn-cp-1
  Warning  FailedMount  27s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[redisdata], unattached volumes=[kube-api-access-mg6v6 redisdata]: timed out waiting for the condition
  Warning  FailedMount  22s (x9 over 2m30s)  kubelet            MountVolume.SetUp failed for volume "pvc-73ed4029-9423-4396-a418-6a178b6da64f" : rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted

@Praveenrajmani
Copy link
Collaborator

Now can you get the logs from DirectPV pod? @jieshiyeskey

@jieshiyeskey
Copy link
Author 

[root@cn-cp-1 ~]# kubectl logs node-server-66ph8  -n directpv -c node-server
I0327 06:42:59.074026    6845 node-server.go:72] Identity server started
I0327 06:42:59.074103    6845 node-server.go:95] Node server started
I0327 06:42:59.074303    6845 ready.go:42] Serving readiness endpoint at :30443
I0327 06:42:59.075612    6845 metrics.go:63] Starting metrics exporter at port 10443
I0327 06:42:59.075883    6845 reflector.go:221] Starting reflector *v1beta1.DirectPVDrive (10m0s) from k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169
I0327 06:42:59.075905    6845 reflector.go:257] Listing and watching *v1beta1.DirectPVDrive from k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169
I0327 06:42:59.076539    6845 reflector.go:221] Starting reflector *v1beta1.DirectPVVolume (10m0s) from k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169
I0327 06:42:59.076550    6845 reflector.go:257] Listing and watching *v1beta1.DirectPVVolume from k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169
I0327 06:42:59.176200    6845 controller.go:141] drive controller synced and ready
I0327 06:42:59.176483    6845 controller.go:141] volume controller synced and ready
I0327 06:43:56.372023    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount" targetPath="/var/lib/kubelet/pods/af9a0de5-007f-4dfb-a414-b36bdfd48ed9/volumes/kubernetes.io~csi/pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507/mount"
E0327 06:43:56.384040    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted
E0327 06:43:56.384109    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted"
I0327 06:45:58.441528    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount" targetPath="/var/lib/kubelet/pods/af9a0de5-007f-4dfb-a414-b36bdfd48ed9/volumes/kubernetes.io~csi/pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507/mount"
E0327 06:45:58.455575    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted
E0327 06:45:58.455611    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount is not mounted"
I0327 06:46:35.708902    6845 stage_unstage.go:76] "Unstage volume requested" volumeID="pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507" StagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount"
I0327 06:46:36.729406    6845 quota_linux.go:230] "SetQuota succeeded" Device="/dev/sdb" Path="/var/lib/directpv/mnt/6f005d73-e0d3-4640-9525-cbfaf6e5160d/.FSUUID.6f005d73-e0d3-4640-9525-cbfaf6e5160d/pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507" VolumeID="pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507" ProjectID=200658450 HardLimit=0
E0327 06:46:36.779419    6845 controller.go:171] Operation cannot be fulfilled on directpvvolumes.directpv.min.io "pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507": StorageError: invalid object, Code: 4, Key: /registry/directpv.min.io/directpvvolumes/pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 90beacc5-0878-4033-ba81-4d00e83cbb71, UID in object meta:
E0327 06:46:36.898635    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
E0327 06:46:37.099143    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
E0327 06:46:37.500211    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
E0327 06:46:38.301079    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
E0327 06:46:39.902128    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
E0327 06:46:43.103125    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
E0327 06:46:49.504665    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
E0327 06:47:02.306944    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
E0327 06:47:27.915905    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
I0327 06:47:44.222361    6845 stage_unstage.go:37] "Stage volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" StagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount"
I0327 06:47:44.230681    6845 quota_linux.go:230] "SetQuota succeeded" Device="/dev/sdb" Path="/var/lib/directpv/mnt/6f005d73-e0d3-4640-9525-cbfaf6e5160d/.FSUUID.6f005d73-e0d3-4640-9525-cbfaf6e5160d/pvc-73ed4029-9423-4396-a418-6a178b6da64f" VolumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" ProjectID=3900925828 HardLimit=1073741824
I0327 06:47:44.246968    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:47:44.254852    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:47:44.255006    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
I0327 06:47:44.823087    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:47:44.830599    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:47:44.830623    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
I0327 06:47:45.843143    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:47:45.850658    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:47:45.850696    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
I0327 06:47:47.911353    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:47:47.920734    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:47:47.920768    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
I0327 06:47:51.960679    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:47:51.970187    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:47:51.970212    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
I0327 06:48:00.059373    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:48:00.069227    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:48:00.069260    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
I0327 06:48:16.136657    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:48:16.147619    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:48:16.147641    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
E0327 06:48:19.116956    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
I0327 06:48:48.248334    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:48:48.259470    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:48:48.259516    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
I0327 06:49:52.348398    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:49:52.356727    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:49:52.356888    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
E0327 06:50:01.518782    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
I0327 06:51:54.431100    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:51:54.440539    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:51:54.440580    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
E0327 06:53:26.325069    6845 controller.go:171] volume pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507 must be released before cleaning up
I0327 06:53:56.532632    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:53:56.543795    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:53:56.543893    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
I0327 06:55:58.568568    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:55:58.577314    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:55:58.577353    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
I0327 06:58:00.668079    6845 publish_unpublish.go:87] "Publish volume requested" volumeID="pvc-73ed4029-9423-4396-a418-6a178b6da64f" stagingTargetPath="/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount" targetPath="/var/lib/kubelet/pods/7eec188a-7181-4379-89c1-492aee11dcd7/volumes/kubernetes.io~csi/pvc-73ed4029-9423-4396-a418-6a178b6da64f/mount"
E0327 06:58:00.677314    6845 publish_unpublish.go:129] stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted
E0327 06:58:00.677354    6845 grpc.go:85] "GRPC failed" err="rpc error: code = Internal desc = stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount is not mounted"
[root@cn-cp-1 ~]#

@Praveenrajmani
Copy link
Collaborator

can you get the output of kubectl get directpvvolume pvc-73ed4029-9423-4396-a418-6a178b6da64f -o yaml @jieshiyeskey

@jieshiyeskey
Copy link
Author 

[root@cn-cp-1 ~]# kubectl get directpvvolume pvc-73ed4029-9423-4396-a418-6a178b6da64f -o yaml
apiVersion: directpv.min.io/v1beta1
kind: DirectPVVolume
metadata:
  creationTimestamp: "2023-03-27T06:47:43Z"
  finalizers:
  - directpv.min.io/pv-protection
  - directpv.min.io/purge-protection
  generation: 2
  labels:
    directpv.min.io/created-by: directpv-controller
    directpv.min.io/drive: 6f005d73-e0d3-4640-9525-cbfaf6e5160d
    directpv.min.io/drive-name: sdb
    directpv.min.io/node: cn-cp-1
    directpv.min.io/version: v1beta1
  name: pvc-73ed4029-9423-4396-a418-6a178b6da64f
  resourceVersion: "359691"
  uid: 216fe54f-1ced-4806-a4f0-91056050c9aa
status:
  availableCapacity: 1073741824
  dataPath: /var/lib/directpv/mnt/6f005d73-e0d3-4640-9525-cbfaf6e5160d/.FSUUID.6f005d73-e0d3-4640-9525-cbfaf6e5160d/pvc-73ed4029-9423-4396-a418-6a178b6da64f
  fsuuid: 6f005d73-e0d3-4640-9525-cbfaf6e5160d
  stagingTargetPath: /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount
  status: Ready
  targetPath: ""
  totalCapacity: 1073741824
  usedCapacity: 0

@Praveenrajmani
Copy link
Collaborator

Praveenrajmani commented Mar 27, 2023 •
edited

kubectl exec -it node-server-66ph8 -n directpv -c node-server -- cat /proc/1/mountinfo | grep -i "plugins"

can you get the output of this? @jieshiyeskey

@jieshiyeskey
Copy link
Author 

[root@cn-cp-1 ~]# kubectl exec -it node-server-66ph8 -n directpv -c node-server -- cat /proc/1/mountinfo | grep -i "plugins"
1835 87 8:16 /pvc-f45a06ec-de2b-4429-8c4f-3b5617d0f507.deleted//deleted /data/kubernetes/kubelet/plugins/kubernetes.io/csi/directpv-min-io/12cef50beac8e7fc71143fc1409b40ca77b2159e1c0ef1c7151d8f9dbda59b1c/globalmount rw,noatime shared:580 - xfs /dev/sdb rw,attr2,inode64,logbufs=8,logbsize=32k,prjquota
1883 87 8:16 /pvc-9b7aad8b-9b1b-4159-bdc1-f22fbce2313a.deleted//deleted /data/kubernetes/kubelet/plugins/kubernetes.io/csi/directpv-min-io/591a997e2347f1ddfd81473958f57258537d8c3b5defac0aaaa7ef86adbf55c8/globalmount rw,noatime shared:580 - xfs /dev/sdb rw,attr2,inode64,logbufs=8,logbsize=32k,prjquota
1677 87 8:16 /pvc-73ed4029-9423-4396-a418-6a178b6da64f /data/kubernetes/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount rw,noatime shared:580 - xfs /dev/sdb rw,attr2,inode64,logbufs=8,logbsize=32k,prjquota
[root@cn-cp-1 ~]#

@Praveenrajmani
Copy link
Collaborator

Praveenrajmani commented Mar 27, 2023 •
edited

1677 87 8:16 /pvc-73ed4029-9423-4396-a418-6a178b6da64f /data/kubernetes/kubelet/plugins/kubernetes.io/csi/directpv-min-io/846783e85629b1d29f439b582e0675aa7e5c0e34a60e6a9ecea7c94729508a67/globalmount rw,noatime shared:580 - xfs /dev/sdb rw,attr2,inode64,logbufs=8,logbsize=32k,prjquota

This is the problem. Looks like your kubelet root-dir is custom or messed up. DirectPV works with /var/lib/

@jieshiyeskey
Copy link
Author

[root@cn-cp-1 ~]# ls -l /var/lib/kubelet
lrwxrwxrwx 1 root root 24 Mar 22 13:55 /var/lib/kubelet -> /data/kubernetes/kubelet

@Praveenrajmani
Copy link
Collaborator

Praveenrajmani commented Mar 27, 2023 •
edited

[root@cn-cp-1 ~]# ls -l /var/lib/kubelet
lrwxrwxrwx 1 root root 24 Mar 22 13:55 /var/lib/kubelet -> /data/kubernetes/kubelet

DirectPV doesn't follow symlinks while checking mounts as it is non-standard

@jieshiyeskey
Copy link
Author

I see. Thank you very much.

Welcome to China in your spare time. Experience China.
Thank you, thank you.

@Praveenrajmani
Copy link
Collaborator

this PR #737 might help here.. to use custom root dir @jieshiyeskey

@Praveenrajmani
Copy link
Collaborator

Welcome to China in your spare time. Experience China.

Sure! :)

@jieshiyeskey
Copy link
Author

DirectPV works with /var/lib/

Excuse me, is this fixed? If you change kubelet root-dir, will DirectPV works automatically adapt?

@Praveenrajmani
Copy link
Collaborator

No, It won't automatically adapt. You need to set KUBELET_DIR_PATH during install kubectl directpv install

btw, this is not yet released. Stay tuned for the next release..

@jieshiyeskey
Copy link
Author

example:

Like this?

[root@cn-cp-1 ~]# export KUBELET_DIR_PATH=/xxxx/
[root@cn-cp-1 ~]# kubectl directpv install

@jieshiyeskey
Copy link
Author

No, It won't automatically adapt. You need to set KUBELET_DIR_PATH during install kubectl directpv install

btw, this is not yet released. Stay tuned for the next release..

OK

@guguducken guguducken mentioned this issue Sep 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@balamurugana @Praveenrajmani @jieshiyeskey