Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pods on different nodes mount Ceph RBD PVC stuck on ContainerCreating #85803

Open
D1abloRUS opened this issue Dec 2, 2019 · 1 comment

Comments

@D1abloRUS
Copy link

@D1abloRUS D1abloRUS commented Dec 2, 2019

What happened:
When use ceph as storageclass, i set accessModes: ReadOnlyMany, and create pods using the claim, only one pod would be running and the others on different nodes would get stuck on ContainerCreating. But if the pods are all on same node, it works fine. I created one pod write permissions, he created fs, killed this under, created read-only pods as a result of which stuck on ContainerCreating anyway

What you expected to happen:
Pods on different nodes can mount ceph rbd volume.

How to reproduce it (as minimally and precisely as possible):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: rbd-nvme

create pod rw perm:

apiVersion: v1
kind: Pod
metadata:
  name: rw-pod
spec:
  containers:
   - name: rw-pod
     image: ubuntu
     command: ["/bin/sh", "-c"]
     args: ["tail -f /dev/null"]
     volumeMounts:
       - name: mypvc-1
         mountPath: /var/lib/www/html
         readOnly: false
  volumes:
   - name: mypvc-1
     persistentVolumeClaim:
       claimName: rbd-pvc
       readOnly: false
  nodeSelector:
    kubernetes.io/hostname: node1

delete rw-pod and create 2 pod on different nodes:

apiVersion: v1
kind: Pod
metadata:
  name: csirbd-demo-pod
spec:
  containers:
   - name: web-server
     image: ubuntu
     command: ["/bin/sh", "-c"]
     args: ["tail -f /dev/null"]
     volumeMounts:
       - name: mypvc-1
         mountPath: /var/lib/www/html
         readOnly: true
  volumes:
   - name: mypvc-1
     persistentVolumeClaim:
       claimName: rbd-pvc
       readOnly: true
  nodeSelector:
    kubernetes.io/hostname: node1

---
apiVersion: v1
kind: Pod
metadata:
  name: csirbd-demo-pod-2
spec:
  containers:
   - name: web-server-2
     image: ubuntu
     command: ["/bin/sh", "-c"]
     args: ["tail -f /dev/null"]
     volumeMounts:
       - name: mypvc
         mountPath: /var/lib/www/html
         readOnly: true
  volumes:
   - name: mypvc
     persistentVolumeClaim:
       claimName: rbd-pvc
       readOnly: true
  nodeSelector:
    kubernetes.io/hostname: node2

kubectl get pod:

csirbd-demo-pod     1/1     Running             0          4h21m
csirbd-demo-pod-2   0/1     ContainerCreating   0          4h21m

kubectl describe pod/csirbd-demo-pod-2:

Events:
  Type     Reason       Age                      From            Message
  ----     ------       ----                     ----            -------
  Warning  FailedMount  15m (x91 over 4h21m)     kubelet, node2  MountVolume.MountDevice failed for volume "pvc-749fe5c6-9e56-4225-82bf-a5b89982ed04" : rpc error: code = Internal desc = rbd image replicapool/csi-vol-68c365b9-14e3-11ea-ab11-0667329af84c is still being used
  Warning  FailedMount  5m52s (x113 over 4h20m)  kubelet, node2  Unable to mount volumes for pod "csirbd-demo-pod-2_default(a567581c-1bbb-494b-925e-7a2730472a31)": timeout expired waiting for volumes to attach or mount for pod "default"/"csirbd-demo-pod-2". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-mnkf2]

Anything else we need to know?:
I used rook:

  • Rook version (use rook version inside of a Rook Pod): v.1.1.7
  • Storage backend version (e.g. for ceph do ceph -v): 14.2.4

Environment:

  • Kubernetes version (use kubectl version): v1.15.5
  • Cloud provider or hardware configuration: baremetal
  • OS (e.g: cat /etc/os-release): ubuntu 18.04.3 LTS
  • Kernel (e.g. uname -a): 4.15.0-66-generic
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:
@D1abloRUS

This comment has been minimized.

Copy link
Author

@D1abloRUS D1abloRUS commented Dec 2, 2019

currently ceph-csi don't support ReadOnlyMany rook/rook#4402

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.