Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CephFS Storage Class #21778

faust64 opened this issue Jan 12, 2019 · 3 comments


Copy link

commented Jan 12, 2019

Trying to leverage CephFS volumes, looking at Kubernetes Incubator samples:



openshift v3.11.0+9b6666e-51
kubernetes v1.11.0+d4cacc0

Steps To Reproduce
  1. Create CephFS provisioner and storage class as illustraged in

Note: would want to edit their clusterrole adding secrets create/update/delete permissions, and add their serviceaccount to the anyUid SCC,

kind: ClusterRole
  name: cephfs-provisioner
  namespace: cephfs
   - apiGroups: [""]
     resources: ["persistentvolumes"]
     verbs: ["get", "list", "watch", "create", "delete"]
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["create", "get", "delete"]
   - apiGroups: [""]
     resources: ["persistentvolumeclaims"]
     verbs: ["get", "list", "watch", "update"]
   - apiGroups: [""]
     resources: ["storageclasses"]
     verbs: ["get", "list", "watch"]
   - apiGroups: [""]
     resources: ["events"]
     verbs: ["create", "update", "patch"]
   - apiGroups: [""]
     resources: ["services"]
     resourceNames: ["kube-dns","coredns"]
     verbs: ["list", "get"]

We could also want to patch their image, forcing the permissions on CephFS shares such as owning group may write them:

apiVersion: v1
kind: Template
  name: cephfs-provisioner
- apiVersion: v1
  kind: ImageStream
    name: cephfs-provisioner
      description: Keeps track of changes in CephFS Provisioner image
- apiVersion: v1
  kind: BuildConfig
    name: cephfs-provisioner
      description: Builds CephFS Provisioner images

        kind: ImageStreamTag
        name: cephfs-provisioner:latest
      dockerfile: |

        USER root

        RUN sed -i 's|0o755|0o775|g' /usr/lib/python2.7/site-packages/
      type: Dockerfile
      type: Docker
    - type: ConfigChange
  1. Create SC, a PVC and a Pod
kind: StorageClass
  name: cephfs
  adminId: admin
  adminSecretName: cephfs-secret
  adminSecretNamespace: cephfs
  claimRoot: /pvc-volumes
apiVersion: v1
kind: PersistentVolumeClaim
  name: test-cephfs-claim
  storageClassName: cephfs
  accessModes: [ ReadWriteMany ]
      storage: 1Gi
apiVersion: v1
kind: Pod
  name: pvc-test-cephfs
  - name: cephfs-rw
    - mountPath: "/mnt/cephfs"
      name: cephfs
    runAsUser: 0
  - name: cephfs
      claimName: test-cephfs-claim

PVC is successfully provisioned. Pod was able to mount it.

$ oc get pvc
NAME                STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-cephfs-claim   Bound     pvc-1add33e9-1671-11e9-b538-525400ddfbf0   1Gi        RWX            cephfs         49m
ocp-node# mount|grep ceph,, on /var/lib/origin/openshift.local.volumes/pods/e1ed0dd9-1675-11e9-8d47-52540016f6b8/volumes/ type ceph (rw,relatime,name=kubernetes-dynamic-user-1ae98874-1671-11e9-aa41-0a580a800140,secret=<hidden>,acl,wsize=16777216)

Note that the provisioner part is not strictly mandatory. We could very well mount CephFS Volumes without registering PVs, using something like the following:

apiVersion: v1
kind: Pod
  name: test-cephfs
  - name: cephfs-rw
    - mountPath: "/mnt/cephfs"
      name: cephfs
  - name: cephfs
      path: /my-test-share/
      user: admin
        name: cephfs-secret
      readOnly: false

Which produces the same result as further described.

Current Result

I can not read nor write from my CephFS volume:

sh-4.2$ ls -lZa /mnt
drwxr-xr-x. root root system_u:object_r:container_file_t:s0:c123,c456 .
drwxr-xr-x. root root system_u:object_r:container_file_t:s0:c123,c456 ..
drwxrwxr-x  root root ?                                cephfs
sh-4.2$ cd /mnt/cephfs
sh-4.2$ ls
ls: cannot open directory .: Permission denied
sh-4.2$ id
uid=1001 gid=0(root) groups=0(root)
sh-4.2$ echo toto >pouet
sh: pouet: Input/output error

Same result adding runAsUser: 0 to my securityContext and trusting my SA into the anyuid SCC.

Expected Result

It would be nice being able to read from and write to CephFS volumes.

Additional Information

Running OKD on CentOS 7.6. Ceph 12.2.7 on Debian Stretch.

Installing ceph-selinux to my OKD nodes did not help, although I do suspect there's something.
Whenever I try to write something, I can confirm a new event is logged by selinux:

# ausearch -m avc --start recent
time->Sat Jan 12 15:55:39 2019
type=PROCTITLE msg=audit(1547304939.081:390228): proctitle="/bin/sh"
type=SYSCALL msg=audit(1547304939.081:390228): arch=c000003e syscall=2 success=no exit=-5 a0=133cd60 a1=241 a2=1b6 a3=0 items=0 ppid=9570 pid=9576 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=4294967295 comm="sh" exe="/usr/bin/bash" subj=system_u:system_r:container_t:s0:c4,c12 key=(null)
type=AVC msg=audit(1547304939.081:390228): avc:  denied  { write } for  pid=9576 comm="sh" name="/" dev="ceph" ino=1099511627787 scontext=system_u:system_r:container_t:s0:c4,c12 tcontext=system_u:object_r:cephfs_t:s0 tclass=dir

Further troubleshooting, I can confirm that write accesses to CephFS also fail when issuing commands from my OpenShift hosts. Yet read accesses seems to work just fine:

# mount | grep ceph,, on /var/lib/origin/openshift.local.volumes/pods/352e28e3-16ae-11e9-b538-525400ddfbf0/volumes/ type ceph (rw,relatime,name=admin,secret=<hidden>,acl,wsize=16777216)
# ls -l /var/lib/origin/openshift.local.volumes/pods/352e28e3-16ae-11e9-b538-525400ddfbf0/volumes/
total 0
# echo toto >/var/lib/origin/openshift.local.volumes/pods/352e28e3-16ae-11e9-b538-525400ddfbf0/volumes/
-bash: /var/lib/origin/openshift.local.volumes/pods/352e28e3-16ae-11e9-b538-525400ddfbf0/volumes/ Input/output error

I could also notice that when turning selinux off on a compute node, its containers can read from cephfs shares - Permission Denied error's gone. Although I stilll can't write to them:

# setenforce 0
# oc rsh pvc-test-cephfs
# ls -l /mnt/cephfs
total 0
# cd /mnt/cephfs
# echo toto >pouet
sh: pouet: Input/output error

While I can confirm being able to mount and use my CephFS share from a non-OpenShift client:

# mount | grep ceph on /mnt/toto type ceph (rw,relatime,name=admin,secret=<hidden>,acl)
root@sisyphe:~# echo abcd >/mnt/toto/pouet
root@sisyphe:~# cat /mnt/toto/pouet 

This comment has been minimized.

Copy link

commented Jan 26, 2019

try installing ceph-fuse and set virt_sandbox_use_fusefs on and virt_use_fusefs on, those errors looks like you are not using kernel v4 on your centos machines, but using fuse is recomended with selinux.


This comment has been minimized.

Copy link

commented Apr 26, 2019

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale


This comment has been minimized.

Copy link

commented May 15, 2019

Sorry for the delay, ... today I had a chance to give it another look.

Having deployed a new Ceph cluster, on CentOS, using v13.2.5, I noticed a few differences.
Connecting to the node hosting my test container, I no longer see errors:

[root@compute4 ~]# mount|grep ' ceph ',, on /var/lib/origin/openshift.local.volumes/pods/f412e0da-7713-11e9-86b6-525400bec0a4/volumes/ type ceph (rw,relatime,name=kubernetes-dynamic-user-be74323c-7713-11e9-8ec1-0a580a810227,secret=<hidden>,acl,wsize=16777216)
[root@compute4 ~]# cd /var/lib/origin/openshift.local.volumes/pods/f412e0da-7713-11e9-86b6-525400bec0a4/volumes/ 
[root@compute4 pvc-be6b919f-7713-11e9-86b6-525400bec0a4]# echo toto >toto
[root@compute4 pvc-be6b919f-7713-11e9-86b6-525400bec0a4]# rm toto
rm: remove regular file ‘toto’? y

Now, from the Pod itself, I'm still getting permission denied, on both read and writes.

Following up on @hniedlich recommendations, I can confirm that installing ceph-fuse does fix the issue.
Having done so, then re-scheduling my test Pod, its PVC gets mounted using ceph-fuse:

[root@compute4 ~]# mount | grep ceph-fus
ceph-fuse on /var/lib/origin/openshift.local.volumes/pods/661e9137-7718-11e9-86b6-525400bec0a4/volumes/ type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

And from there, everything works:

[root@master1 ~]# oc rsh pvc-test-cephfs
sh-4.2# cd /mnt/cephfs
sh-4.2# ls
sh-4.2# date >toto
sh-4.2# cat toto
Wed May 15 13:57:12 UTC 2019

I guess we can close that issue. I would have preferred using cephfs kernel driver, I guess we'll just have to wait a little longer.

Thanks to @hniedlich (and @lusoheart ), that both suggested me with ceph-fuse.

@faust64 faust64 closed this May 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
4 participants
You can’t perform that action at this time.