Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CephFS Storage Class #21778

Closed
faust64 opened this issue Jan 12, 2019 · 3 comments

Comments

@faust64
Copy link

commented Jan 12, 2019

Trying to leverage CephFS volumes, looking at Kubernetes Incubator samples: https://github.com/kubernetes-incubator/external-storage/blob/master/ceph/cephfs

Version

OKD

openshift v3.11.0+9b6666e-51
kubernetes v1.11.0+d4cacc0

Steps To Reproduce
  1. Create CephFS provisioner and storage class as illustraged in https://github.com/kubernetes-incubator/external-storage/blob/master/ceph/cephfs

Note: would want to edit their clusterrole adding secrets create/update/delete permissions, and add their serviceaccount to the anyUid SCC,

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-provisioner
  namespace: cephfs
 rules:
   - apiGroups: [""]
     resources: ["persistentvolumes"]
     verbs: ["get", "list", "watch", "create", "delete"]
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["create", "get", "delete"]
   - apiGroups: [""]
     resources: ["persistentvolumeclaims"]
     verbs: ["get", "list", "watch", "update"]
   - apiGroups: ["storage.k8s.io"]
     resources: ["storageclasses"]
     verbs: ["get", "list", "watch"]
   - apiGroups: [""]
     resources: ["events"]
     verbs: ["create", "update", "patch"]
   - apiGroups: [""]
     resources: ["services"]
     resourceNames: ["kube-dns","coredns"]
     verbs: ["list", "get"]

We could also want to patch their image, forcing the permissions on CephFS shares such as owning group may write them:

apiVersion: v1
kind: Template
metadata:
  name: cephfs-provisioner
objects:
- apiVersion: v1
  kind: ImageStream
  metadata:
    name: cephfs-provisioner
    annotations:
      description: Keeps track of changes in CephFS Provisioner image
- apiVersion: v1
  kind: BuildConfig
  metadata:
    name: cephfs-provisioner
    annotations:
      description: Builds CephFS Provisioner images

  spec:
    output:
      to:
        kind: ImageStreamTag
        name: cephfs-provisioner:latest
    source:
      dockerfile: |
        FROM quay.io/external_storage/cephfs-provisioner:latest

        USER root

        RUN sed -i 's|0o755|0o775|g' /usr/lib/python2.7/site-packages/ceph_volume_client.py
      type: Dockerfile
    strategy:
      type: Docker
    triggers:
    - type: ConfigChange
  1. Create SC, a PVC and a Pod
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cephfs
provisioner: ceph.com/cephfs
parameters:
  adminId: admin
  adminSecretName: cephfs-secret
  adminSecretNamespace: cephfs
  claimRoot: /pvc-volumes
  monitors: 10.42.253.110:6789,10.42.253.111:6789,10.42.253.112:6789
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-cephfs-claim
spec:
  storageClassName: cephfs
  accessModes: [ ReadWriteMany ]
  resources:
    requests:
      storage: 1Gi
apiVersion: v1
kind: Pod
metadata:
  name: pvc-test-cephfs
spec:
  containers:
  - name: cephfs-rw
    image: docker.io/openshift/origin-pod
    volumeMounts:
    - mountPath: "/mnt/cephfs"
      name: cephfs
  securityContext:
    runAsUser: 0
  volumes:
  - name: cephfs
    persistentVolumeClaim:
      claimName: test-cephfs-claim

PVC is successfully provisioned. Pod was able to mount it.

$ oc get pvc
NAME                STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-cephfs-claim   Bound     pvc-1add33e9-1671-11e9-b538-525400ddfbf0   1Gi        RWX            cephfs         49m
ocp-node# mount|grep ceph
10.42.253.110:6789,10.42.253.111:6789,10.42.253.112:6789:/pvc-volumes/kubernetes/kubernetes-dynamic-pvc-1ae987df-1671-11e9-aa41-0a580a800140 on /var/lib/origin/openshift.local.volumes/pods/e1ed0dd9-1675-11e9-8d47-52540016f6b8/volumes/kubernetes.io~cephfs/pvc-1add33e9-1671-11e9-b538-525400ddfbf0 type ceph (rw,relatime,name=kubernetes-dynamic-user-1ae98874-1671-11e9-aa41-0a580a800140,secret=<hidden>,acl,wsize=16777216)

Note that the provisioner part is not strictly mandatory. We could very well mount CephFS Volumes without registering PVs, using something like the following:

apiVersion: v1
kind: Pod
metadata:
  name: test-cephfs
spec:
  containers:
  - name: cephfs-rw
    image: docker.io/openshift/origin-pod
    volumeMounts:
    - mountPath: "/mnt/cephfs"
      name: cephfs
  volumes:
  - name: cephfs
    cephfs:
      monitors:
      - 10.42.253.110:6789
      - 10.42.253.111:6789
      - 10.42.253.112:6789
      path: /my-test-share/
      user: admin
      secretRef:
        name: cephfs-secret
      readOnly: false

Which produces the same result as further described.

Current Result

I can not read nor write from my CephFS volume:

sh-4.2$ ls -lZa /mnt
drwxr-xr-x. root root system_u:object_r:container_file_t:s0:c123,c456 .
drwxr-xr-x. root root system_u:object_r:container_file_t:s0:c123,c456 ..
drwxrwxr-x  root root ?                                cephfs
sh-4.2$ cd /mnt/cephfs
sh-4.2$ ls
ls: cannot open directory .: Permission denied
sh-4.2$ id
uid=1001 gid=0(root) groups=0(root)
sh-4.2$ echo toto >pouet
sh: pouet: Input/output error

Same result adding runAsUser: 0 to my securityContext and trusting my SA into the anyuid SCC.

Expected Result

It would be nice being able to read from and write to CephFS volumes.

Additional Information

Running OKD on CentOS 7.6. Ceph 12.2.7 on Debian Stretch.

Installing ceph-selinux to my OKD nodes did not help, although I do suspect there's something.
Whenever I try to write something, I can confirm a new event is logged by selinux:

# ausearch -m avc --start recent
----
time->Sat Jan 12 15:55:39 2019
type=PROCTITLE msg=audit(1547304939.081:390228): proctitle="/bin/sh"
type=SYSCALL msg=audit(1547304939.081:390228): arch=c000003e syscall=2 success=no exit=-5 a0=133cd60 a1=241 a2=1b6 a3=0 items=0 ppid=9570 pid=9576 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=4294967295 comm="sh" exe="/usr/bin/bash" subj=system_u:system_r:container_t:s0:c4,c12 key=(null)
type=AVC msg=audit(1547304939.081:390228): avc:  denied  { write } for  pid=9576 comm="sh" name="/" dev="ceph" ino=1099511627787 scontext=system_u:system_r:container_t:s0:c4,c12 tcontext=system_u:object_r:cephfs_t:s0 tclass=dir

Further troubleshooting, I can confirm that write accesses to CephFS also fail when issuing commands from my OpenShift hosts. Yet read accesses seems to work just fine:

# mount | grep ceph
10.42.253.110:6789,10.42.253.111:6789,10.42.253.112:6789:/pvc-volumes/kubernetes/kubernetes-dynamic-pvc-cb752727-1685-11e9-aa41-0a580a800140 on /var/lib/origin/openshift.local.volumes/pods/352e28e3-16ae-11e9-b538-525400ddfbf0/volumes/kubernetes.io~cephfs/cephfs type ceph (rw,relatime,name=admin,secret=<hidden>,acl,wsize=16777216)
# ls -l /var/lib/origin/openshift.local.volumes/pods/352e28e3-16ae-11e9-b538-525400ddfbf0/volumes/kubernetes.io~cephfs/cephfs
total 0
# echo toto >/var/lib/origin/openshift.local.volumes/pods/352e28e3-16ae-11e9-b538-525400ddfbf0/volumes/kubernetes.io~cephfs/cephfs/toto
-bash: /var/lib/origin/openshift.local.volumes/pods/352e28e3-16ae-11e9-b538-525400ddfbf0/volumes/kubernetes.io~cephfs/cephfs/toto: Input/output error

I could also notice that when turning selinux off on a compute node, its containers can read from cephfs shares - Permission Denied error's gone. Although I stilll can't write to them:

# setenforce 0
# oc rsh pvc-test-cephfs
# ls -l /mnt/cephfs
total 0
# cd /mnt/cephfs
# echo toto >pouet
sh: pouet: Input/output error

While I can confirm being able to mount and use my CephFS share from a non-OpenShift client:

# mount | grep ceph
10.42.253.110:6789:/ on /mnt/toto type ceph (rw,relatime,name=admin,secret=<hidden>,acl)
root@sisyphe:~# echo abcd >/mnt/toto/pouet
root@sisyphe:~# cat /mnt/toto/pouet 
abcd
@hniedlich

This comment has been minimized.

Copy link

commented Jan 26, 2019

try installing ceph-fuse and set virt_sandbox_use_fusefs on and virt_use_fusefs on, those errors looks like you are not using kernel v4 on your centos machines, but using fuse is recomended with selinux.

@openshift-bot

This comment has been minimized.

Copy link

commented Apr 26, 2019

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@faust64

This comment has been minimized.

Copy link
Author

commented May 15, 2019

Sorry for the delay, ... today I had a chance to give it another look.

Having deployed a new Ceph cluster, on CentOS, using v13.2.5, I noticed a few differences.
Connecting to the node hosting my test container, I no longer see errors:

[root@compute4 ~]# mount|grep ' ceph '
10.42.253.110:6789,10.42.253.111:6789,10.42.253.112:6789:/pvc-volumes/kubernetes/kubernetes-dynamic-pvc-be7431be-7713-11e9-8ec1-0a580a810227 on /var/lib/origin/openshift.local.volumes/pods/f412e0da-7713-11e9-86b6-525400bec0a4/volumes/kubernetes.io~cephfs/pvc-be6b919f-7713-11e9-86b6-525400bec0a4 type ceph (rw,relatime,name=kubernetes-dynamic-user-be74323c-7713-11e9-8ec1-0a580a810227,secret=<hidden>,acl,wsize=16777216)
[root@compute4 ~]# cd /var/lib/origin/openshift.local.volumes/pods/f412e0da-7713-11e9-86b6-525400bec0a4/volumes/kubernetes.io~cephfs/pvc-be6b919f-7713-11e9-86b6-525400bec0a4 
[root@compute4 pvc-be6b919f-7713-11e9-86b6-525400bec0a4]# echo toto >toto
[root@compute4 pvc-be6b919f-7713-11e9-86b6-525400bec0a4]# rm toto
rm: remove regular file ‘toto’? y

Now, from the Pod itself, I'm still getting permission denied, on both read and writes.

Following up on @hniedlich recommendations, I can confirm that installing ceph-fuse does fix the issue.
Having done so, then re-scheduling my test Pod, its PVC gets mounted using ceph-fuse:

[root@compute4 ~]# mount | grep ceph-fus
ceph-fuse on /var/lib/origin/openshift.local.volumes/pods/661e9137-7718-11e9-86b6-525400bec0a4/volumes/kubernetes.io~cephfs/pvc-be6b919f-7713-11e9-86b6-525400bec0a4 type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

And from there, everything works:

[root@master1 ~]# oc rsh pvc-test-cephfs
sh-4.2# cd /mnt/cephfs
sh-4.2# ls
sh-4.2# date >toto
sh-4.2# cat toto
Wed May 15 13:57:12 UTC 2019

I guess we can close that issue. I would have preferred using cephfs kernel driver, I guess we'll just have to wait a little longer.

Thanks to @hniedlich (and @lusoheart ), that both suggested me with ceph-fuse.

@faust64 faust64 closed this May 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.