cephfs: go with default permissions while creating subvolumes #3204

humblec · 2022-06-23T04:46:16Z

While creating subvolumes, CephFS driver set the mode to 777
and pass it along to go ceph apis which cause the subvolume
permission to be on 777, however if we create a subvolume
directly in the ceph cluster, the default permission bits are
set which is 755 for the subvolume. This commit try to stick
to the default behaviour even while creating the subvolume.

This also means that we can work with fsgrouppolicy set to
File in csiDriver object which is also addressed in this commit.

Additional Note for reviewer:

Rook Changes: rook/rook#10503

Signed-off-by: Humble Chirammal hchiramm@redhat.com

humblec · 2022-06-23T14:45:31Z

Jun 23 15:21:45.090: INFO: ExecWithOptions: execute(POST https://192.168.39.23:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-f87879d85-7c26z/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+ls+--format%3Djson+--pool%3Dreplicapool&container=rook-ceph-tools&container=rook-ceph-tools&stderr=true&stdout=true)
Jun 23 15:21:45.470: FAIL: backend images not matching kubernetes resource count,image count 22 kubernetes resource count 21

known one.

humblec · 2022-06-24T03:31:09Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

humblec · 2022-06-24T03:31:30Z

/retest ci/centos/mini-e2e/k8s-1.22

humblec · 2022-06-24T05:57:17Z

@Madhu-1ptal.. thanks

Madhu-1 · 2022-06-24T06:02:50Z

charts/ceph-csi-cephfs/templates/csidriver-crd.yaml

@@ -9,3 +9,4 @@ metadata:
 spec:
  attachRequired: false
  podInfoOnMount: false
+  fsGroupPolicy: File


@humblec can you provide details on why it should be File here, we have set fsGroupPolicy to ReadWriteOnceWithFSType in Rook for cephfs.

@Madhu-1 dropping the PR for Rook to change that.. pretty much ready .. I will link the same here.

@Madhu-1 rook/rook#10503 address the same

At present the CephFS CSI driver works with default mode ie, `ReadWriteOncewithFsType`. However `File` type is more apt for the CephFS CSI driver and this commit bring that change. The similar change is also introduced in ceph csi driver here: ceph/ceph-csi#3204 considering the `File` mode has been GAd in 1.23 kubernetes version and also we are lifting one of the problematic code path in this area via Ceph CSI driver changes, it is good to move the fsgrouppolicy to `File`. Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

humblec · 2022-06-24T07:28:20Z

/retest ci/centos/mini-e2e-helm/k8s-1.23

humblec · 2022-06-24T07:28:30Z

/retest ci/centos/mini-e2e/k8s-1.22

humblec · 2022-06-24T09:15:03Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

humblec · 2022-06-27T07:53:02Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

humblec · 2022-06-27T10:33:41Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

humblec · 2022-06-29T05:00:06Z

@Mergifyio refresh

humblec · 2022-06-29T05:00:30Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

mergify · 2022-06-29T05:00:40Z

refresh

✅ Pull request refreshed

humblec · 2022-06-29T09:16:18Z

/retest ci/centos/mini-e2e/k8s-1.22

humblec · 2022-06-29T09:16:33Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

humblec · 2022-06-29T09:26:59Z

/retest ci/centos/mini-e2e-helm/k8s-1.23

humblec · 2022-07-12T07:39:52Z

@Mergifyio rebase

mergify · 2022-07-12T07:40:34Z

rebase

✅ Branch has been successfully rebased

humblec · 2022-07-12T09:10:22Z

/retest ci/centos/k8s-e2e-external-storage/1.23

humblec · 2022-07-12T09:10:31Z

/retest ci/centos/k8s-e2e-external-storage/1.22

humblec · 2022-07-12T13:06:50Z

2 weeks to get the CI pass with merge commit !!! still not the PR in..

humblec · 2022-07-12T13:08:34Z

[Show complete log](https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/rest/organizations/jenkins/pipelines/mini-e2e-helm_k8s-1.22/runs/3737/nodes/99/steps/102/log/?start=0)

ls Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}

Jul 12 10:34:59.038: INFO: >>> kubeConfig: /root/.kube/config

Jul 12 10:34:59.039: INFO: ExecWithOptions: Clientset creation

Jul 12 10:34:59.040: INFO: ExecWithOptions: execute(POST https://192.168.39.249:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-6944779c47-s9xtt/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+image-meta+get+--pool%3Dreplicapool+--image%3Dcsi-vol-d9824c79-01c5-11ed-b9d0-da8054cf23ee+csi.storage.k8s.io%2Fvolumesnapshot%2Fnamespace&container=rook-ceph-tools&container=rook-ceph-tools&stderr=true&stdout=true)

Jul 12 10:35:00.093: INFO: stdErr occurred: failed to get metadata csi.storage.k8s.io/volumesnapshot/namespace of image : (2) No such file or directory

rbd: getting metadata failed: (2) No such file or directory

https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/organizations/jenkins/mini-e2e-helm_k8s-1.22/detail/mini-e2e-helm_k8s-1.22/3737/pipeline

humblec · 2022-07-12T13:12:06Z

/retest ci/centos/mini-e2e/k8s-1.21

humblec · 2022-07-12T13:12:20Z

/retest ci/centos/mini-e2e-helm/k8s-1.23

humblec · 2022-07-12T13:12:33Z

/retest ci/centos/mini-e2e-helm/k8s-1.22

humblec · 2022-07-12T13:12:49Z

/retest ci/centos/k8s-e2e-external-storage/1.23

humblec · 2022-07-12T16:05:23Z

/retest ci/centos/mini-e2e-helm/k8s-1.22

While creating subvolumes, CephFS driver set the mode to `777` and pass it along to go ceph apis which cause the subvolume permission to be on 777, however if we create a subvolume directly in the ceph cluster, the default permission bits are set which is 755 for the subvolume. This commit try to stick to the default behaviour even while creating the subvolume. This also means that we can work with fsgrouppolicy set to `File` in csiDriver object which is also addressed in this commit. Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

ceph-csi-bot · 2022-07-13T03:54:00Z

/retest ci/centos/mini-e2e-helm/k8s-1.22

ceph-csi-bot · 2022-07-13T03:54:01Z

@humblec "ci/centos/mini-e2e-helm/k8s-1.22" test failed. Logs are available at location for debugging

ceph-csi-bot · 2022-07-13T03:54:03Z

/retest ci/centos/k8s-e2e-external-storage/1.22

ceph-csi-bot · 2022-07-13T03:54:04Z

@humblec "ci/centos/k8s-e2e-external-storage/1.22" test failed. Logs are available at location for debugging

ceph-csi-bot · 2022-07-13T03:54:05Z

/retest ci/centos/k8s-e2e-external-storage/1.21

ceph-csi-bot · 2022-07-13T03:54:06Z

@humblec "ci/centos/k8s-e2e-external-storage/1.21" test failed. Logs are available at location for debugging

ceph-csi-bot · 2022-07-13T03:54:07Z

@Mergifyio requeue

mergify · 2022-07-13T03:54:33Z

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

ceph-csi-bot · 2022-07-13T04:46:03Z

@Mergifyio requeue

mergify · 2022-07-13T04:46:21Z

requeue

☑️ This pull request is already queued

humblec · 2022-07-13T04:51:22Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

At present the CephFS CSI driver works with default mode ie, `ReadWriteOncewithFsType`. However `File` type is more apt for the CephFS CSI driver and this commit bring that change. The similar change is also introduced in ceph csi driver here: ceph/ceph-csi#3204 considering the `File` mode has been GAd in 1.23 kubernetes version and also we are lifting one of the problematic code path in this area via Ceph CSI driver changes, it is good to move the fsgrouppolicy to `File`. Signed-off-by: Humble Chirammal <hchiramm@redhat.com> (cherry picked from commit 58a2f66)

mergify bot added the component/cephfs Issues related to CephFS label Jun 23, 2022

humblec changed the title ~~cephfs: go with default permissions while creating subvolumes~~ [wip] cephfs: go with default permissions while creating subvolumes Jun 23, 2022

humblec force-pushed the cephfs-permission-new branch from ec5deb0 to 6dc5d12 Compare June 23, 2022 13:23

humblec changed the title ~~[wip] cephfs: go with default permissions while creating subvolumes~~ cephfs: go with default permissions while creating subvolumes Jun 23, 2022

humblec requested review from Madhu-1 and nixpanic June 23, 2022 13:26

humblec requested a review from a team June 24, 2022 05:27

humblec added this to the release-3.7 milestone Jun 24, 2022

Madhu-1 reviewed Jun 24, 2022

View reviewed changes

humblec mentioned this pull request Jun 24, 2022

csi: change the fsgrouppolicy for CephFS driver to "File" mode rook/rook#10503

Merged

7 tasks

humblec requested a review from a team June 24, 2022 10:13

Madhu-1 approved these changes Jun 29, 2022

View reviewed changes

yati1998 approved these changes Jun 29, 2022

View reviewed changes

humblec force-pushed the cephfs-permission-new branch from 6dc5d12 to 7b03eb9 Compare June 29, 2022 07:06

humblec force-pushed the cephfs-permission-new branch from 11c2150 to e4e93af Compare July 6, 2022 04:57

humblec force-pushed the cephfs-permission-new branch from e4e93af to d7d64ca Compare July 12, 2022 07:40

humblec force-pushed the cephfs-permission-new branch from d7d64ca to 720dbf6 Compare July 12, 2022 18:19

humblec added the ci/retry/e2e Label to retry e2e retesting on approved PR's label Jul 13, 2022

mergify bot merged commit 1856647 into ceph:devel Jul 13, 2022
25 checks passed

miguelhar mentioned this pull request Mar 1, 2023

Unable to write to volume after modeAllRWX removal #3700

Closed

cephfs: go with default permissions while creating subvolumes #3204

cephfs: go with default permissions while creating subvolumes #3204

Conversation

humblec commented Jun 23, 2022 • edited

humblec commented Jun 23, 2022 • edited

humblec commented Jun 24, 2022

humblec commented Jun 24, 2022

humblec commented Jun 24, 2022

Madhu-1 Jun 24, 2022

Choose a reason for hiding this comment

humblec Jun 24, 2022

Choose a reason for hiding this comment

humblec Jun 24, 2022

Choose a reason for hiding this comment

humblec commented Jun 24, 2022

humblec commented Jun 24, 2022

humblec commented Jun 24, 2022

humblec commented Jun 27, 2022

humblec commented Jun 27, 2022

humblec commented Jun 29, 2022

humblec commented Jun 29, 2022

mergify bot commented Jun 29, 2022

✅ Pull request refreshed

humblec commented Jun 29, 2022

humblec commented Jun 29, 2022

humblec commented Jun 29, 2022

humblec commented Jul 12, 2022

mergify bot commented Jul 12, 2022

✅ Branch has been successfully rebased

humblec commented Jul 12, 2022

humblec commented Jul 12, 2022

humblec commented Jul 12, 2022

humblec commented Jul 12, 2022 • edited

humblec commented Jul 12, 2022

humblec commented Jul 12, 2022

humblec commented Jul 12, 2022

humblec commented Jul 12, 2022

humblec commented Jul 12, 2022

ceph-csi-bot commented Jul 13, 2022

ceph-csi-bot commented Jul 13, 2022

ceph-csi-bot commented Jul 13, 2022

ceph-csi-bot commented Jul 13, 2022

ceph-csi-bot commented Jul 13, 2022

ceph-csi-bot commented Jul 13, 2022

ceph-csi-bot commented Jul 13, 2022

mergify bot commented Jul 13, 2022

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

ceph-csi-bot commented Jul 13, 2022

mergify bot commented Jul 13, 2022

☑️ This pull request is already queued

humblec commented Jul 13, 2022

humblec commented Jun 23, 2022 •

edited

humblec commented Jun 23, 2022 •

edited

humblec commented Jul 12, 2022 •

edited