Modify CephFs provisioner to use the ceph mgr commands #400

poornimag · 2019-06-04T06:07:41Z

Currently CephFs provisioner mounts the ceph filesystem
and creates a subdirectory as a part of provisioning the
volume. Ceph now supports commands to provision fs subvolumes,
hance modify the provisioner to use ceph mgr commands to
(de)provision fs subvolumes.

Signed-off-by: Poornima G pgurusid@redhat.com

humblec · 2019-06-04T06:12:42Z

Thanks @poornimag . We ( + @Madhu-1 ) are reviewing this change.

humblec · 2019-06-04T06:15:29Z

@poornimag please check at the build failure.

humblec · 2019-06-04T06:24:28Z

@ajarr PTAL.

humblec · 2019-06-04T06:24:53Z

examples/cephfs/storageclass.yaml

@@ -22,6 +22,9 @@ parameters:
  # Required for provisionVolume: "true"
  pool: cephfs_data

+  # Name of the ceph filesystem with in which the volumes will be created


with in/within ?

Madhu-1 · 2019-06-04T07:56:44Z

examples/cephfs/storageclass.yaml

@@ -22,6 +22,9 @@ parameters:
  # Required for provisionVolume: "true"
  pool: cephfs_data

+  # Name of the ceph filesystem with in which the volumes will be created
+  fsName: cephfs


need to update E2E for this parameter here

need to update readme on how to get this parameter

Madhu-1 · 2019-06-04T08:00:05Z

pkg/cephfs/volume.go

-		return fmt.Errorf("error mounting ceph root: %v", err)
-	}
+func createVolume(volOptions *volumeOptions, volID volumeID, bytesQuota int64) error {
+	_, _, err := util.ExecCommand(


@poornimag do we need any changes in Node plugin for this?

Nope, we do not need

Madhu-1 · 2019-06-04T08:19:33Z

Fixes: #328

Madhu-1 · 2019-06-04T08:19:41Z

@gman0 PTAL

humblec · 2019-06-06T06:55:19Z

@poornimag Please revisit

humblec · 2019-06-08T07:40:04Z

@poornimag do we need any changes in the readme/documentation ?

poornimag · 2019-06-08T07:41:25Z

Succesfully tested the PVC creation and mounting:

[root@Centos1 ceph-csi]# docker images
REPOSITORY                                 TAG                                   IMAGE ID            CREATED             SIZE
quay.io/cephcsi/cephcsi                    canary                                fc60f8587fb3        2 hours ago         926MB
rook/ceph                                  master                                c42f5954684b        11 hours ago        921MB
ajarr/ceph                                 v14.2.1-20190430-and-mgr-subvolumes   fdd956f8d15d        40 hours ago        850MB
k8s.gcr.io/kube-proxy                      v1.14.3                               004666307c5b        2 days ago          82.1MB
k8s.gcr.io/kube-controller-manager         v1.14.3                               ac2ce44462bc        2 days ago          158MB
k8s.gcr.io/kube-apiserver                  v1.14.3                               9946f563237c        2 days ago          210MB
k8s.gcr.io/kube-scheduler                  v1.14.3                               953364a3ae7a        2 days ago          81.6MB
nginx                                      latest                                62c261073ecf        3 days ago          109MB
quay.io/cephcsi/cephfsplugin               v1.0.0                                06ae0938b3a2        11 days ago         924MB
ceph/ceph                                  v14.2                                 c3c0d6aa89f0        5 weeks ago         850MB
k8s.gcr.io/kube-addon-manager              v9.0                                  119701e77cbc        4 months ago        83.1MB
k8s.gcr.io/coredns                         1.3.1                                 eb516548c180        4 months ago        40.3MB
quay.io/k8scsi/csi-node-driver-registrar   v1.0.2                                8fed7d1b1015        5 months ago        20.1MB
quay.io/k8scsi/csi-provisioner             v1.0.1                                2004b031bce2        6 months ago        48MB
quay.io/k8scsi/csi-attacher                v1.0.1                                2ef5329b7139        6 months ago        50.2MB
k8s.gcr.io/etcd                            3.3.10                                2c4adeb21b4f        6 months ago        258MB
k8s.gcr.io/pause                           3.1                                   da86e6ba6ca1        17 months ago       742kB
gcr.io/k8s-minikube/storage-provisioner    v1.8.1                                4689081edb10        19 months ago       80.8MB
[root@Centos1 ceph-csi]# 
[root@Centos1 ceph-csi]# 
[root@Centos1 ceph-csi]# kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cephfs-pvc   Bound    pvc-66d0584c-89b8-11e9-800b-52540091bbf9   96Mi       RWO            csi-cephfs     57m
[root@Centos1 ceph-csi]# kubectl get pods
NAME                 READY   STATUS    RESTARTS   AGE
csicephfs-demo-pod   1/1     Running   0          53m
[root@Centos1 ceph-csi]# 
[root@Centos1 ceph-csi]# kubectl describe pod csicephfs-demo-pod
Name:               csicephfs-demo-pod
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               minikube/192.168.121.48
Start Time:         Sat, 08 Jun 2019 06:45:26 +0000
Labels:             <none>
Annotations:        kubectl.kubernetes.io/last-applied-configuration:
                      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"csicephfs-demo-pod","namespace":"default"},"spec":{"containers":[{"im...
Status:             Running
IP:                 172.17.0.12
Containers:
  web-server:
    Container ID:   docker://7d51f01e6203c98890ab721d5584945794ef59882afd7f434279882c37f944f3
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:12db363acf5b2d2f9f5fed240e228a04692bdac68402430fbd2f720c3a967d01
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sat, 08 Jun 2019 06:45:53 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/lib/www/html from mypvc (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-s4wp6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  mypvc:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  cephfs-pvc
    ReadOnly:   false
  default-token-s4wp6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-s4wp6
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                  Age   From                     Message
  ----    ------                  ----  ----                     -------
  Normal  Scheduled               53m   default-scheduler        Successfully assigned default/csicephfs-demo-pod to minikube
  Normal  SuccessfulAttachVolume  53m   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-66d0584c-89b8-11e9-800b-52540091bbf9"
  Normal  Pulling                 53m   kubelet, minikube        Pulling image "nginx"
  Normal  Pulled                  52m   kubelet, minikube        Successfully pulled image "nginx"
  Normal  Created                 52m   kubelet, minikube        Created container web-server
  Normal  Started                 52m   kubelet, minikube        Started container web-server
[root@Centos1 ceph-csi]# 




[root@Centos1 ceph-csi]# kubectl exec -it rook-ceph-tools-b8c679f95-56mbl -nrook-ceph /bin/bash
[root@Centos1 /]# ll /mnt/myfs/csi-volumes/                
total 1
drwxr-x---. 2 root root 0 Jun  8 06:33 csi-cephfs-pvc-6386ec62-89b7-11e9-800b-52540091bbf9-creating
drwxr-x---. 2 root root 0 Jun  8 06:46 csi-cephfs-pvc-66d0584c-89b8-11e9-800b-52540091bbf9
[root@Centos1 /]# ll /mnt/myfs/csi-volumes/csi-cephfs-pvc-66d0584c-89b8-11e9-800b-52540091bbf9
total 0
-rw-r--r--. 1 root root 0 Jun  8 06:46 abc
[root@Centos1 /]# 
[root@Centos1 ceph]# kubectl exec -it csicephfs-demo-pod /bin/bash
root@csicephfs-demo-pod:/# 
root@csicephfs-demo-pod:/# ls /var/lib/www/html
abc

poornimag · 2019-06-08T07:42:18Z

@poornimag do we need any changes in the readme/documentation ?

No, the fsName parameter is already added in a different path, so this patch has not added any new parameters

humblec · 2019-06-08T07:46:09Z

@poornimag Awesome! Can you also please test or write some data in this volume from the pod and check its writable ?

poornimag · 2019-06-08T07:48:03Z

@poornimag Awesome! Can you also please test or write some data in this volume from the pod and check its writable ?

Yes, have created a file abc in the ngnix test pod, and i see that its created from a different mount point (in the tool box). The log i have pasted also contains this...

poornimag · 2019-06-08T07:50:58Z

However, during the testing came across an issue:

There was a stale subvolume that was created csi-cephfs-pvc-6386ec62-89b7-11e9-800b-52540091bbf9-creating. There were some attempts of pv creation that failed, may be one of the failure code paths didn't cleanup the temp pvc
Will retest and raise an issue.

humblec · 2019-06-08T07:54:25Z

There was a stale subvolume that was created csi-cephfs-pvc-6386ec62-89b7-11e9-800b-52540091bbf9-creating. There were some attempts of pv creation that failed, may be one of the failure code paths didn't cleanup the temp pvc
Will retest and raise an issue.

Do we know what is the error logged at time of failed PV creation? that would become handy while raising the issue too. Yeah, lets log it if there is such condition.

humblec · 2019-06-08T08:06:47Z

pkg/cephfs/volume.go

 )

 const (
 	cephVolumesRoot = "csi-volumes"

-	namespacePrefix = "ns-"
+	namespacePrefix = "fsvolumens-"


@poornimag any specific reason for namespace prefix change here ?

Its changed to the default value specified in the ceph subvol create code

Thanks for clarifying

@poornimag but Isnt it fsvolumens_

Yes. This needs to be 'fsvolumens_'

Ideally, once user creation moves into the ceph fs subvolume create ... command I assume CSI code will not deal with this namespace name.

@ajarr Supporting user creation (based on some flag) is part of the implementation plan in the future?

humblec · 2019-06-08T08:12:54Z

The CI is failed and I believe its expected to fail as the ceph cluster in travis is not the version of nautilus we used for testing or rook templates dont have these latest images. The error what I can see there is:

Jun 8 08:05:39.755: INFO: At 2019-06-08 07:55:49 +0000 UTC - event for csi-cephfs-pvc: {cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-0_cd9cb849-89c2-11e9-864c-0242ac11000b } ProvisioningFailed: failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = an error (exit status 1) occurred while running ceph

I believe we could list better error to identify the exact reason though.

poornimag · 2019-06-08T08:21:34Z

The CI is failed and I believe its expected to fail as the ceph cluster in travis is not the version of nautilus we used for testing or rook templates dont have these latest images. The error what I can see there is:

Jun 8 08:05:39.755: INFO: At 2019-06-08 07:55:49 +0000 UTC - event for csi-cephfs-pvc: {cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-0_cd9cb849-89c2-11e9-864c-0242ac11000b } ProvisioningFailed: failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = an error (exit status 1) occurred while running ceph

I believe we could list better error to identify the exact reason though.

Yes, this is the exact error i got without Ramana's fix for quota in subvolumes. Then we wait for his patch to be merged and csi ci to succeed?
The error message is what is returned by the ceph subvolume command execution. Will see how to change

poornimag · 2019-06-08T08:23:35Z

Do we know what is the error logged at time of failed PV creation? that would become handy while raising the issue too. Yeah, lets log it if there is such condition.

The error was due to, not providing the monitor ip in the storage class.

humblec · 2019-06-08T08:32:12Z

@poornimag do we need createBackingVolume() func now ? or in short, do we need getAdminCredentials() and createCephUser() funcs from it?

poornimag · 2019-06-08T09:30:53Z

@poornimag do we need createBackingVolume() func now ? or in short, do we need getAdminCredentials() and createCephUser() funcs from it?

I did try to remove that, but createCephUser wasn't necessary even without this patch. Its the same as issue #269 . Will send it out as a separate patch.

pkg/cephfs/volume.go

ajarr · 2019-07-08T10:18:43Z

LGTM, from the POV of ceph-mgr subvolume commands

humblec · 2019-07-08T10:31:32Z

Thanks @poornimag ! LGTM. Approving.

humblec · 2019-07-08T10:39:24Z

@poornimag please update the PR description :)

Madhu-1 · 2019-07-08T15:44:12Z

@poornimag E2E is passing locally?

Currently CephFs provisioner mounts the ceph filesystem and creates a subdirectory as a part of provisioning the volume. Ceph now supports commands to provision fs subvolumes, hance modify the provisioner to use ceph mgr commands to (de)provision fs subvolumes. Signed-off-by: Poornima G <pgurusid@redhat.com>

humblec · 2019-07-09T13:55:18Z

@Madhu-1 Can you revisit this PR ? urgency due to release .

Madhu-1

please update readme for min ceph version, can be done in next PR

humblec · 2019-07-09T14:52:41Z

please update readme for min ceph version, can be done in next PR

Yeah , we will take care in a follow up PR!

Signed-off-by: Poornima G <pgurusid@redhat.com>

poornimag

WIP, testing the same

Madhu-1 · 2019-07-11T11:32:12Z

pkg/cephfs/volume.go

-
-	if err = m.mount(cephRoot, adminCr, volOptions); err != nil {
-		return fmt.Errorf("error mounting ceph root: %v", err)
+		klog.Errorf("failed to purge subvolume %s(%s) in fs %s", string(volID), err, volOptions.FsName)


avoid volID to string conversion use %v for printing

Madhu-1

LGTM, nit can be fixed later

humblec · 2019-07-12T05:24:47Z

Thanks all!! I am removing DNM as per our agreement.

humblec reviewed Jun 4, 2019

View reviewed changes

poornimag force-pushed the cephfs-cmd-mgr-fork branch from 2ba9478 to e1f5b46 Compare June 4, 2019 07:32

Madhu-1 requested changes Jun 4, 2019

View reviewed changes

humblec mentioned this pull request Jun 4, 2019

Release v1.1.0 of Ceph-CSI driver #353

Closed

5 tasks

humblec added the release-1.1.0 Track release 1.1 items label Jun 7, 2019

poornimag force-pushed the cephfs-cmd-mgr-fork branch 2 times, most recently from 181e802 to f0bda41 Compare June 8, 2019 07:37

poornimag force-pushed the cephfs-cmd-mgr-fork branch from f0bda41 to 7e9f509 Compare June 8, 2019 07:44

humblec reviewed Jun 8, 2019

View reviewed changes

This was referenced Jun 10, 2019

Add resizer sidecar to this repo. #413

Closed

Resize CephFS Volumes #412

Merged

humblec reviewed Jun 10, 2019

View reviewed changes

pkg/cephfs/volume.go Outdated Show resolved Hide resolved

poornimag force-pushed the cephfs-cmd-mgr-fork branch from b36e48c to c811ed3 Compare July 8, 2019 09:23

humblec approved these changes Jul 8, 2019

View reviewed changes

poornimag force-pushed the cephfs-cmd-mgr-fork branch from c811ed3 to 3d88b2c Compare July 8, 2019 11:36

poornimag force-pushed the cephfs-cmd-mgr-fork branch 5 times, most recently from 228cea1 to 5cd7d5b Compare July 9, 2019 10:54

poornimag force-pushed the cephfs-cmd-mgr-fork branch from 5cd7d5b to 9b552f1 Compare July 9, 2019 11:03

Madhu-1 approved these changes Jul 9, 2019

View reviewed changes

Backward compatibility for deleting and mounting old volumes

6310aed

Signed-off-by: Poornima G <pgurusid@redhat.com>

poornimag commented Jul 10, 2019

View reviewed changes

Madhu-1 reviewed Jul 11, 2019

View reviewed changes

Madhu-1 approved these changes Jul 12, 2019

View reviewed changes

humblec removed the DNM DO NOT MERGE label Jul 12, 2019

Merge branch 'master' into cephfs-cmd-mgr-fork

227438b

mergify bot merged commit 0d566ee into ceph:master Jul 12, 2019

ajarr mentioned this pull request Jul 22, 2019

Failed to create cephfs PVC #486

Closed

ajarr mentioned this pull request Aug 9, 2019

cephfs: optimize getVolumeRootPathCeph #539

Closed

jianglingxia mentioned this pull request Aug 27, 2019

cephfs create volume failed of subvolumegroup #574

Closed

jianglingxia mentioned this pull request Sep 7, 2019

pod delete but globalmount dir not delete #599

Closed

ShyamsundarR mentioned this pull request Jan 22, 2020

getVolumeRootPathCeph should detect and return ENOENT errors #474

Closed

Modify CephFs provisioner to use the ceph mgr commands #400

Modify CephFs provisioner to use the ceph mgr commands #400

Conversation

poornimag commented Jun 4, 2019 • edited

humblec commented Jun 4, 2019

humblec commented Jun 4, 2019

humblec commented Jun 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Madhu-1 commented Jun 4, 2019

Madhu-1 commented Jun 4, 2019

humblec commented Jun 6, 2019 • edited

humblec commented Jun 8, 2019

poornimag commented Jun 8, 2019 • edited

poornimag commented Jun 8, 2019

humblec commented Jun 8, 2019

poornimag commented Jun 8, 2019

poornimag commented Jun 8, 2019

humblec commented Jun 8, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajarr Jun 10, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humblec commented Jun 8, 2019

poornimag commented Jun 8, 2019

poornimag commented Jun 8, 2019

humblec commented Jun 8, 2019

poornimag commented Jun 8, 2019

ajarr commented Jul 8, 2019

humblec commented Jul 8, 2019 • edited

humblec commented Jul 8, 2019

Madhu-1 commented Jul 8, 2019

humblec commented Jul 9, 2019

Madhu-1 left a comment

Choose a reason for hiding this comment

humblec commented Jul 9, 2019

poornimag left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Madhu-1 left a comment

Choose a reason for hiding this comment

humblec commented Jul 12, 2019

poornimag commented Jun 4, 2019 •

edited

humblec commented Jun 6, 2019 •

edited

poornimag commented Jun 8, 2019 •

edited

humblec commented Jun 8, 2019 •

edited

ajarr Jun 10, 2019 •

edited

humblec commented Jul 8, 2019 •

edited