csi: option to customize csi driver name prefix

For now we are using the operator namespace name
as the prefix for the csi driver, This PR provides
an option for the users if someone wants to have
their own prefix for the csi driver, if someone tries
to change the prefix for existing csi driver rook
operator will fail to reconcile the csi driver.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

Documentation/Helm-Charts/ceph-cluster-chart.md

@@ -69,6 +69,7 @@ The following table lists the configurable parameters of the rook-operator chart
 | `cephObjectStores` | A list of CephObjectStore configurations to deploy | See [below](#ceph-object-stores) |
 | `clusterName` | The metadata.name of the CephCluster CR | The same as the namespace |
 | `configOverride` | Cluster ceph.conf override | `nil` |
+| `csiDriverNamePrefix` | CSI driver name prefix for cephfs, rbd and nfs. | `namespace name where rook-ceph operator is deployed` |
 | `ingress.dashboard` | Enable an ingress for the ceph-dashboard | `{}` |
 | `kubeVersion` | Optional override of the target kubernetes version | `nil` |
 | `monitoring.createPrometheusRules` | Whether to create the Prometheus rules for Ceph alerts | `false` |

Documentation/Helm-Charts/operator-chart.md

@@ -70,6 +70,7 @@ The following table lists the configurable parameters of the rook-operator chart
 | `csi.csiCephFSPluginVolume` | The volume of the CephCSI CephFS plugin DaemonSet | `nil` |
 | `csi.csiCephFSPluginVolumeMount` | The volume mounts of the CephCSI CephFS plugin DaemonSet | `nil` |
 | `csi.csiCephFSProvisionerResource` | CEPH CSI CephFS provisioner resource requirement list | see values.yaml |
+| `csi.csiDriverNamePrefix` | CSI driver name prefix for cephfs, rbd and nfs. | `namespace name where rook-ceph operator is deployed` |
 | `csi.csiLeaderElectionLeaseDuration` | Duration in seconds that non-leader candidates will wait to force acquire leadership. | `137s` |
 | `csi.csiLeaderElectionRenewDeadline` | Deadline in seconds that the acting leader will retry refreshing leadership before giving up. | `107s` |
 | `csi.csiLeaderElectionRetryPeriod` | Retry period in seconds the LeaderElector clients should wait between tries of actions. | `26s` |

Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md

@@ -35,6 +35,52 @@ example, if the Rook operator is running in the namespace `my-namespace` the
 provisioner value should be `my-namespace.rbd.csi.ceph.com`. The same provisioner
 name must be set in both the storageclass and snapshotclass.
 
+## Configure custom Driver name prefix for CSI Drivers
+
+To use a custom prefix for the CSI drivers, you need to set the
+`CSI_DRIVER_NAME_PREFIX` environment variable in the operator configmap. For
+instance, to use the prefix `my-prefix` for the CSI drivers, you can set the
+following in the operator configmap:
+
+```console
+kubectl patch cm rook-ceph-operator-config -n rook-ceph -p $'data:\n "CSI_DRIVER_NAME_PREFIX": "my-prefix"'
+```
+
+Once the configmap is updated, the CSI drivers will be deployed with the
+`my-prefix` prefix. You must set the same prefix in both the storageclass and
+snapshotclass. For example, if you want to use the prefix `my-prefix` for the
+CSI drivers, you can set the following in the storageclass:
+
+```yaml
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: rook-ceph-block
+provisioner: my-prefix.rbd.csi.ceph.com
+...
+```
+
+Same prefix must be set in the volumesnapshotclass as well:
+
+```yaml
+apiVersion: snapshot.storage.k8s.io/v1
+kind: VolumeSnapshotClass
+metadata:
+  name: rook-ceph-block
+driver: my-prefix.rbd.csi.ceph.com
+...
+```
+
+Please note that for the RBD driver, the prefix will be
+`my-prefix.rbd.csi.ceph.com`, for the CephFS driver, the prefix will be
+`my-prefix.cephfs.csi.ceph.com`, and for the NFS driver, the prefix will be
+`my-prefix.nfs.csi.ceph.com`.
+
+!!! note
+ Please be careful when setting the `CSI_DRIVER_NAME_PREFIX`
+environment variable. It should be done only in fresh deployments because
+changing the prefix in an existing cluster will result in unexpected behavior.
+
 ## Liveness Sidecar
 
 All CSI pods are deployed with a sidecar container that provides a Prometheus
@@ -116,7 +162,6 @@ Refer to the [ephemeral-doc](https://kubernetes.io/docs/concepts/storage/ephemer
 See example manifests for an [RBD ephemeral volume](https://github.com/rook/rook/tree/master/deploy/examples/csi/rbd/pod-ephemeral.yaml)
 and a [CephFS ephemeral volume](https://github.com/rook/rook/tree/master/deploy/examples/csi/cephfs/pod-ephemeral.yaml).
 
-
 ## CSI-Addons Controller
 
 The CSI-Addons Controller handles requests from users. Users create a CR

deploy/charts/rook-ceph-cluster/templates/cephblockpool.yaml

@@ -16,7 +16,11 @@ metadata:
   name: {{ $blockpool.storageClass.name }}
   annotations:
     storageclass.kubernetes.io/is-default-class: "{{ if default false $blockpool.storageClass.isDefault }}true{{ else }}false{{ end }}"
+{{- if $root.Values.csiDriverNamePrefix }}
+provisioner: {{ $root.Values.csiDriverNamePrefix }}.rbd.csi.ceph.com
+{{- else }}
 provisioner: {{ $root.Values.operatorNamespace }}.rbd.csi.ceph.com
+{{- end }}
 parameters:
   pool: {{ $blockpool.name }}
   clusterID: {{ $root.Release.Namespace }}

deploy/charts/rook-ceph-cluster/templates/cephecblockpool.yaml

@@ -16,7 +16,11 @@ apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
   name: {{ $cephEcStorage.name }}
-provisioner: {{ $cephEcStorage.provisioner }}
+{{- if $root.Values.csiDriverNamePrefix }}
+provisioner: {{ $root.Values.csiDriverNamePrefix }}.rbd.csi.ceph.com
+{{- else }}
+provisioner: {{ $root.Values.operatorNamespace }}.rbd.csi.ceph.com
+{{- end }}
 parameters:
   clusterID: {{ $cephEcStorage.parameters.clusterID }}
   dataPool: {{ $cephEcStorage.parameters.dataPool }}

deploy/charts/rook-ceph-cluster/templates/cephfilesystem.yaml

@@ -35,7 +35,11 @@ metadata:
   name: {{ $filesystem.storageClass.name }}
   annotations:
     storageclass.kubernetes.io/is-default-class: "{{ if default false $filesystem.storageClass.isDefault }}true{{ else }}false{{ end }}"
+{{- if $root.Values.csiDriverNamePrefix }}
+provisioner: {{ $root.Values.csiDriverNamePrefix }}.cephfs.csi.ceph.com
+{{- else }}
 provisioner: {{ $root.Values.operatorNamespace }}.cephfs.csi.ceph.com
+{{- end }}
 parameters:
   fsName: {{ $filesystem.name }}
   pool: {{ $filesystem.name }}-{{ default "data0" $filesystem.storageClass.pool }}

deploy/charts/rook-ceph-cluster/templates/volumesnapshotclass.yaml

@@ -16,7 +16,11 @@ metadata:
 {{- if $filesystemvsc.annotations }}
 {{ toYaml $filesystemvsc.annotations | indent 4 }}
 {{- end }}
-driver: {{ .Values.operatorNamespace }}.cephfs.csi.ceph.com
+{{- if $root.Values.csiDriverNamePrefix }}
+driver: {{ $root.Values.csiDriverNamePrefix }}.cephfs.csi.ceph.com
+{{- else }}
+driver: {{ $root.Values.operatorNamespace }}.cephfs.csi.ceph.com
+{{- end }}
 parameters:
   clusterID: {{ .Release.Namespace }}
   csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner
@@ -42,7 +46,12 @@ metadata:
 {{- if $blockpoolvsc.annotations }}
 {{ toYaml $blockpoolvsc.annotations | indent 4 }}
 {{- end }}
-driver: {{ .Values.operatorNamespace }}.rbd.csi.ceph.com
+{{- if-}}
+{{- if $root.Values.csiDriverNamePrefix }}
+driver: {{ $root.Values.csiDriverNamePrefix }}.rbd.csi.ceph.com
+{{- else }}
+driver: {{ $root.Values.operatorNamespace }}.rbd.csi.ceph.com
+{{- end }}
 parameters:
   clusterID: {{ .Release.Namespace }}
   csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner

deploy/charts/rook-ceph-cluster/values.yaml

@@ -646,9 +646,6 @@ cephObjectStores:
 # cephECStorageClass also is disabled by default, please remove the comments and set desired values to enable it
 # if cephECBlockPools are uncommented you must remove the comments of cephEcStorageClass as well
 #cephECStorageClass:
-#  name: rook-ceph-block
-#  # Change "rook-ceph" provisioner prefix to match the operator namespace if needed
-#  provisioner: rook-ceph.rbd.csi.ceph.com # driver:namespace:operator
 #  parameters:
 #    # clusterID is the namespace where the rook cluster is running
 #    # If you change this namespace, also change the namespace below where the secret namespaces are defined
@@ -687,3 +684,7 @@ cephObjectStores:
 #    imageFeatures: layering
 #  allowVolumeExpansion: true
 #  reclaimPolicy: Delete
+
+# -- CSI driver name prefix for cephfs, rbd and nfs.
+# @default -- `namespace name where rook-ceph operator is deployed`
+csiDriverNamePrefix:

deploy/charts/rook-ceph/templates/configmap.yaml

@@ -25,6 +25,9 @@ data:
   CSI_ENABLE_OMAP_GENERATOR: {{ .Values.csi.enableOMAPGenerator | quote }}
   CSI_ENABLE_HOST_NETWORK: {{ .Values.csi.enableCSIHostNetwork | quote }}
   CSI_ENABLE_METADATA: {{ .Values.csi.enableMetadata | quote }}
+{{- if .Values.csi.csiDriverNamePrefix }}
+  CSI_DRIVER_NAME_PREFIX: {{ .Values.csi.csiDriverNamePrefix | quote }}
+{{- end }}
 {{- if .Values.csi.pluginPriorityClassName }}
   CSI_PLUGIN_PRIORITY_CLASSNAME: {{ .Values.csi.pluginPriorityClassName | quote }}
 {{- end }}

deploy/charts/rook-ceph/values.yaml

@@ -147,6 +147,10 @@ csi:
   # @default -- `0`
   sidecarLogLevel:
 
+  # -- CSI driver name prefix for cephfs, rbd and nfs.
+  # @default -- `namespace name where rook-ceph operator is deployed`
+  csiDriverNamePrefix:
+
   # -- CSI RBD plugin daemonset update strategy, supported values are OnDelete and RollingUpdate
   # @default -- `RollingUpdate`
   rbdPluginUpdateStrategy:

deploy/examples/operator-openshift.yaml

@@ -614,6 +614,10 @@ data:
   # (Optional) Retry period in seconds the LeaderElector clients should wait between tries of actions. Defaults to 26 seconds.
   # CSI_LEADER_ELECTION_RETRY_PERIOD: "26s"
 
+  # csi driver name prefix for cephfs, rbd and nfs. if not specified, default
+  # will be the namespace name where rook-ceph operator is deployed.
+  # CSI_DRIVER_NAME_PREFIX: "rook-ceph"
+
   # Rook Discover toleration. Will tolerate all taints with all keys.
   # (Optional) Rook Discover tolerations list. Put here list of taints you want to tolerate in YAML format.
   # DISCOVER_TOLERATIONS: |

deploy/examples/operator.yaml

@@ -61,6 +61,10 @@ data:
   # Supported values from 0 to 5. 0 for general useful logs (the default), 5 for trace level verbosity.
   # CSI_SIDECAR_LOG_LEVEL: "0"
 
+  # csi driver name prefix for cephfs, rbd and nfs. if not specified, default
+  # will be the namespace name where rook-ceph operator is deployed.
+  # CSI_DRIVER_NAME_PREFIX: "rook-ceph"
+
   # Set replicas for csi provisioner deployment.
   CSI_PROVISIONER_REPLICAS: "2"
 

pkg/operator/ceph/csi/csi.go

@@ -320,5 +320,8 @@ func (r *ReconcileCSI) setParams(ver *version.Info) error {
 	if strings.EqualFold(k8sutil.GetValue(r.opConfig.Parameters, "CSI_NFS_ATTACH_REQUIRED", "true"), "false") {
 		CSIParam.NFSAttachRequired = false
 	}
+
+	CSIParam.DriverNamePrefix = k8sutil.GetValue(r.opConfig.Parameters, "CSI_DRIVER_NAME_PREFIX", r.opConfig.OperatorNamespace)
+
 	return nil
 }

pkg/operator/ceph/csi/spec.go

@@ -20,6 +20,7 @@ import (
 	"context"
 	_ "embed"
 	"fmt"
+	"strings"
 	"time"
 
 	"github.com/rook/rook/pkg/operator/ceph/cluster/telemetry"
@@ -36,6 +37,7 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/version"
+	"k8s.io/client-go/kubernetes"
 
 	cephcsi "github.com/ceph/ceph-csi/api/deploy/kubernetes"
 )
@@ -263,11 +265,17 @@ const (
 	csiCephFSProvisioner = "csi-cephfsplugin-provisioner"
 	csiNFSProvisioner    = "csi-nfsplugin-provisioner"
 
+	// cephcsi container names
+	csiRBDContainerName    = "csi-rbdplugin"
+	csiCephFSContainerName = "csi-cephfsplugin"
+	csiNFSContainerName    = "csi-nfsplugin"
+
 	RBDDriverShortName    = "rbd"
 	CephFSDriverShortName = "cephfs"
 	NFSDriverShortName    = "nfs"
 	rbdDriverSuffix       = "rbd.csi.ceph.com"
 	cephFSDriverSuffix    = "cephfs.csi.ceph.com"
+	nfsDriverSuffix       = "nfs.csi.ceph.com"
 )
 
 func CSIEnabled() bool {
@@ -307,11 +315,51 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 		Namespace: r.opConfig.OperatorNamespace,
 	}
 
-	tp.DriverNamePrefix = fmt.Sprintf("%s.", r.opConfig.OperatorNamespace)
+	if strings.HasSuffix(tp.DriverNamePrefix, ".") {
+		// As operator is adding a dot at the end of the prefix, we should not
+		// allow the user to add a dot at the end of the prefix. as it will
+		// result in two dots at the end of the prefix. which cases the csi
+		// driver name creation failure
+		return errors.Errorf("driver name prefix %q should not end with a dot", tp.DriverNamePrefix)
+	}
+
+	if EnableRBD {
+		rbdDriverNamePrefix, err := getCSIDriverNamePrefixFromDeployment(r.opManagerContext, r.context.Clientset, r.opConfig.OperatorNamespace, csiRBDProvisioner, "csi-rbdplugin")
+		if err != nil {
+			return err
+		}
+		if rbdDriverNamePrefix != "" && rbdDriverNamePrefix != tp.DriverNamePrefix {
+			return errors.Errorf("rbd driver already exists with prefix %q, cannot use prefix %q", rbdDriverNamePrefix, tp.DriverNamePrefix)
+		}
+	}
+
+	if EnableCephFS {
+		cephFSDriverNamePrefix, err := getCSIDriverNamePrefixFromDeployment(r.opManagerContext, r.context.Clientset, r.opConfig.OperatorNamespace, csiCephFSProvisioner, "csi-cephfsplugin")
+		if err != nil {
+			return err
+		}
+		if cephFSDriverNamePrefix != "" && cephFSDriverNamePrefix != tp.DriverNamePrefix {
+			return errors.Errorf("cephFS driver already exists with prefix %q, cannot use prefix %q", cephFSDriverNamePrefix, tp.DriverNamePrefix)
+		}
+	}
+
+	if EnableNFS {
+		nfsDriverNamePrefix, err := getCSIDriverNamePrefixFromDeployment(r.opManagerContext, r.context.Clientset, r.opConfig.OperatorNamespace, csiNFSProvisioner, "csi-nfsplugin")
+		if err != nil {
+			return err
+		}
+		if nfsDriverNamePrefix != "" && nfsDriverNamePrefix != tp.DriverNamePrefix {
+			return errors.Errorf("nfs driver already exists with prefix %q, cannot use prefix %q", nfsDriverNamePrefix, tp.DriverNamePrefix)
+		}
+	}
+
+	// Add a dot at the end of the prefix for having the driver name prefix
+	// with format <prefix>.<driver-name>
+	tp.DriverNamePrefix = fmt.Sprintf("%s.", tp.DriverNamePrefix)
 
 	CephFSDriverName = tp.DriverNamePrefix + cephFSDriverSuffix
 	RBDDriverName = tp.DriverNamePrefix + rbdDriverSuffix
-	NFSDriverName = tp.DriverNamePrefix + "nfs.csi.ceph.com"
+	NFSDriverName = tp.DriverNamePrefix + nfsDriverSuffix
 
 	tp.Param.MountCustomCephConf = CustomCSICephConfigExists
 
@@ -935,3 +983,39 @@ func GenerateNetNamespaceFilePath(ctx context.Context, client client.Client, clu
 func generateNetNamespaceFilePath(kubeletDirPath, driverFullName, clusterNamespace string) string {
 	return fmt.Sprintf("%s/plugins/%s/%s.net.ns", kubeletDirPath, driverFullName, clusterNamespace)
 }
+
+func getCSIDriverNamePrefixFromDeployment(ctx context.Context, clientset kubernetes.Interface, namespace, deploymentName, containerName string) (string, error) {
+	deployment, err := clientset.AppsV1().Deployments(namespace).Get(ctx, deploymentName, metav1.GetOptions{})
+	if kerrors.IsNotFound(err) {
+		return "", nil
+	}
+	if err != nil {
+		return "", errors.Wrapf(err, "failed to get deployment %q", deploymentName)
+	}
+
+	for _, container := range deployment.Spec.Template.Spec.Containers {
+		if container.Name == containerName {
+			for _, arg := range container.Args {
+				if prefix, ok := getPrefixFromArg(arg); ok {
+					return prefix, nil
+				}
+			}
+		}
+	}
+
+	return "", errors.Errorf("failed to get CSI driver name from deployment %q", deploymentName)
+}
+
+func getPrefixFromArg(arg string) (string, bool) {
+	if strings.Contains(arg, "--drivername=") {
+		driverName := strings.Split(arg, "=")[1]
+
+		for _, suffix := range []string{rbdDriverSuffix, cephFSDriverSuffix, nfsDriverSuffix} {
+			// Add a dot as we are adding it to the Prefix
+			if prefix, ok := strings.CutSuffix(driverName, "."+suffix); ok {
+				return prefix, true
+			}
+		}
+	}
+	return "", false
+}