Allow pod-overrides of volumes and container-overrides of volumeMounts in Devfiles

[cgruver]

Is your enhancement related to a problem? Please describe

It is not currently possible to add CSI volume mounts to a users workspace.

CSI is not permitted in the SCC used by Dev Spaces.

There is no mechanism in the devfile to specify volumes other than from the cluster storage provisioner

Describe the solution you'd like

Create a mechanism to add CSI volume mounts to a workspace.

The ideal solution is probably through the volumes fields in a devfile, but that might take longer to implement.

A near-term solution is to allow adding volumes to the workspace pod with pod-overrides and volume-mounts to a container with container-overrides

schemaVersion: 2.2.0
attributes:
  controller.devfile.io/storage-type: per-workspace
metadata:
  name: che-demo
  attributes:
    pod-overrides: {"spec": {"volumes": [{"name": "mysecret-volume", "csi": {"driver": "some-csi-driver", "readOnly" : true, "volumeAttributes": {"attrib1": "some-value"}}}]}}
components:
- name: workspace
  attributes:
    container-overrides: {"volumeMounts": [{"name": "mysecret", "mountPath": "/var/my/path", "readOnly": true}]}
  container: 
    image: quay.io/some/image:latest
    etc...

Describe alternatives you've considered

No response

Additional context

No response

[l0rd]

There are currently a couple of options to mount a CSI volume:

1. Che administrator can specify a specific storage class using .spec.devEnvironments.storage.perWorkspaceStrategyPvcConfig.storageClass in the CheCluster CR
2. The developer can pre-create the CSI volume and a corresponding PVC with the label controller.devfile.io/mount-to-devworkspace: "true" and this will be automounted in workspaces.

@cgruver can you confirm that those 2 options don't work for your use case? And that what you are requesting is to add the capability in the Devfile API? For example:

schemaVersion: 2.2.0
metadata:
  name: mydevfile
components:
  - name: mydevfile
    container:
      (...)
      volumeMounts:
        - name: cache
          path: /.cache
  - name: cache
    volume:
      size: 2Gi
+      csi:
+        driver: some-csi-driver

My worry is this will make the Devfile less portable (i.e. it works on my cluster but doesn't on yours).

[cgruver]

@l0rd The use case in question here is not for PVCs, but for things like AWS Secrets Manager or SPIFFE.

The CheCluster CR config will not work, because these CSI volumes will often be workspace specific, and it's possible that there will be a need for volumes from different drivers.

The easiest option is probably to allow them to be injected via pod-overrides and container-overrides

pod-overrides: {"spec": {"volumes": [{"name": "mysecret-volume", "csi": {"driver": "some-csi-driver", "readOnly" : true, "volumeAttributes": {"attrib1": "some-value"}}}]}}
container-overrides: {"volumeMounts": [{"name": "mysecret", "mountPath": "/var/my/path", "readOnly": true}]}

Portability is less of an issue, because in this scenario, the workspace is dependent on the volume anyway. If a given cluster does not have the needed CSI drivers, then the workspace won't work anyway.

[l0rd]

@cgruver ok, I am still not sure if the developer can or not pre-create the volume. And adding this information in the devfile still looks like a hack. However I agree with you that the pod/container-overrides you are trying to apply should work and we need to figure out how to fix it (if that's a problem with the default SCC, then it's configurable and doesn't require any change in the code).

For a better solution to this use case: have you thought about leveraging the cloud-credential-operator? We introduced the support for SA token volume projection to add GCP credentials in workspaces (see GCP workload identity federation) when the cloud-credential-operator is installed and configured in the cluster. The cloud-credential-operator supports AWS out of the box (and I assume that SPIFFE could be supported too).

[dkwon17]

Pod/container overrides for volumes/volumemounts is now possible thanks to devfile/devworkspace-operator#1223

I'm able to add a CSI volume to a devworkspace pod/container by following these steps to create a SharedSecret: https://docs.openshift.com/container-platform/4.14/storage/container_storage_interface/ephemeral-storage-shared-resource-csi-driver-operator.html

Example overrides:

schemaVersion: 2.2.0
attributes:
  controller.devfile.io/storage-type: per-workspace
metadata:
  name: che-demo
  attributes:
    pod-overrides:
      spec:
        volumes:
          - csi:
              driver: csi.sharedresource.openshift.io
              readOnly: true
              volumeAttributes:
                sharedSecret: my-share
            name: my-csi-volume
components:
- name: workspace
  attributes:
    container-overrides:
      volumeMounts:
        - mountPath: /etc/my-csi-volume
           name: my-csi-volume
           readOnly: true
  container: 
    ...

I believe this covers your use case @cgruver ?

[dkwon17]

I have a permissions issue when trying with Che however (my cluster died):

Status

status:
  conditions:
    - lastTransitionTime: '2024-01-31T01:54:49Z'
      message: 'Error creating DevWorkspace deployment: Detected unrecoverable deployment condition: FailedCreate pods "workspace07eac1a5666447ee-6d865d86cb-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: "SETGID": capability may not be added, provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: "SETUID": capability may not be added, provider restricted-v2: .initContainers[0].allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "SETGID": capability may not be added, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "SETUID": capability may not be added, provider restricted-v2: .containers[0].allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed, provider "restricted": Forbidden: not usable by user or serviceaccount, spec.volumes[0]: Invalid value: "csi": csi volumes are not allowed to be used, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]'

[cgruver]

Pod/container overrides for volumes/volumemounts is now possible thanks to devfile/devworkspace-operator#1223

I'm able to add a CSI volume to a devworkspace pod/container by following these steps to create a SharedSecret: https://docs.openshift.com/container-platform/4.14/storage/container_storage_interface/ephemeral-storage-shared-resource-csi-driver-operator.html

Example overrides:

schemaVersion: 2.2.0
attributes:
  controller.devfile.io/storage-type: per-workspace
metadata:
  name: che-demo
  attributes:
    pod-overrides:
      spec:
        volumes:
          - csi:
              driver: csi.sharedresource.openshift.io
              readOnly: true
              volumeAttributes:
                sharedSecret: my-share
            name: my-csi-volume
components:
- name: workspace
  attributes:
    container-overrides:
      volumeMounts:
        - mountPath: /etc/my-csi-volume
           name: my-csi-volume
           readOnly: true
  container: 
    ...

I believe this covers your use case @cgruver ?

Yes! That covers the use case.

[dkwon17]

Mounting the SharedSecret like so: #22751 (comment) is working in the latest Che:

To get it working, I manually added csi in the list of volumes in the container-build scc (the default scc for workspaces).