Cannot get my deployments to work "csi volumes are not allowed to be used" - Openshift

[jonatiao]

Hello,
The following POD manifest works fine, it mounts the secret inside the POD:

---
apiVersion: v1
kind: Pod
metadata:
  name: srv-java-test
  namespace: myns
spec:
  containers:
  - name: srv-java-test
    image: abc/my-java-ms:latest
    ports:
      - containerPort: 8087
        protocol: TCP
    imagePullPolicy: Always
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "mykv"
        nodePublishSecretRef:
          name: secrets-store-creds

But when I try to do the same using deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: srv-java-test
  namespace: myns
  labels:
    app: srv-java-test
spec:
  selector:
    matchLabels:
      app: srv-java-test
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: srv-java-test
    spec:
      containers:
        - name: srv-java-test
          image: abc/my-java-ms:latest
          ports:
            - containerPort: 8087
              protocol: TCP
          imagePullPolicy: Always
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "mykv"
            nodePublishSecretRef:
              name: secrets-store-creds

I get the following error in the replicaset:

0s          Warning   FailedCreate              replicaset/srv-java-test-7877bdd445         Error creating: pods "srv-java-test-7877bdd445-" is forbidden: unable to validate against any security context constraint: [spec.volumes[0]: Invalid value: "csi": csi volumes are not allowed to be used, provider "dynatrace-activegate": Forbidden: not usable by user or serviceaccount, provider "dynatrace-operator": Forbidden: not usable by user or serviceaccount, provider "dynatrace-webhook": Forbidden: not usable by user or serviceaccount, provider "dynatrace-dynakube-oneagent-unprivileged": Forbidden: not usable by user or serviceaccount, provider "dynatrace-dynakube-oneagent-privileged": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "elasticsearch-scc": Forbidden: not usable by user or serviceaccount, provider "log-collector-scc": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "privileged-genevalogging": Forbidden: not usable by user or serviceaccount]

I'm not understanding what I'm doing wrong. Did I forget something during the install?

thx!

[aramase]

Did you follow the documentation here for enabling AKV provider for Secrets Store CSI Driver in openshift?

[jonatiao]

We actually followed this one:
https://mobb.ninja/docs/misc/secrets-store-csi/azure-key-vault/

Update:
I've created an SA called "test" and then gave privileged scc access to it.

oc adm policy add-scc-to-user privileged...

Then I added to the deployment the following:
serviceAccount: test
serviceAccountName: test

Then, the deployment started to work.

I wonder.. there'a anything specific I missed for deployments?
Still, what is odd is the sentence: "POD works but DEPLOYMENTS don't"

Thx!

[jonatiao]

To fix this.. I'm not sure if this is correct, but we added to the scc restricted the volume "csi". Is this a fair assumption of a correction? XD

thx,

[bmaupin]

TL;DR: Upgrade to OpenShift 4.13 if you can. If not, keep reading.

I ran into the same issue. In my case I was creating the pod manually to test, as per the Azure documentation.

What I noticed is that the pod created manually had the node-exporter SCC applied:

$ oc get pod/busybox-secrets-store-inline -o yaml | grep scc
    openshift.io/scc: node-exporter

Whereas the pod created from a Deployment had the restricted-v2 SCC:

$ oc get pod/deleteme-azure-openshift-poc-2-development-6d84747cc-5gmbs -o yaml | grep scc
    openshift.io/scc: restricted-v2

In OpenShift 4.12 (which is what I have right now), the restricted-v2 SCC doesn't have permissions to use CSI volumes. Apparently starting with OpenShift 4.13, they've added CSI volumes to many SCCs, so that's the simplest solution.

You can add the csi volume to one of the SCCs that come with OpenShift, but it's not recommended:

Do not modify the default SCCs. Customizing the default SCCs can lead to issues when some of the platform pods deploy or OpenShift Container Platform is upgraded. Additionally, the default SCC values are reset to the defaults during some cluster upgrades, which discards all customizations to those SCCs.

Instead of modifying the default SCCs, create and modify your own SCCs as needed. For detailed steps, see Creating security context constraints.

(https://docs.openshift.com/container-platform/4.13/authentication/managing-security-context-constraints.html)

You can also add the privileged SCC to your service account but that's going to give it a lot more permissions than just CSI volumes.

I think the recommended solution is:

1. Create a new SCC with the CSI volume included (source)

   ⚠️ If you're using OpenShift < 4.11, replace restricted-v2 with restricted

   1. This will create a new SCC from restricted-v2 with CSI volumes added:

      oc get scc restricted-v2 -ojson | \
          jq 'del(.metadata | .uid,.creationTimestamp,.resourceVersion,.generation)
              | .metadata.name = "restricted-v2-csi-custom"
              | .metadata.annotations["kubernetes.io/description"]="restricted-v2-csi-custom denies access to all host features and requires pods to be run with a UID, and SELinux context that are allocated to the namespace, but allows access to CSI volumes.  This is *NOT* a default SCC."
              | .volumes += ["csi"]' \
          > restricted-v2-csi-custom.json
      

   2. Apply the new SCC

      oc apply -f restricted-v2-csi-custom.json
      

   3. Verify the new SCC was created, e.g.

      $ oc get scc/restricted-v2-csi-custom
      NAME                       PRIV    CAPS                   SELINUX     RUNASUSER        FSGROUP     SUPGROUP   PRIORITY     READONLYROOTFS   VOLUMES
      restricted-v2-csi-custom   false   ["NET_BIND_SERVICE"]   MustRunAs   MustRunAsRange   MustRunAs   RunAsAny   <no value>   false            ["configMap","csi","downwardAPI","emptyDir","persistentVolumeClaim","projected","secret"]
      

2. Add the new SCC to your service account, e.g.

   oc adm add-scc-to-user restricted-v2-csi-custom -z test -n namespace
   

3. Then as you mentioned, add serviceAccountName to your deployments (serviceAccount is deprecated), e.g.

       spec:
         serviceAccountName: test

Alternatively, if you add the SCC to the default account then you won't need to modify your deployment, e.g.

oc adm add-scc-to-user restricted-v2-csi-custom -z default -n namespace