CSI Driver ignores fsGroup and mounts with root as permission

[LuminousPath]

Bug Reporting

Expected Behavior

When I set a pod's securityContext.fsGroup then the mounted volume should mount with that group as owner

Actual Behavior

typically the device gets mounted with root user and group ownership, and any subdirectories are mounted with existing ownership.

Steps to Reproduce the Problem

an example of the base Traefik helm chart deployment.yaml:

# Source: traefik/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-example-traefik
  labels:
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-10.9.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: test-example
  annotations:
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: traefik
      app.kubernetes.io/instance: test-example
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 0
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "9100"
      labels:
        app.kubernetes.io/name: traefik
        helm.sh/chart: traefik-10.9.1
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/instance: test-example
    spec:
      serviceAccountName: test-example-traefik
      terminationGracePeriodSeconds: 60
      hostNetwork: false
      containers:
      - image: "traefik:2.5.6"
        imagePullPolicy: IfNotPresent
        name: test-example-traefik
        resources:
        readinessProbe:
          httpGet:
            path: /ping
            port: 9000
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /ping
            port: 9000
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        ports:
        - name: "metrics"
          containerPort: 9100
          protocol: "TCP"
        - name: "traefik"
          containerPort: 9000
          protocol: "TCP"
        - name: "web"
          containerPort: 8000
          protocol: "TCP"
        - name: "websecure"
          containerPort: 8443
          protocol: "TCP"
        securityContext:
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsGroup: 65532
          runAsNonRoot: true
          runAsUser: 65532
        volumeMounts:
          - name: data
            mountPath: /data
          - name: tmp
            mountPath: /tmp
        args:
          - "--global.checknewversion"
          - "--global.sendanonymoususage"
          - "--entryPoints.metrics.address=:9100/tcp"
          - "--entryPoints.traefik.address=:9000/tcp"
          - "--entryPoints.web.address=:8000/tcp"
          - "--entryPoints.websecure.address=:8443/tcp"
          - "--api.dashboard=true"
          - "--ping=true"
          - "--metrics.prometheus=true"
          - "--metrics.prometheus.entrypoint=metrics"
          - "--providers.kubernetescrd"
          - "--providers.kubernetesingress"
      volumes:
        - name: data
          emptyDir: {}
        - name: tmp
          emptyDir: {}
      securityContext:
        fsGroup: 65532

as you can see, there is a securityContext.fsGroup there at the bottom set to 65532 , usually this would mount the volumes with that user id as the owner, however the resulting permissions looks more like this:

/ $ ls -al /tmp
total 8
drwxrwsrwx    2 root     65532         4096 Jan 11 15:58 .
drwxr-xr-x    1 root     root          4096 Jan 11 15:58 ..
/ $ ls -al /data
total 24
drwxr-xr-x    3 root     root          4096 Jan 11 15:43 .
drwxr-xr-x    1 root     root          4096 Jan 11 15:58 ..
drw-------    2 65532    65532        16384 Jan 11 15:43 lost+found

which is consistent with my findings in that it's an inconsistent application of the folder permissions

Additional Notes

according to the documentation at https://kubernetes-csi.github.io/docs/support-fsgroup.html#supported-modes
the fsType needs to be set in the PV in order for fsGroup settings to apply properly, but it doesn't seem like this is set on PV creation:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: linodebs.csi.linode.com
  creationTimestamp: "2022-01-11T15:43:22Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/linodebs-csi-linode-com
  name: **pvc-name**
  resourceVersion: "60035185"
  uid: **uid**
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: **pv-name**
    namespace: traefik
    resourceVersion: "60035137"
    uid: **uid**
  csi:
    driver: linodebs.csi.linode.com
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1641738510306-8081-linodebs.csi.linode.com
    volumeHandle: **volume-handle**
  persistentVolumeReclaimPolicy: Retain
  storageClassName: linode-block-storage-retain
  volumeMode: Filesystem
status:
  phase: Bound

under csi there should be a an fsType according to the current spec:
https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/persistent-volume-v1/#persistent-volumes

[rombert]

This is reproducible out-of-the-box on LKE. A simple pod with a persistent volume claim fails to run on LKE, but works fine on different clusters

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fsgrouptest
  labels:
    run: fsgrouptest
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: fsgrouptest
  name: fsgrouptest
spec:
  containers:
  - name: alpine
    image: alpine
    name: fsgrouptest
    command:
      - /bin/sh
      - -c
      - "id; touch /data/foo; sleep 3600"
    volumeMounts:
      - mountPath: /data
        name: data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: fsgrouptest
  securityContext:
    runAsUser: 65500
    runAsGroup: 65500
    fsGroup: 65500

The output of the failed container is

uid=65500 gid=65500 groups=65500
touch: /data/foo: Permission denied

On other clusters (obviously, using other PV provisioners) this works by ensuring that the /data directory has the correct ownership:

$ kubectl exec -it fsgrouptest -- stat /data
  File: /data
  Size: 6         	Blocks: 0          IO Block: 1048576 directory
Device: 76h/118d	Inode: 2328287     Links: 1
Access: (0777/drwxrwxrwx)  Uid: (65534/  nobody)   Gid: (65534/  nobody)
Access: 2022-01-22 21:16:25.717588566 +0000
Modify: 2022-01-22 21:16:00.501461518 +0000
Change: 2022-01-22 21:16:00.501461518 +0000

[wlritchi]

This can be fixed by adding a parameter to the StorageClass objects. Unfortunately they're not mutable once created so on existing clusters you'll need to create new storage classes for both of the Linode-provided ones.

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linode-block-storage-fsgroup-fix
provisioner: linodebs.csi.linode.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
  csi.storage.k8s.io/fstype: ext4  # passing this parameter fixes fsGroup

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linode-block-storage-retain-fsgroup-fix
provisioner: linodebs.csi.linode.com
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
  csi.storage.k8s.io/fstype: ext4

For me, using these storage classes has fixed the fsGroup issue, with no changes to the driver itself. I would think that the long term fix is to add the parameter to the StorageClasses deployed with this driver. I'm not sure about the implications of trying to deploy that on an existing cluster, though.

[tchinmai7]

As far as I can tell, this should have been fixed in the newer releases - I do see the csi.fsType set correctly in a PV

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: linodebs.csi.linode.com
  creationTimestamp: "2023-11-09T01:50:28Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/linodebs-csi-linode-com
  name: pvc-1de3585f361b479a
  resourceVersion: "12130"
  uid: bdaa601a-b126-419f-aa4f-1c9ad438f7fc
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: logging-operator-in-cluster-fluentd-buffer-logging-operator-in-cluster-fluentd-0
    namespace: monitoring
    resourceVersion: "11755"
    uid: 1de3585f-361b-479a-b102-0bd0ef2a4b2e
  csi:
    driver: linodebs.csi.linode.com
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1699494494797-8081-linodebs.csi.linode.com
    volumeHandle: 1959918-pvc1de3585f361b479a
  persistentVolumeReclaimPolicy: Retain
  storageClassName: linode-block-storage-retain

I've installed the csi-driver using the helm chart.

Closing the issue for now, feel free to re-open if this is still a problem and we'll investigate.