Pod with PVC provisioned by csi-driver-nfs fails to start due to FailedMount with error mkdir /var/snap: read-only file system

[chheller]

What happened:
Using dynamic provisioning, after mounting a PV to an NFS share on my TrueNAS server, pods fail to start with the reason FailedMount and the following error

MountVolume.SetUp failed for volume "pvc-63f0962b-5970-4d1a-b8df-35322e0d5dc7" : rpc error: code = Internal desc = mkdir /var/snap: read-only file system

What you expected to happen:
The pod should be able to start normally
How to reproduce it:
Using a fresh microk8s installation, on Ubuntu server, against an NFS share provisioned with TrueNAS

The TrueNAS NFS share is configured simply with NFSv4 enabled, and NFSv3 ownership model for NFSv4

Commands to install the driver and storage class

# Install NFS CSI
helm upgrade --install -n kube-system csi-driver-nfs csi-driver-nfs/csi-driver-nfs --version v4.4.0 --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet" --set controller.replicas=1 --set feature.enableFSGroupPolicy=true 
kc apply -f ~/k8s/nfs/nfs.storageclass.yaml
kubectl patch storageclass csi-driver-nfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

nfs.storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-driver-nfs
provisioner: nfs.csi.k8s.io
parameters:
  server: 192.168.1.200 # Fill in your own server details
  share: /mnt/Primary/kubernetes/csi-driver-nfs
  subdir: /${pvc.metadata.namespace}/${pvc.metadata.name}
  mountPermissions: '0777'
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - nfsvers=4.1

Using the democratic-csi zfs-nfs csi driver, I do not face these same issues. Notably, however, I do face some issues where the containers lack the permissions to `chown` or `chmod` files or directories in mounted directories. There aren't any specific restrictions on the NAS, I've tried to configure to be as loosely permissioned as possible to prevent these issues. I've tried as many options and configurations as I can think to try with the storage class, pod securityContexts, and NFS settings to no avail. Also worth noting is that I was successfully using the csi-driver-nfs with volume mounts on NAS successfully for the last 6 months or so, but recently took down my cluster and re-installed it from scratch. In doing so, I likely upgraded the version of the csi-driver-nfs which may be responsible, but have not had time to try and reverse engineer which version of the driver I might have been using previously.

Environment:

• CSI Driver version: helm.sh/chart: csi-driver-nfs-v4.4.0
• Kubernetes version:

Client Version: v1.27.5
Kustomize Version: v5.0.1
Server Version: v1.27.5

• OS (e.g. from /etc/os-release): Ubuntu 22.04.3 LTS
• Kernel: Linux homelab 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: Microk8s 1.27.5
• Others
  • TrueNAS 13.0-U5.3

[andyzhangx]

it seems the nfs csi driver does not have permission to mkdir /var/snap

[kaktooss]

I have pretty same setup and facing the same problem

• MicroK8s v1.27.5 revision 5891
• TrueNAS-13.0-U5.3
• CSI Driver version: helm.sh/chart: csi-driver-nfs-v4.4.0

Error message for the pod being created

MountVolume.SetUp failed for volume "pvc-dd295906-c30a-489a-b129-2779783d6a26" : rpc error: code = Internal desc = mkdir /var/snap: read-only file system

Can confirm that other NFS-based persistence implementations, like https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/ do not suffer from this problem, i am able to create PVC+PV with this solution and it mounted on the node under
/var/snap/microk8s/common/var/lib/kubelet/pods/1f78a43a-6fef-4353-b51b-3aeb66e999c8/volumes/kubernetes.io~nfs/pvc-3901c382-754c-427d-b972-c415c6f98afc

[chheller]

it seems the nfs csi driver does not have permission to mkdir /var/snap

Yeah, though that's the error quoted from the digging that I've done that error doesn't make any sense. The pod isn't mounting any internal volumes under /var/snap, so it should/will have permission to that path (if it even exists within the pod). That would mean it's the driver itself trying to mkdir /var/snap on the host machine, but that probably shouldn't happen either. I can confirm that the nfs csi driver is able to make directories on the attached storage, as it makes the root directory following the naming scheme described in the storage class.

if the nfs csi driver is attempting to make any directories under /var/snap and not under directories that belong to microk8s itself or something similar, there'll be obvious permission problems because Snap locks down the filesystem to prevent abuse. I did try opening up the permissions in /var/snap just to see if that changed anything, but it caused my entire cluster to fail.

[chheller]

I was able to install version 4.2.0 and I am now able to mount PVCs to my NAS. I have found file permission issues that seem to appear to stem from the csi-driver-nfs itself, as I can use the same mount command via CLI and have no issues. So the original issue reported appears to be with newer versions of the csi driver

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
csi-driver-nfs  kube-system     5               2023-10-23 04:41:03.869148025 +0000 UTC deployed        csi-driver-nfs-v4.2.0   v4.2.0    

[securitypedant]

I also have the same error...

• microk8s 1.24.17
• csi-provisioner:v3.5.0
• livenessprobe:v2.10.0
• snapshot-controller:v6.2.2

My error is...

MountVolume.SetUp failed for volume "pvc-68e70909-9c25-44bc-933a-c1ddffcf02d9" : rpc error: code = Internal desc = mkdir /var/snap: read-only file system

[chheller]

I was able to fix the immediate error mkdir /var/snap: read-only file system by downgrading to v4.2.0, however I still had issues actually writing not to the NFS but to the intermediate file at /tmp/pvc-*. This was because the official documentation suggests to use the flag --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet", however per the microk8s docs we should actually use --set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet, exluding the linux. prefix. This fixed all issues for me, and it may fix the mkdir /var/snap: read-only file system error in later versions.

[andyzhangx]

--set kubeletDir is the right setting, I have worked out a doc PR to fix this issue: #537