[BUG] Cannot mount XFS PV

[PhanLe1010]

Describe the bug (🐛 if you encounter this issue)

Longhorn CSI plugin failed to mount xfs PV

To Reproduce

1. Deploy the following pod with a PVC provisioned by a SC that has xfs fsType

   kind: StorageClass
   apiVersion: storage.k8s.io/v1
   metadata:
     name: longhorn-xfs
   provisioner: driver.longhorn.io
   allowVolumeExpansion: true
   reclaimPolicy: Delete
   volumeBindingMode: Immediate
   parameters:
     numberOfReplicas: "3"
     staleReplicaTimeout: "2880" # 48 hours in minutes
     fromBackup: ""
     fsType: "xfs"
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: longhorn-simple-pvc
     namespace: default
   spec:
     accessModes:
       - ReadWriteOnce
     storageClassName: longhorn-xfs
     resources:
       requests:
         storage: 1Gi
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: longhorn-simple-pod
     namespace: default
   spec:
     restartPolicy: Always
     containers:
       - name: volume-test
         image: nginx:stable-alpine
         imagePullPolicy: IfNotPresent
         volumeMounts:
           - name: volv
             mountPath: /data
         ports:
           - containerPort: 80
     volumes:
       - name: volv
         persistentVolumeClaim:
           claimName: longhorn-simple-pvc
   

2. The pod cannot becomes running due to PV failed to mount

     Warning  FailedMount             99s (x9 over 3m48s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-b44a1e7c-29eb-4200-8bf2-6ea066aa1d85" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-b44a1e7c-29eb-4200-8bf2-6ea066aa1d85" failed: type:("xfs") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/d525c060b49d93252a9e9d29487622c4becf1d44d28aa1700389a75736e64876/globalmount") options:("defaults") errcode:(executable file not found in $PATH) output:()
   

Expected behavior

pod is able to become running

Environment

• Longhorn version: master-head 11/17/2023

[innobead]

Is this related to the recent commit for reducing image size?

cc @ChanYiLin

[PhanLe1010]

Looks like there is no mkfs.xfs command inside longhorn-csi-plugin pod

longhorn-csi-plugin-9sgss:/ # mkfs.xfs 
bash: mkfs.xfs: command not found

This lead to the error above
Warning FailedMount 99s (x9 over 3m48s) kubelet MountVolume.MountDevice failed for volume "pvc-b44a1e7c-29eb-4200-8bf2-6ea066aa1d85" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-b44a1e7c-29eb-4200-8bf2-6ea066aa1d85" failed: type:("xfs") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/d525c060b49d93252a9e9d29487622c4becf1d44d28aa1700389a75736e64876/globalmount") options:("defaults") errcode:(executable file not found in $PATH) output:()

Which could be the side effect of the PR longhorn/longhorn-manager#2299

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: issue description

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at Fix bug cannot mount xfs PV longhorn-manager#2302

[roger-ryao]

Our regression test includes a case covering this issue at https://ci.longhorn.io/job/public/job/master/job/sles/job/amd64/job/longhorn-tests-sles-amd64/724/testReport/junit/tests/test_csi/test_xfs_pv/. We can verify this issue through our regression testing.

[roger-ryao]

Verified on master-head 20231123

• longhorn master-head 57bf3ed
• longhorn-manager master-head longhorn/longhorn-manager@3a66afa

Result Passed

https://ci.longhorn.io/job/public/job/master/job/sles/job/amd64/job/longhorn-tests-sles-amd64/730/testReport/tests/test_csi/test_xfs_pv/