Problem stale mount point after upgrading

[jackchuong]

Hi all,
In my environment I have 4 VM in k8s cluster build by k3s

kubectl get nodes --show-labels
NAME             STATUS   ROLES                  AGE      VERSION        LABELS
k3s-dc-worker1   Ready    <none>                 2y236d   v1.28.5+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s-dc-worker1,kubernetes.io/os=linux,longhorn-node=true,node.kubernetes.io/instance-type=k3s
k3s-dc-worker2   Ready    <none>                 2y167d   v1.28.5+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s-dc-worker2,kubernetes.io/os=linux,longhorn-node=true,node.kubernetes.io/instance-type=k3s
k3s-dc-worker3   Ready    <none>                 2y110d   v1.28.5+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s-dc-worker3,kubernetes.io/os=linux,longhorn-node=true,node.kubernetes.io/instance-type=k3s
k3s-dc-master1   Ready    control-plane,master   2y237d   v1.27.5+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s-dc-master1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s

I installed longhorn by helm on k3s-dc-worker1 , k3s-dc-worker2 , k3s-dc-worker3 , mount /mnt/longhorn-sdb on each server

cat longhorn-values.yaml
global:
  nodeSelector:
    longhorn-node: "true"
  timezone: "Asia/Ho_Chi_Minh"

persistence:
  defaultClass: false

ingress:
  enabled: true
  ingressClassName: "nginx"
  host: "longhorn-k8s-local.mydomain.com"
  tls: true
  tlsSecret: "mydomain.com-tls"
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: longhorn-basic-auth
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - Longhorn'

defaultSettings:
  deletingConfirmationFlag: "true"
  v2DataEngine: "true"

I accidentally upgraded longhorn release
helm -n longhorn upgrade longhorn longhorn/longhorn -f longhorn-values.yaml

helm -n longhorn list
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
longhorn        longhorn        3               2026-05-06 10:23:53.459494963 +0700 +07 deployed        longhorn-1.11.1 v1.11.1

A new engineimage appears

kubectl -n longhorn get engineimage
NAME          INCOMPATIBLE   STATE      IMAGE                                          REFCOUNT   BUILDDATE   AGE
ei-3154f3aa   false          deployed   longhornio/longhorn-engine:v1.10.1             95         175d        104d
ei-75a03ec3   false          deployed   docker.io/longhornio/longhorn-engine:v1.11.1   8          53d         178m

since then, I get error if pod (with pvc mount) is assigned to k3s-dc-worker2, for ex:

cat example.yaml
# mysql-deployment.yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  ports:
    - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      nodeSelector:
        #longhorn-node: "true"
        kubernetes.io/hostname: "k3s-dc-worker2"
      restartPolicy: Always
      containers:
        - image: mysql:5.6
          name: mysql
          livenessProbe:
            exec:
              command:
                - ls
                - /var/lib/mysql/lost+found
            initialDelaySeconds: 5
            periodSeconds: 5
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: changeme
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-volume
              mountPath: /var/lib/mysql
        - name: mysql-cron-sidecar
          image: alpine:3.18
          command: ["/bin/sh", "-c"]
          args:
            - |
              apk add --no-cache mysql-client dcron

              echo '5 * * * * mysql -h 127.0.0.1 -uroot -pchangeme -e "USE demo; INSERT INTO timedate (event_time) VALUES (NOW());"' > /etc/crontabs/root


              crond -f
          securityContext:
            runAsUser: 0
      volumes:
        - name: mysql-volume
          persistentVolumeClaim:
            claimName: mysql-pvc

kubectl describe pod/mysql-744fd776f-jwjm4
Name:             mysql-744fd776f-jwjm4
Namespace:        default
Priority:         0
Service Account:  default
Node:             k3s-dc-worker2/192.168.0.202
Start Time:       Wed, 06 May 2026 13:29:50 +0700
Labels:           app=mysql
                  pod-template-hash=744fd776f
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Controlled By:    ReplicaSet/mysql-744fd776f
Containers:
  mysql:
    Container ID:
    Image:          mysql:5.6
    Image ID:
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Liveness:       exec [ls /var/lib/mysql/lost+found] delay=5s timeout=1s period=5s #success=1 #failure=3
    Environment:
      MYSQL_ROOT_PASSWORD:  changeme
    Mounts:
      /var/lib/mysql from mysql-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jlgnp (ro)
  mysql-cron-sidecar:
    Container ID:
    Image:         alpine:3.18
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
    Args:
      apk add --no-cache mysql-client dcron

      echo '5 * * * * mysql -h 127.0.0.1 -uroot -pchangeme -e "USE demo; INSERT INTO timedate (event_time) VALUES (NOW());"' > /etc/crontabs/root


      crond -f

    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jlgnp (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  mysql-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pvc
    ReadOnly:   false
  kube-api-access-jlgnp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/hostname=k3s-dc-worker2
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age        From                     Message
  ----     ------                  ----       ----                     -------
  Warning  FailedScheduling        18s        default-scheduler        0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod..
  Warning  FailedScheduling        16s        default-scheduler        0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod..
  Normal   Scheduled               13s        default-scheduler        Successfully assigned default/mysql-744fd776f-jwjm4 to k3s-dc-worker2
  Normal   SuccessfulAttachVolume  0s         attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-e7dd331a-484f-4c94-ae82-694a6a2684d8"
  Warning  FailedMount             <invalid>  kubelet                  MountVolume.MountDevice failed for volume "pvc-e7dd331a-484f-4c94-ae82-694a6a2684d8" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-e7dd331a-484f-4c94-ae82-694a6a2684d8" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/89c1226061de52b4bde4c06e665854b20480a100d9116239edb5379478f44b8f/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
/dev/longhorn/pvc-e7dd331a-484f-4c94-ae82-694a6a2684d8 is apparently in use by the system; will not make a filesystem here!
)

kubectl get pod,svc,pvc
NAME                        READY   STATUS              RESTARTS        AGE
pod/mysql-744fd776f-jwjm4   0/2     ContainerCreating   0               88s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                        AGE
service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP                        2y237d
service/kubelet      ClusterIP   None         <none>        10250/TCP,10255/TCP,4194/TCP   2y41d
service/mysql        ClusterIP   None         <none>        3306/TCP                       88s

NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mysql-pvc   Bound    pvc-e7dd331a-484f-4c94-ae82-694a6a2684d8   2Gi        RWO            longhorn       88s

kubectl -n longhorn get lhv
NAME                                       DATA ENGINE   STATE      ROBUSTNESS   SCHEDULED   SIZE         NODE             AGE
pvc-f6bd062e-7596-4376-b553-9c1efb607e96   v1            attached   healthy                  2147483648   k3s-dc-worker2   100d
pvc-4be811b0-17d1-4f56-acfe-0f394756b9be   v1            attached   healthy                  2147483648   k3s-dc-worker2   100d
pvc-a1156762-db17-49f8-b129-c724c0af8b04   v1            attached   healthy                  2147483648   k3s-dc-worker3   34d
pvc-fd6a0fcb-8c41-446b-b011-c18ce60adc60   v1            attached   healthy                  2147483648   k3s-dc-worker3   98d
pvc-c646939b-6fc0-47f9-bef9-88847e9a93f3   v1            attached   healthy                  524288000    k3s-dc-worker1   99d
pvc-9ce83ee8-a7af-482b-8505-c43aba0fdef8   v1            attached   healthy                  2147483648   k3s-dc-worker1   99d
pvc-a9c1563b-761a-450f-b28f-9ee2660428a9   v1            attached   healthy                  5368709120   k3s-dc-worker1   99d
pvc-cb3058a2-1ca8-4424-bcbe-45f7d440898c   v1            attached   healthy                  2147483648   k3s-dc-worker2   96d
pvc-64c57515-b09e-4c71-b812-3b11a2c26b80   v1            attached   healthy                  2147483648   k3s-dc-worker3   100d
pvc-7c3c678b-bfd0-4c7e-a904-43ab0e8c09c1   v1            attached   healthy                  1073741824   k3s-dc-worker3   8d
pvc-e94a5bbb-ddbd-413b-8bfe-7745bac6710f   v1            attached   healthy                  2147483648   k3s-dc-worker3   99d
postgres-clone-v2                          v1            detached   unknown                  2147483648                    83m
pvc-075d134d-3903-466c-9fd0-e5bdb234c0c8   v1            attached   healthy                  2147483648   k3s-dc-worker2   97d
pvc-da7b8bc5-b81d-4e2c-ae69-29bd573f65b5   v1            detached   unknown                  2147483648                    96d
pvc-8e8c614f-e0b7-4eb6-8838-eab9c3557cd7   v1            attached   healthy                  2147483648   k3s-dc-worker1   44m
pvc-0fbd22bb-2ec2-4166-bb70-05d26c89406e   v1            attached   healthy                  2147483648   k3s-dc-worker2   97d
pvc-96d0b030-0f30-45fa-be5c-e25570bbb005   v1            attached   healthy                  2147483648   k3s-dc-worker2   98d
pvc-89c796fb-360f-4c1e-a7ec-97ce0a1ee330   v1            attached   healthy                  5368709120   k3s-dc-worker2   100d
pvc-e7dd331a-484f-4c94-ae82-694a6a2684d8   v1            attached   healthy                  2147483648   k3s-dc-worker2   91s
pvc-90fbc81b-9132-4425-a0eb-2635ff15ae5b   v1            attached   healthy                  2147483648   k3s-dc-worker2   97d
pvc-4b5a0556-ed10-4450-9fe7-9565197ac7fd   v1            attached   healthy                  2147483648   k3s-dc-worker1   97d
pvc-2be3a7bd-9ace-4a81-ab4d-a7a6b1b80edf   v1            attached   healthy                  5368709120   k3s-dc-worker1   97d

How can I fix this issue ?
Should I change engineimage of all volumes to docker.io/longhornio/longhorn-engine:v1.11.1 ?

[mantissahz]

Could you check if the multipathd is running?
https://longhorn.io/kb/troubleshooting-volume-with-multipath/

[jackchuong]

I checked the link that you gave

root@k3s-dc-worker2:/home/jack.chuong# ls -lah /dev/longhorn/
total 0
drwxr-xr-x  2 root root    220 May  6 13:29 .
drwxr-xr-x 19 root root   4.6K May  6 13:29 ..
brw-rw----  1 root disk 8,  96 Jan 29 10:37 pvc-075d134d-3903-466c-9fd0-e5bdb234c0c8
brw-rw----  1 root disk 8, 128 Jan 29 12:05 pvc-0fbd22bb-2ec2-4166-bb70-05d26c89406e
brw-rw----  1 root disk 8,  48 Jan 26 14:14 pvc-4be811b0-17d1-4f56-acfe-0f394756b9be
brw-rw----  1 root disk 8,  32 Jan 26 11:46 pvc-89c796fb-360f-4c1e-a7ec-97ce0a1ee330
brw-rw----  1 root disk 8, 112 Jan 29 10:56 pvc-90fbc81b-9132-4425-a0eb-2635ff15ae5b
brw-rw----  1 root disk 8,  80 Jan 28 12:20 pvc-96d0b030-0f30-45fa-be5c-e25570bbb005
brw-rw----  1 root disk 8, 144 Jan 30 14:30 pvc-cb3058a2-1ca8-4424-bcbe-45f7d440898c
brw-rw----  1 root disk 8, 160 May  6 13:29 pvc-e7dd331a-484f-4c94-ae82-694a6a2684d8
brw-rw----  1 root disk 8,  64 Jan 26 14:14 pvc-f6bd062e-7596-4376-b553-9c1efb607e96

root@k3s-dc-worker2:/home/jack.chuong# ls -l /dev | grep "8, 160"
brw-rw---- 1 root disk      8, 160 May  6 13:29 sdk

root@k3s-dc-worker2:/home/jack.chuong# lsof | grep sdk
lsof: no pwd entry for UID 10001
lsof: no pwd entry for UID 10001
lsof: no pwd entry for UID 10001
lsof: no pwd entry for UID 10001

root@k3s-dc-worker2:/home/jack.chuong# ps aux | grep 10001
10001     823308  0.0  0.0    972     4 ?        Ss    2025   0:00 /pause
10001     823354  0.2  0.0 1264676 18708 ?       Ssl   2025 410:43 /usr/bin/loki-canary -addr=loki-gateway.monitoring.svc.cluster.local.:80 -labelname=pod -labelvalue=loki-canary-fc2mk -push=true
10001    1009635  0.0  0.0  61516  8516 ?        Ssl  Jan29   0:00 /opt/mssql/bin/sqlservr
10001    1009723  1.1  2.8 20293932 1410552 ?    Sl   Jan29 1597:45 /opt/mssql/bin/sqlservr
10001    1131041  0.0  0.0   4364   788 ?        Ss   Jan29   0:00 /bin/bash /opt/mssql/bin/launch_sqlservr.sh /opt/mssql/bin/sqlservr
10001    1131089  0.0  0.0  62824 10928 ?        Sl   Jan29   0:00 /opt/mssql/bin/sqlservr
10001    1131114  1.4  7.4 16465520 3688652 ?    Sl   Jan29 1960:58 /opt/mssql/bin/sqlservr
root     1754057  0.0  0.0   3868  1992 pts/1    S+   14:41   0:00 grep --color=auto 10001

root@k3s-dc-worker2:/home/jack.chuong# cat /etc/multipath.conf
defaults {
    user_friendly_names yes
}
root@k3s-dc-worker2:/home/jack.chuong# multipath -t
defaults {
        verbosity 2
        polling_interval 5
        max_polling_interval 20
        reassign_maps "no"
        multipath_dir "//lib/multipath"
        path_selector "service-time 0"
        path_grouping_policy "failover"
        uid_attribute "ID_SERIAL"
        prio "const"
        prio_args ""
        features "0"
        path_checker "tur"
        alias_prefix "mpath"
        failback "manual"
        rr_min_io 1000
        rr_min_io_rq 1
        max_fds "max"
        rr_weight "uniform"
        queue_without_daemon "no"
        allow_usb_devices "no"
        flush_on_last_del "no"
        user_friendly_names "yes"
        fast_io_fail_tmo 5
        bindings_file "/etc/multipath/bindings"
        wwids_file "/etc/multipath/wwids"
        prkeys_file "/etc/multipath/prkeys"
        log_checker_err "always"
        all_tg_pt "no"
        retain_attached_hw_handler "yes"
        detect_prio "yes"
        detect_checker "yes"
        force_sync "yes"
        strict_timing "no"
        deferred_remove "no"
        config_dir "/etc/multipath/conf.d"
        delay_watch_checks "no"
        delay_wait_checks "no"
        san_path_err_threshold "no"
        san_path_err_forget_rate "no"
        san_path_err_recovery_time "no"
        marginal_path_err_sample_time "no"
        marginal_path_err_rate_threshold "no"
        marginal_path_err_recheck_gap_time "no"
        marginal_path_double_failed_time "no"
        find_multipaths "on"
        uxsock_timeout 4000
        retrigger_tries 0
        retrigger_delay 10
        missing_uev_wait_timeout 30
        skip_kpartx "no"
        disable_changed_wwids "ignored"
        remove_retries 0
        ghost_delay "no"
        find_multipaths_timeout -10
        enable_foreign "NONE"
        marginal_pathgroups "no"
        recheck_wwid "no"
}
blacklist {
        devnode "!^(sd[a-z]|dasd[a-z]|nvme[0-9])"
        device {
                vendor "SGI"
                product "Universal Xport"
        }
        device {
                vendor "^DGC"
                product "LUNZ"
        }
        device {
                vendor "EMC"
                product "LUNZ"
        }
        device {
                vendor "DELL"
                product "Universal Xport"
        }
        device {
                vendor "FUJITSU"
                product "Universal Xport"
        }
        device {
                vendor "IBM"
                product "Universal Xport"
        }
        device {
                vendor "IBM"
                product "S/390"
        }
        device {
                vendor "LENOVO"
                product "Universal Xport"
        }
        device {
                vendor "(NETAPP|LSI|ENGENIO)"
                product "Universal Xport"
        }
        device {
                vendor "STK"
                product "Universal Xport"
        }
        device {
                vendor "SUN"
                product "Universal Xport"
        }
        device {
                vendor "(Intel|INTEL)"
                product "VTrak V-LUN"
        }
        device {
                vendor "Promise"
                product "VTrak V-LUN"
        }
        device {
                vendor "Promise"
                product "Vess V-LUN"
        }
}
blacklist_exceptions {
        property "(SCSI_IDENT_|ID_WWN)"
}
devices {
        device {
                vendor "NVME"
                product ".*"
                uid_attribute "ID_WWN"
                path_checker "none"
                retain_attached_hw_handler "no"
        }
        device {
                vendor "APPLE"
                product "Xserve RAID"
                path_grouping_policy "multibus"
        }
        device {
                vendor "3PARdata"
                product "VV"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry 18
                fast_io_fail_tmo 10
                dev_loss_tmo "infinity"
                vpd_vendor "hp3par"
        }
        device {
                vendor "DEC"
                product "HSG80"
                path_grouping_policy "group_by_prio"
                path_checker "hp_sw"
                hardware_handler "1 hp_sw"
                prio "hp_sw"
                no_path_retry "queue"
        }
        device {
                vendor "HP"
                product "A6189A"
                path_grouping_policy "multibus"
                no_path_retry 12
        }
        device {
                vendor "(COMPAQ|HP)"
                product "(MSA|HSV)1[01]0"
                path_grouping_policy "group_by_prio"
                path_checker "hp_sw"
                hardware_handler "1 hp_sw"
                prio "hp_sw"
                no_path_retry 12
        }
        device {
                vendor "(COMPAQ|HP)"
                product "MSA VOLUME"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 12
        }
        device {
                vendor "(COMPAQ|HP)"
                product "(HSV1[01]1|HSV2[01]0|HSV3[046]0|HSV4[05]0)"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 12
        }
        device {
                vendor "HP"
                product "(MSA2[02]12fc|MSA2012i)"
                path_grouping_policy "multibus"
                no_path_retry 18
        }
        device {
                vendor "HP"
                product "(MSA2012sa|MSA23(12|24)(fc|i|sa)|MSA2000s VOLUME)"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 18
        }
        device {
                vendor "HP"
                product "MSA [12]0[456]0 SA[NS]"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 18
        }
        device {
                vendor "HP"
                product "HSVX700"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry 12
        }
        device {
                vendor "HP"
                product "LOGICAL VOLUME"
                path_grouping_policy "multibus"
                no_path_retry 12
        }
        device {
                vendor "HP"
                product "(P2000 G3 FC|P2000G3 FC/iSCSI|P2000 G3 SAS|P2000 G3 iSCSI)"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 18
        }
        device {
                vendor "LEFTHAND"
                product "(P4000|iSCSIDisk|FCDISK)"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 18
        }
        device {
                vendor "Nimble"
                product "Server"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "SGI"
                product "TP9100"
                path_grouping_policy "multibus"
        }
        device {
                vendor "SGI"
                product "TP9[3457]00"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "SGI"
                product "IS"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "SGI"
                product "^DD[46]A-"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "DDN"
                product "SAN DataDirector"
                path_grouping_policy "multibus"
        }
        device {
                vendor "DDN"
                product "^EF3010"
                path_grouping_policy "multibus"
                no_path_retry 30
        }
        device {
                vendor "DDN"
                product "^(EF3015|S2A|SFA)"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "NEXENTA"
                product "COMSTAR"
                path_grouping_policy "group_by_serial"
                no_path_retry 30
        }
        device {
                vendor "TEGILE"
                product "(ZEBI-(FC|ISCSI)|INTELLIFLASH)"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry 10
        }
        device {
                vendor "EMC"
                product "SYMMETRIX"
                path_grouping_policy "multibus"
                no_path_retry 6
        }
        device {
                vendor "^DGC"
                product "^(RAID|DISK|VRAID)"
                product_blacklist "LUNZ"
                path_grouping_policy "group_by_prio"
                path_checker "emc_clariion"
                hardware_handler "1 emc"
                prio "emc"
                failback "immediate"
                no_path_retry 60
        }
        device {
                vendor "EMC"
                product "Invista"
                product_blacklist "LUNZ"
                path_grouping_policy "multibus"
                no_path_retry 5
        }
        device {
                vendor "XtremIO"
                product "XtremApp"
                path_grouping_policy "multibus"
        }
        device {
                vendor "COMPELNT"
                product "Compellent Vol"
                path_grouping_policy "group_by_prio"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "DELL"
                product "^MD3"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "NVME"
                product "^EMC PowerMax_"
                path_grouping_policy "multibus"
        }
        device {
                vendor "DellEMC"
                product "PowerStore"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry 3
                fast_io_fail_tmo 15
        }
        device {
                vendor "DellEMC"
                product "ME4"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
        }
        device {
                vendor "FSC"
                product "CentricStor"
                path_grouping_policy "group_by_serial"
        }
        device {
                vendor "FUJITSU"
                product "ETERNUS_DX(H|L|M|400|8000)"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 10
        }
        device {
                vendor "(EUROLOGC|EuroLogc)"
                product "FC2502"
                path_grouping_policy "multibus"
        }
        device {
                vendor "FUJITSU"
                product "E[234]000"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 10
        }
        device {
                vendor "FUJITSU"
                product "E[68]000"
                path_grouping_policy "multibus"
                no_path_retry 10
        }
        device {
                vendor "FUJITSU"
                product "ETERNUS_AHB"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "(HITACHI|HP)"
                product "^OPEN-"
                path_grouping_policy "multibus"
        }
        device {
                vendor "HITACHI"
                product "^DF"
                path_grouping_policy "group_by_prio"
                prio "hds"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "HITACHI"
                product "^DF600F"
                path_grouping_policy "multibus"
        }
        device {
                vendor "IBM"
                product "ProFibre 4000R"
                path_grouping_policy "multibus"
        }
        device {
                vendor "IBM"
                product "^1722-600"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "IBM"
                product "^1724"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "IBM"
                product "^1726"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "IBM"
                product "^1742"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "IBM"
                product "^1746"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "IBM"
                product "^1813"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "IBM"
                product "^1814"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "IBM"
                product "^1815"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "IBM"
                product "^1818"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "IBM"
                product "^3526"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "IBM"
                product "^(3542|3552)"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "IBM"
                product "^2105"
                path_grouping_policy "multibus"
                no_path_retry "queue"
        }
        device {
                vendor "IBM"
                product "^1750500"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "IBM"
                product "^2107900"
                path_grouping_policy "group_by_prio"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "IBM"
                product "^2145"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "IBM"
                product "S/390 DASD ECKD"
                product_blacklist "S/390"
                path_grouping_policy "multibus"
                uid_attribute "ID_UID"
                path_checker "directio"
                no_path_retry "queue"
        }
        device {
                vendor "IBM"
                product "S/390 DASD FBA"
                product_blacklist "S/390"
                path_grouping_policy "multibus"
                uid_attribute "ID_UID"
                path_checker "directio"
                no_path_retry "queue"
        }
        device {
                vendor "IBM"
                product "^IPR"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "IBM"
                product "1820N00"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "(XIV|IBM)"
                product "(NEXTRA|2810XIV)"
                path_grouping_policy "group_by_prio"
                failback 15
                no_path_retry "queue"
        }
        device {
                vendor "(TMS|IBM)"
                product "(RamSan|FlashSystem)"
                path_grouping_policy "multibus"
        }
        device {
                vendor "IBM"
                product "^(DCS9900|2851)"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "AIX"
                product "VDASD"
                path_grouping_policy "multibus"
                no_path_retry 60
        }
        device {
                vendor "IBM"
                product "3303[ ]+NVDISK"
                no_path_retry 60
        }
        device {
                vendor "AIX"
                product "NVDISK"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry 60
        }
        device {
                vendor "LENOVO"
                product "DE_Series"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "NETAPP"
                product "LUN"
                path_grouping_policy "group_by_prio"
                features "2 pg_init_retries 50"
                prio "ontap"
                failback "immediate"
                no_path_retry "queue"
                flush_on_last_del "yes"
                dev_loss_tmo "infinity"
                user_friendly_names "no"
        }
        device {
                vendor "(NETAPP|LSI|ENGENIO)"
                product "INF-01-00"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "SolidFir"
                product "SSD SAN"
                path_grouping_policy "multibus"
                no_path_retry 24
        }
        device {
                vendor "NVME"
                product "^NetApp ONTAP Controller"
                path_grouping_policy "multibus"
                no_path_retry "queue"
        }
        device {
                vendor "NEC"
                product "DISK ARRAY"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
        }
        device {
                vendor "^Pillar"
                product "^Axiom"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
        }
        device {
                vendor "^Oracle"
                product "^Oracle FS"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
        }
        device {
                vendor "STK"
                product "BladeCtlr"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "STK"
                product "OPENstorage"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "STK"
                product "FLEXLINE 380"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "SUN"
                product "StorEdge 3"
                path_grouping_policy "multibus"
        }
        device {
                vendor "SUN"
                product "STK6580_6780"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "SUN"
                product "CSM[12]00_R"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "SUN"
                product "LCSM100_[IEFS]"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "SUN"
                product "SUN_6180"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "SUN"
                product "ArrayStorage"
                product_blacklist "Universal Xport"
                path_grouping_policy "group_by_prio"
                path_checker "rdac"
                features "2 pg_init_retries 50"
                hardware_handler "1 rdac"
                prio "rdac"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "SUN"
                product "(Sun Storage|ZFS Storage|COMSTAR)"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "PIVOT3"
                product "RAIGE VOLUME"
                path_grouping_policy "multibus"
                no_path_retry "queue"
        }
        device {
                vendor "(NexGen|Pivot3)"
                product "(TierStore|vSTAC)"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "(Intel|INTEL)"
                product "Multi-Flex"
                product_blacklist "VTrak V-LUN"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "(LIO-ORG|SUSE)"
                product "RBD"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry 12
        }
        device {
                vendor "DataCore"
                product "SANmelody"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "DataCore"
                product "Virtual Disk"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry "queue"
        }
        device {
                vendor "PURE"
                product "FlashArray"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                fast_io_fail_tmo 10
        }
        device {
                vendor "HUAWEI"
                product "XSG1"
                path_grouping_policy "group_by_prio"
                failback "immediate"
                no_path_retry 15
        }
        device {
                vendor "KOVE"
                product "XPD"
                path_grouping_policy "multibus"
        }
        device {
                vendor "NFINIDAT"
                product "InfiniBox"
                path_grouping_policy "group_by_prio"
                path_selector "round-robin 0"
                prio "alua"
                failback 30
                rr_weight "priorities"
                no_path_retry "fail"
                rr_min_io 1
                rr_min_io_rq 1
                flush_on_last_del "yes"
                fast_io_fail_tmo 15
                dev_loss_tmo 15
        }
        device {
                vendor "KMNRIO"
                product "K2"
                path_grouping_policy "multibus"
        }
        device {
                vendor "NEXSAN"
                product "NXS-B0"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 15
        }
        device {
                vendor "NEXSAN"
                product "SATAB"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 15
        }
        device {
                vendor "Nexsan"
                product "(NestOS|NST5000)"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "VIOLIN"
                product "SAN ARRAY$"
                path_grouping_policy "group_by_serial"
                no_path_retry 30
        }
        device {
                vendor "VIOLIN"
                product "SAN ARRAY ALUA"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "VIOLIN"
                product "CONCERTO ARRAY"
                path_grouping_policy "multibus"
                no_path_retry 30
        }
        device {
                vendor "(XIOTECH|XIOtech)"
                product "ISE"
                path_grouping_policy "multibus"
                no_path_retry 12
        }
        device {
                vendor "(XIOTECH|XIOtech)"
                product "IGLU DISK"
                path_grouping_policy "multibus"
                no_path_retry 30
        }
        device {
                vendor "(XIOTECH|XIOtech)"
                product "Magnitude"
                path_grouping_policy "multibus"
                no_path_retry 30
        }
        device {
                vendor "Vexata"
                product "VX"
                path_grouping_policy "multibus"
                no_path_retry 30
        }
        device {
                vendor "Promise"
                product "VTrak"
                product_blacklist "VTrak V-LUN"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "Promise"
                product "Vess"
                product_blacklist "Vess V-LUN"
                path_grouping_policy "group_by_prio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "^IFT"
                product ".*"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "DotHill"
                product "SANnet"
                path_grouping_policy "multibus"
                no_path_retry 30
        }
        device {
                vendor "DotHill"
                product "R/Evo"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "DotHill"
                product "^DH"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
        device {
                vendor "AStor"
                product "NeoSapphire"
                path_grouping_policy "multibus"
                no_path_retry 30
        }
        device {
                vendor "INSPUR"
                product "MCS"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
        }
        device {
                vendor "MacroSAN"
                product "LU"
                path_grouping_policy "group_by_prio"
                prio "alua"
                failback "immediate"
                no_path_retry 30
        }
}
overrides {
}

systemctl status multipathd.service
● multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-12-24 09:05:32 +07; 4 months 11 days ago
TriggeredBy: ● multipathd.socket
   Main PID: 675 (multipathd)
     Status: "up"
      Tasks: 7
     Memory: 23.0M
        CPU: 14min 35.240s
     CGroup: /system.slice/multipathd.service
             └─675 /sbin/multipathd -d -s

May 06 12:27:07 k3s-dc-worker2 multipathd[675]: mpatha: map flushed
May 06 12:27:07 k3s-dc-worker2 multipathd[675]: mpatha: removed map after removing all paths
May 06 12:27:08 k3s-dc-worker2 multipathd[675]: dm-1: devmap not registered, can't remove
May 06 12:45:14 k3s-dc-worker2 multipathd[675]: mpatha: addmap [0 4194304 multipath 0 0 1 1 service-time 0 1 1 8:160 1]
May 06 12:45:14 k3s-dc-worker2 multipathd[675]: sdk [8:160]: path added to devmap mpatha
May 06 12:46:08 k3s-dc-worker2 multipathd[675]: mpatha: map flushed
May 06 12:46:08 k3s-dc-worker2 multipathd[675]: mpatha: removed map after removing all paths
May 06 12:46:08 k3s-dc-worker2 multipathd[675]: dm-1: devmap not registered, can't remove
May 06 13:29:54 k3s-dc-worker2 multipathd[675]: mpatha: addmap [0 4194304 multipath 0 0 1 1 service-time 0 1 1 8:160 1]
May 06 13:29:54 k3s-dc-worker2 multipathd[675]: sdk [8:160]: path added to devmap mpatha

multipathd is running but I don't think it causes this issue
because if I change nodeSelector of deployment to kubernetes.io/hostname: "k3s-dc-worker1" or kubernetes.io/hostname: "k3s-dc-worker3" , pod will start and running fine

[jackchuong]

Update:
I edited /etc/multipath.conf

cat /etc/multipath.conf
defaults {
    user_friendly_names yes
}
blacklist {
    devnode "^sd[a-z0-9]+"
}

systemctl restart multipathd.service
root@k3s-dc-worker2:/etc# systemctl status multipathd.service
● multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-05-06 14:56:31 +07; 4s ago
TriggeredBy: ● multipathd.socket
    Process: 1797031 ExecStartPre=/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath (code=exited, status=0/SUCCESS)
   Main PID: 1797032 (multipathd)
     Status: "up"
      Tasks: 7
     Memory: 18.5M
        CPU: 64ms
     CGroup: /system.slice/multipathd.service
             └─1797032 /sbin/multipathd -d -s

May 06 14:56:31 k3s-dc-worker2 systemd[1]: Starting Device-Mapper Multipath Device Controller...
May 06 14:56:31 k3s-dc-worker2 multipathd[1797032]: --------start up--------
May 06 14:56:31 k3s-dc-worker2 multipathd[1797032]: read /etc/multipath.conf
May 06 14:56:31 k3s-dc-worker2 multipathd[1797032]: path checkers start up
May 06 14:56:31 k3s-dc-worker2 systemd[1]: Started Device-Mapper Multipath Device Controller.

multipath -t
blacklist {
        devnode "^sd[a-z0-9]+"
        devnode "!^(sd[a-z]|dasd[a-z]|nvme[0-9])"
        device {
                vendor "SGI"
                product "Universal Xport"
        }
...

Now pod can start and running on k3s-dc-worker2 successfully. Thank you very much.