secstorage.nfs.version = NULL results in vers=null mount option and SSVM NFS mount failure

[RosiKyu]

problem

When secstorage.nfs.version is SQL NULL, CloudStack incorrectly generates an NFS mount option vers=null when mounting Secondary Storage on the Secondary Storage VM (SSVM).

This results in mount.nfs failing with a parsing error, even though the configuration description states that NULL should trigger default NFS version negotiation.

Expected Behavior: If secstorage.nfs.version is NULL, CloudStack should omit the vers= mount option entirely and allow the OS to auto-negotiate the NFS version.

Actual Behavior: CloudStack passes vers=null to mount.nfs, which results in a mount failure.

Evidence

1. Configuration value (in DB)

mysql> SELECT name,value,description
    -> FROM configuration
    -> WHERE name RLIKE 'secstorage.*nfs.*version|(^|\\.)nfs\\..*version|(^|\\.)nfs\\.version'
    -> ORDER BY name;
+------------------------+-------+-------------------------------------------------------------------------------------------------------+
| name                   | value | description                                                                                           |
+------------------------+-------+-------------------------------------------------------------------------------------------------------+
| secstorage.nfs.version | NULL  | Enforces specific NFS version when mounting Secondary Storage. If NULL default selection is performed |
+------------------------+-------+-------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

2. SSVM mount command generated by CloudStack

From SSVM logs:

/var/log/cloud.log:192:2026-01-13T07:34:23,658 WARN  [storage.resource.NfsSecondaryStorageResource] (AgentRequest-Handler-7:[]) Execution of process [4146] for command [mount -t nfs -o soft,timeo=133,retrans=2147483647,tcp,acdirmax=0,acdirmin=0,vers=null 10.0.32.4:/acs/secondary/ref-trl-10584-k-Mol9-rositsa-kyuchukova/ref-trl-10584-k-Mol9-rositsa-kyuchukova-sec3 /mnt/SecStorage/6ff0eb2d-ffed-31c8-8f8c-e2c5c1fb2db5 ] failed.

3. Error produced:

/var/log/cloud.log:214:2026-01-13T07:34:23,684 ERROR [storage.resource.NfsSecondaryStorageResource] (AgentRequest-Handler-7:[]) failed to copy filetemplate/tmpl/1/4/0c364cce-8288-3a92-9234-4ffde372aec7.qcow2 com.cloud.utils.exception.CloudRuntimeException: GetRootDir for nfs://10.0.32.4/acs/secondary/ref-trl-10584-k-Mol9-rositsa-kyuchukova/ref-trl-10584-k-Mol9-rositsa-kyuchukova-sec3 failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to mount 10.0.32.4:/acs/secondary/ref-trl-10584-k-Mol9-rositsa-kyuchukova/ref-trl-10584-k-Mol9-rositsa-kyuchukova-sec3 at /mnt/SecStorage/6ff0eb2d-ffed-31c8-8f8c-e2c5c1fb2db5 due to mount.nfs: parsing error on 'vers=' option

4. SSVM environment confirmation:

root@s-4-VM:~# uname -a
Linux s-4-VM 6.1.0-40-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.153-1 (2025-09-20) x86_64 GNU/Linux
root@s-4-VM:~# 
root@s-4-VM:~# 
root@s-4-VM:~# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@s-4-VM:~# 
root@s-4-VM:~# 
root@s-4-VM:~# mount.nfs -V
mount.nfs: (linux nfs-utils 2.6.2)

5. NFS works for the same Secondary Storage export when an invalid vers= option is not supplied

The same NFS export path used for sec3 mounts successfully on the SSVM with negotiated NFSv4.2, as shown by existing mounts:

root@s-4-VM:~# mount | egrep -i 'nfs|sec' || true
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)

10.0.32.4:/acs/secondary/ref-trl-10584-k-Mol9-rositsa-kyuchukova/ref-trl-10584-k-Mol9-rositsa-kyuchukova-sec3 on /mnt/SecStorage/6ff0eb2d-ffed-31c8-8f8c-e2c5c1fb2db5 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=133,retrans=2147483647,sec=sys,clientaddr=10.0.42.148,local_lock=none,addr=10.0.32.4)

Impact

• Adding a new Secondary Storage may fail intermittently or fully
• Template copy operations can fail with misleading “copy failed” errors
• Behavior contradicts documented configuration semantics

versions

Environment

• Apache CloudStack: 4.20.3.0-SNAPSHOT
• Hypervisor: KVM
• Secondary Storage: NFS
• SSVM OS: Debian GNU/Linux 12 (bookworm)
• Kernel: Linux 6.1.0-40-amd64
• nfs-utils: mount.nfs (linux nfs-utils 2.6.2)

The steps to reproduce the bug

1. Deploy Apache CloudStack 4.20.3.0-SNAPSHOT with:

• KVM hypervisor
• NFS-based Secondary Storage
• Secondary Storage VM (SSVM) running Debian 12

2. Ensure the following global configuration is unset (SQL NULL):

mysql> SELECT name,value FROM configuration WHERE name='secstorage.nfs.version';
+------------------------+-------+
| name                   | value |
+------------------------+-------+
| secstorage.nfs.version | NULL  |
+------------------------+-------+
1 row in set (0.00 sec)

3. Add a new Secondary Storage (image store) backed by NFS
   (or trigger an operation that causes the SSVM to mount a Secondary Storage path, e.g. template copy/sync).

4. Observe the SSVM attempting to mount the Secondary Storage (check SSVM logs)

5. Observe that CloudStack generates the following mount command:

mount -t nfs -o soft,timeo=133,retrans=2147483647,tcp,acdirmax=0,acdirmin=0,vers=null \
<nfs-server>:/acs/secondary/...-sec3 \
/mnt/SecStorage/<uuid>

6. Observe the mount failure: mount.nfs: parsing error on 'vers=' option

What to do about it?