NAS backup of large VMs fails

[draggeta]

problem

When backing up up larger VMs (the ones we've tested have 2+ TB of storage), scheduled and manual backups eventually fail to NFS storage. The files are written until some sort of timeout seems to happen.

The files are available on the NFS storage but every backup they stop after a certain period of time. A side effect of this is that cloudstack doesn't apply retention to the backups in this case so they keep filling up storage.

Smaller servers (tested until 600 GB backup correctly. There is no firewall in the path. The hypervisors and NFS storage are in the same subnet.

It does seem to be a timeout issue as it seems the management server times out after an hour, but the backup continues running.

2026-08-15 06:04:06,169 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-f0772149]) (logid:34dc33a0) Scheduling 1 backup for VM (VM instance {"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}) for backup schedule (BackupSchedule {"id":6,"schedule":"00:06:7","scheduleType":2,"vmId":1083}) at [2026-08-15T06:00:00+0000].
2026-08-15 06:04:06,607 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b) Seq 10-4578471971175552676: Sending  { Cmd , MgmtId: 90520740805077, via: 10(nc-lab03), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-10-1083-VM","backupPath":"i-10-1083-VM/2026.08.15.06.04.06","backupRepoType":"nfs","backupRepoAddress":"[fd33:9999:1::21]:/mnt/POOL01/Backup/cs-backup","wait":"0","bypassHostMaintenance":"false"}}] }
2026-08-15 06:08:39,558 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-6:[ctx-9f798002]) (logid:1d647320) Trying to sync backups of VM [VM instance {"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}] using backup provider [nas].
2026-08-15 06:13:40,517 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-5:[ctx-55244db9]) (logid:55fc3c21) Trying to sync backups of VM [VM instance {"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}] using backup provider [nas].
2026-08-15 07:04:06,609 WARN  [c.c.a.m.ClusteredAgentAttache] (API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b) Seq 10-4578471971175552676: Timed out on Seq 10-4578471971175552676:  { Cmd , MgmtId: 90520740805077, via: 10(nc-lab03), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-10-1083-VM","backupPath":"i-10-1083-VM/2026.08.15.06.04.06","backupRepoType":"nfs","backupRepoAddress":"[fd33:9999:1::21]:/mnt/POOL01/Backup/cs-backup","wait":"0","bypassHostMaintenance":"false"}}] }
2026-08-15 07:04:06,609 ERROR [o.a.c.b.NASBackupProvider] (API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b) Operation to initiate backup timed out for VM i-10-1083-VM

Looking at the files:

$ ls -la  ./cs-backup/i-10-1083-VM/2026.08.15.06.04.06/
total 996803735
drwxr-xr-x 2 nobody nogroup            8 Aug 15 07:32 .
drwxr-xr-x 4 nobody nogroup            4 Aug 15 09:56 ..
-rw------- 1 nobody nogroup 950586441728 Aug 15 07:32 datadisk.d2538e1e-7166-4e57-ac85-09cdcd97822e.qcow2
-rw-r--r-- 1 nobody nogroup         8060 Aug 15 06:04 domain-config.xml
-rw-r--r-- 1 nobody nogroup          272 Aug 15 06:04 domblklist.xml
-rw-r--r-- 1 nobody nogroup          178 Aug 15 06:04 domiflist.xml
-rw-r--r-- 1 nobody nogroup          421 Aug 15 06:04 dominfo.xml
-rw------- 1 nobody nogroup 261140578304 Aug 15 06:33 root.69d79690-5753-4972-8268-5fd38b791af7.qcow2

versions

Ubuntu 24.04.4
10G networking over IPv6
Truenas NFS storage
Primary storage is Ceph

Other backup solutions for other hypervisors to the same storage have no issues with long running backups.

The steps to reproduce the bug

1. Schedule or run backup
2. Runs for a while
3. Fails, no backups show in UI, data files seen on NFS storage

What to do about it?

No response

[draggeta]

Now that I looked at it more, it does seem related to the setting:

Job cancel threshold minutes (job.cancel.threshold.minutes)
Time (in minutes) for async-jobs to be forcely cancelled if it has been in process for long

The default is 60 minute. I don't know if it's safe to raise this and maybe giving backups their separate timeout value would be better, as I don't want all jobs to have a longer timeout if possible.

EDIT: nvm, changed the setting and started a backup and it still stopped after 60 minutes

[TadiosAbebe]

@draggeta You can add a comma separated key value pairs containing timeouts for different command for the global setting commands.timeout.

For example TakeBackupCommand=7200,RestoreBackupCommand=7200 and restart your management servers. Then you will be able to see that parameter logged on the management server logs when taking or restoring a backup

2026-08-16 05:09:27.883	
2026-08-16 05:09:27,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-1:[ctx-7ceb121d, job-116486, ctx-16299080]) (logid:f5b8d17e) Wait time setting on org.apache.cloudstack.backup.TakeBackupCommand is 7200 seconds

[draggeta]

Wow, didn't even see that option documented somewhere. I'll try it.

[DaanHoogland]

we welcome any improvements to the docs, @draggeta .

[draggeta]

That seems to have worked. Thank you guys.