KVM NAS backup: VM remains paused indefinitely when backup job fails

[jmsperu]

Description

When using the NAS backup plugin on KVM, if a backup job fails (e.g. due to backup storage being full or I/O errors on the NFS target), the VM remains indefinitely paused at the hypervisor level. CloudStack marks the backup as Error but does not resume the VM, leaving it unresponsive until manually resumed via virsh resume.

Steps to Reproduce

1. Configure NAS backup with NFS storage for a running KVM VM
2. Fill up the NFS backup storage to 100% capacity
3. Wait for the scheduled backup to trigger
4. Observe the VM becomes paused and never resumes

Expected Behavior

The VM should be automatically resumed after a backup failure. The backup should be marked as failed, but the VM should continue running normally.

Actual Behavior

The VM remains in a paused state indefinitely. The backup monitoring loop in nasbackup.sh enters an infinite cycle:

1. virsh backup-begin pauses the QEMU domain for consistent snapshot
2. Backup write fails (storage full)
3. domjobinfo reports Failed status
4. cleanup() is called but does not resume the VM
5. No exit statement after cleanup — loop continues, repeatedly detecting the failed job

Root Cause Analysis

Three bugs in scripts/vm/hypervisor/kvm/nasbackup.sh:

Bug 1: Missing exit after failed backup cleanup (line 144)

case "$status" in
  Failed)
    echo "Virsh backup job failed"
    cleanup ;;   # <-- no exit, falls through to sleep and loops forever
esac

Bug 2: cleanup() never resumes the VM (line 222)

The cleanup() function only removes files and unmounts storage. It never checks if the VM is paused or attempts to resume it, even though virsh backup-begin may have paused the domain.

Bug 3: Missing exit in backup_stopped_vm() (line 181)

Similar to Bug 1, backup_stopped_vm() calls cleanup() on qemu-img convert failure but does not exit, allowing the loop to continue processing subsequent disks.

Impact

• Production outage: All services on the affected VM become unresponsive
• Cascading failures: When backup storage fills up, ALL VMs being backed up get paused simultaneously
• Silent failure: CloudStack UI shows the VM as "Running" while it is actually paused at the KVM level
• No automatic recovery: Manual intervention (virsh resume) is required per VM

In our environment, NFS backup storage filling to 100% caused 8 production VMs to become paused simultaneously across 3 KVM hosts, with some VMs remaining paused for over 6 hours before detection.

Environment

• CloudStack 4.19/4.20/main (code is unchanged across versions)
• KVM hypervisor
• NAS backup plugin with NFS storage
• File: scripts/vm/hypervisor/kvm/nasbackup.sh

Proposed Fix

PR forthcoming with the following changes:

1. Add VM state check and virsh resume to cleanup() function
2. Add missing exit 1 after cleanup() in the Failed backup job case
3. Add missing exit 1 after cleanup() in backup_stopped_vm() on qemu-img convert failure

[github-actions]

🏷️ Issue Triaged

Hi @jmsperu! I've categorized this issue as bug based on the following analysis:

Reasoning: This is a well-documented bug report with clear reproduction steps, root cause analysis identifying three specific bugs in nasbackup.sh, and significant production impact (VMs remaining paused indefinitely after backup failures).

View Triage Details

Analysis

• Keywords detected: "bug", "failed", "unexpected behavior", "production outage", "root cause analysis"
• Issue type indicators:
  • Detailed steps to reproduce the issue
  • Clear expected vs. actual behavior comparison
  • Root cause analysis with code references
  • Production impact documentation
  • Proposed fix mentioned
• Confidence: High

Recommended Next Steps

• Review the proposed PR when submitted
• Consider adding automated tests for backup failure scenarios
• Validate the fix addresses all three identified bugs (missing exits and VM resume logic)
• Test with various backup failure scenarios (storage full, I/O errors, network issues)

References: Triage run §23199463464

Generated by Issue Triage Agent

To install this workflow, run gh aw add github/gh-aw/.github/workflows/issue-triage-agent.md@94662b1dee8ce96c876ba9f33b3ab8be32de82a4. View source at https://github.com/github/gh-aw/tree/94662b1dee8ce96c876ba9f33b3ab8be32de82a4/.github/workflows/issue-triage-agent.md.