Backups not completing since update to 5.33

[Jarli01]

Just filling in the details since I'm not the OP to this.

Error: ENOENT: no such file or directory, open '/var/nfsmounts/---nfsServerPath---/xo-vm-backups/831fcdfb-0304-216a-feec-8f549646288e/vdis/d8b24ab1-2bdf-484d-bd59-ea23bc4d2e9d/37d994ad-36c2-47ad-8242-a1e7db4f4b80/20190112T060003Z.vhd'

From post 12

Seems to kind of possibly be related to #3873

[julien-f]

Please provide the full log (via https://pastebin.com/ for instance) of the backup run, which can be copied from the details window in the Backup NG Overview.

You can send it to julien.fontanet@vates.fr if it contains sensitive data.

[bvessey]

DeltaFail.txt

On a related note, a newly created job completes fine for the initial full and the first delta.

[julien-f]

Thanks for the report I think I figured out what's the issue is, I'll take care of it tomorrow!

[julien-f]

@Jarli01 Do you confirm this bug is fixed?

[Jarli01]

@julien-f I'm not the originator of finding this issue. @Danp2 will have to confirm.

[bnrstnr]

I'm still seeing this same error with my Delta Backups after updating just now.

[bnrstnr]

I created a new backup job and it's working, but the old job fails.

[Danp2]

@bnrstnr What are the retention settings on the new job? Have you hit the limits where it force a merge of the backups?

[bnrstnr]

Edit: I’m sorry, I didn’t see the “new job” part. I will set the retention to 3 and run it manually a couple times tomorrow and let you know.

Old: My retention is set to 7 and I had about 4 pages (~40 days) of successful backups.

I was getting the ENOENT error after about a minute. I believe the last time I had a merge problem it just wouldn’t ever finish the backups.

[bnrstnr]

@bnrstnr What are the retention settings on the new job? Have you hit the limits where it force a merge of the backups?

OK, so once my recently updated system hits retention it stalls. A few of the VMs were skipped because "The job was canceled to protect the VDI chain." But there are 2 that are stuck at the merge stage.

[Danp2]

@Jarli01 @julien-f Can someone reopen this issue?

[Danp2]

Still seeing this behavior after upgrading to 5.34. It's like XO stops receiving "feedback" via XAPI. Here's an example --

The export is stuck at 17% in XO but shows as completed via xe task-list

Restarting the toolstack outside of XO didn't change anything. Restarting the toolstack using XO allowed the job to resume but the backup failed.

@julien-f Let me know if you need any additional details or if there is something else I can try to help isolate the issue.

[julien-f]

Thanks for the info, I don't know why I'm unable to reproduce that.

Please provide me with the following info:

• backup job config
• XenServer/XCP-ng version
• number of disks of the VM and size

[Danp2]

backup job config

Delta backup using smart mode (Running VMs) to an NFS remote

XenServer/XCP-ng version

XCP-ng 7.6 (have not applied the patches released earlier this week)

number of disks of the VM and size

This varies by VM. Most have 1 disk, but I believe there is a Windows VM with 2 disks. Size ranges from 10GB to 256GB

[julien-f]

Thanks, I'll try to reproduce.

[Danp2]

FYI, I built a new XO VM and restored the config from the existing one. Backup is now running and appears to be working properly.

[bnrstnr]

FYI, I built a new XO VM and restored the config from the existing one. Backup is now running and appears to be working properly.

I did the same thing, and everything is working correctly for me too.

[julien-f]

It probably means that there is an error use case that XO cannot sort itself out, I'll try to understand it and fix it 🙂

Thank you all for your feedbacks 👍

[julien-f]

It appears I f*cked up the fix, I'll push a new version in the few minutes.

Sorry for this folks 😖

[julien-f]

That should do it, please keep me posted if that's not the case!