[Backup Ng deltaBackup] VDI_IO_ERROR(Device I/O errors)

[bchool]

Context

• XO origin: the sources
• Versions:
  • Node: 8.11.3
  • xo-web: 5.22.0
  • xo-server: 5.22.1
  • commit: c9f08a2

Current behavior

During the execution of a "deltabackup" job, the backup of some vms randomly fails with VDI_IO_ERROR(Device I/O errors). See the attached log.

I noticed the following behaviors:

• When a vm has previously failed, it is more likely to fail again in the next execution.
• Some vms fail after a few minutes and others fail when they are almost finishing to complete.
• The number of vms in which it fails makes it impossible to continue using this backup mechanism.
• I hadn't found any errors in the smlog.

Could you please help me to solve this issue?

xo-issue.log

[julien-f]

Which version of XenServer are you using?

[bchool]

XS75E002

[olivierlambert]

This explain the issue. There is something "new" in XS 7.5 that breaks VDI export for backup. See https://bugs.xenserver.org/browse/XSO-873

Please retry with the latest commits on master and tell us if it persists.

edit: what kind of remote are you using? NFS, SMB, local?

[mike59999]

Hello,

i'm getting the same error. We use Local Storage and NFS.
The latest commit didn't fixed the error.

[olivierlambert]

Can you try with only local storage and see if the error persists?

[bchool]

Thank you very much for your quick reply. It is really appreciated.

I have just updated to the commit b07b4e3 and the error persists.
I am using NFS shared repositories only.

[mike59999]

The error also comes when using local storage. Tested with 2 VM's.
If you need additional logfiles i can upload them, just need to now which files.

[olivierlambert]

I think there is a new behavior in XenServer, with stream backpressure. If the remote is slowed down a bit, XenServer will close the export prematurely (which is not the case in previous XS versions).

We'll see if there is a way to fix that.

[Armstrong1992]

• xo-web: 5.22.1
• xo-server: 5.22.0
• xenserver: 7.5.0 Free

Same issue randomly fails with: VDI_IO_ERROR(Device I/O errors)
Using Local SR (LVM) to Local SR (LVM) on second host.

[olivierlambert]

We'll have a branch for you to test and come back to us if our workaround works. Pinging @pdonias for the rest!

[pdonias]

Hi! You may test again using the branch pierre-vdi-io-error-workaround (commit 693aee1).
Please:

• be aware that it may not work
• don't delete any log that we could use to understand what went wrong
• don't mind the tons of characters printed in the console during the export :)

Let me know how it goes!

[bchool]

I’m glad to inform you everything is working fine since the upgrade to commit 693aee1. Thank you very much for your work on this case.
Could you please keep me informed when this fix will be released in a next version?

[pdonias]

Thanks for taking the time to test it, it helps a lot! We'll most likely release it in a few hours. I'll keep you posted here when it's available.

[pdonias]

The patch has been merged and released. You can use the master branch again for your backups. Please open a new issue if you notice anything wrong with it and thanks again for helping us test it.