Backup job failed with Connection Timeout Error

[Videothek]

Are you using XOA or XO from the sources?

XO from the sources

Which release channel?

None

Provide your commit number

e52b1

Describe the bug

When the Backup job either is started automatically or manually, the backup job fails every time with the error: Connect Timeout Error

Which VMs fail is random and diffrent every time the job runs.

This problem appears on all two of my delta-backup jobs since saturday of last week (16.03.2024).

Error message

Connect Timeout Error

This Xen Orchestra syslog error from `/var/log/syslog` seems to be related to the issue:

Mar 26 08:22:21 xen-orchestra xo-server[102437]: 2024-03-26T07:22:21.498Z xo:backups:worker ERROR unhandled error event {
Mar 26 08:22:22 xen-orchestra xo-server[102437]:   error: RequestAbortedError [AbortError]: Request aborted
Mar 26 08:22:22 xen-orchestra xo-server[102437]:       at BodyReadable.destroy (/opt/xo/xo-builds/xen-orchestra-202403200821/node_modules/undici/lib/api/readable.js:48:13)
Mar 26 08:22:22 xen-orchestra xo-server[102437]:       at iterator (/opt/xo/xo-builds/xen-orchestra-202403200821/packages/vhd-lib/createStreamNbd.js:136:18)
Mar 26 08:22:22 xen-orchestra xo-server[102437]:       at iterator.next (<anonymous>)
Mar 26 08:22:22 xen-orchestra xo-server[102437]:       at next (node:internal/streams/from:85:26)
Mar 26 08:22:22 xen-orchestra xo-server[102437]:       at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Mar 26 08:22:22 xen-orchestra xo-server[102437]:     code: 'UND_ERR_ABORTED'
Mar 26 08:22:22 xen-orchestra xo-server[102437]:   }
Mar 26 08:22:22 xen-orchestra xo-server[102437]: }

To reproduce

1. Go to 'Backup'
2. Click on 'Run backup job once'
3. Wait till the backup finishes
4. See that the backup failed
5. Click on 'Click to display last run details'
6. See failed VMs with the before mentioned error

Expected behavior

The backup job is completed successfully and no error is thrown.

Screenshots

Node

18.19.1

Hypervisor

XCP-ng 8.2.1

Additional context

No response

[julien-f]

Hello, please try with Node 18 and keep us posted.

[Videothek]

Hello @julien-f,

thank you for the fast respons.

For the risk of outing myself as a noob, what does the Node and Node-Version reffer to?

I was unsure during the ticket creation and just put in the suggested value of 16.12.1.

[julien-f]

Oh.

xo-server is a JavaScript application using the Node.js engine.

To use XO it you had to install Node.

You should be able to know its version by using node -v, if you need more help, ask on the forum.

[olivierlambert]

@Videothek how did you install it? Because following the official doc, you should have setup yourself Node and such :) (so you should know the current Node version)

https://xen-orchestra.com/docs/installation.html#from-the-sources

[Videothek]

@olivierlambert, @julien-f ok i understand.

I installed it via the XenOrchestraInstallerUpdater skript from Github: https://github.com/ronivay/XenOrchestraInstallerUpdater

I ran node -v on my Xen Orchestra instance and it seems like i am on version 18.19.1.

[Videothek]

I just updated my Xen Orchestra instance to the newest commit after i saw that there was an update available.

I am now on commit version: 583c7.

I ran my failed backup job from yesterday evening manually for all failed VMs and there was one VM still failing.

[olivierlambert]

In the case you are using a 3rd party installer, we suggest to report the issue on the repo of the installer itself (so create an issue at https://github.com/ronivay/XenOrchestraInstallerUpdater ). Because we can't guess what's in each 3rd party installer, we tend to keep this place with bugs that can be reproduced following our official doc (otherwise, we might spend some time chasing unrelated problems).

Anyway, I think your issue might be related to a recent change, that should be solved with the latest commits. You are not the first to report it (already 4 similar issues open and many threads). Please check before creating a new issue, avoiding an overload for our developers :)

[julien-f]

@olivierlambert Nope, it's a different error and I'm investigating it with @fbeauchamp

[Videothek]

@olivierlambert thank you for your advise.

I will do so.

Please believe me when i say, that i checked other issues and that i tried everything i found to fix the issue.

Since i wasn´t able to fix this, i figured that it would be best to open a new ticket.

I am aware that such tickets are creating work for you and your team and that was the last thing i wanted to do.

[olivierlambert]

Don't worry @Videothek your feedback is important. Just trying to rationalize the load for our dev team. But it seems you pointed another issue (as Julien's said) so you did right!

[Videothek]

Ok thank you, so this ticket should stay open?

[olivierlambert]

Yes 👍

[Videothek]

@julien-f do you have any new information about the error?

[fbeauchamp]

hi @Videothek , can you share the full backup log ? the download ico on the top of the backup log popup

[Videothek]

Hey @fbeauchamp,

sure.

2024-04-12T05_15_39.171Z - backup NG.json

This is a backup that failed today

[Videothek]

Hi @fbeauchamp,

do you have any news on this topic?

Was the log helpful?

[Videothek]

Hi @fbeauchamp,

just following up if you have any news on the topic.

The problem is very annoying and i would appreciate if you would have a suggestion on how to fix the issue.

Thank you in advance 😃

[olivierlambert]

Can you reproduce the issue with XOA? That would be important to check (please check on latest release channel)

[Videothek]

I installed XOA and created a similar Test-Backup to my real Backup and it actually works without issues.

[olivierlambert]

Okay so the problem is elsewhere and not in XO code. I'm closing here, but please open an issue to the 3rd party script installation repo or alternatively a thread on our forum so the community can suggest various changes or review your manual installation