Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

debian 8 . CPU high "node". not working #16

Closed
txsastre opened this issue Apr 30, 2018 · 37 comments

Comments

Projects
None yet
3 participants
@txsastre
Copy link

commented Apr 30, 2018

Hi, I installed previously on a debian 9 and everything was fine.
I tried on a debian 8 and it does not work, although it says it has finished.
Then I can see that the CPU really working at almost 100%, a task named "node"

any clues ?

thanks.

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

Type '''node -v''' and paste the output.

Node is a required dependency of XO.

@txsastre

This comment has been minimized.

Copy link
Author

commented Apr 30, 2018

v8.11.1
nowadays I reinstaled several times.
tried with debian 9 (it works) also an ubuntu server 16.04 (it works)
the main problem is when trying Continous Replications, they only works with XOA the appliance.

so I investigated the XOA and I saw that it was running on a debian 8, so tried to replicate it, but its not working. Also I tried to create by hand, it worked, but once again unable to do CR.

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

CR should work with XOCE, I know it was working at one time for certain.

I'd have to see what limit you're encountering with CR though to begin.

@txsastre

This comment has been minimized.

Copy link
Author

commented Apr 30, 2018

hi, thanks for your interest.
you can see here the thread about this problem
https://xen-orchestra.com/forum/topic/638/error-continous-replications-5-18-0/34

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

Node v 8 is LTS, which should be fine.

Please post the details as listed in the issue submission form.

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

Are you attempting to have a CR job run from XS to XCP-ng?

@txsastre

This comment has been minimized.

Copy link
Author

commented Apr 30, 2018

Expected Behavior

once used the script, XOCE works great exception of Continous Replicaltion.


Actual Behavior

firts backup (full) is done, but when trying another on the same VM it brings an error and its not done. Also if I try again, another Full backup is done instead an incremental one.

a large discusion and tests have been discused on the XOA forum it can be read here
https://xen-orchestra.com/forum/topic/638/error-continous-replications-5-18-0/34


Confirm XOCE services are running

systemctl status xo-server.service

   Loaded: loaded (/lib/systemd/system/xo-server.service; enabled; vendor preset: enabled)
   Active: active (running) since dl 2018-04-30 12:56:16 CEST; 21min ago
 Main PID: 995 (node)
    Tasks: 16
   Memory: 155.2M
      CPU: 4min 23.878s
   CGroup: /system.slice/xo-server.service
           ├─ 995 /usr/local/bin/node ./bin/xo-server
           └─1186 /usr/local/bin/node /opt/xen-orchestra/node_modules/jest-worker/build/child.js

abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:83:30
abr 30 13:13:04 ubuntu16 xo-server[995]:     at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at _combinedTickCallback (internal/process/next_tick.js:131:7)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at process._tickCallback (internal/process/next_tick.js:180:9)
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.511Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.701Z xo:xapi Deleting VDI S-XO-deb8
abr 30 13:13:05 ubuntu16 xo-server[995]: 2018-04-30T11:13:05.429Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:05 ubuntu16 xo-server[995]: 2018-04-30T11:13:05.537Z xo:xapi Deleting VDI S-XO-deb8
abr 30 13:17:29 ubuntu16 xo-server[995]: 2018-04-30T11:17:29.345Z xo:main + WebSocket connection (::ffff:192.168.222.8)
abr 30 13:17:29 ubuntu16 xo-server[995]: 2018-04-30T11:17:29.642Z xo:perf blocked for 70ms

Provide the output of journalctl logs

journalctl -u xo-server -f -n 50

-- Logs begin at dl 2018-04-30 12:56:11 CEST. --
abr 30 13:07:12 ubuntu16 xo-server[995]: 2018-04-30T11:07:12.265Z xo:perf blocked for 60ms
abr 30 13:07:13 ubuntu16 xo-server[995]: 2018-04-30T11:07:13.571Z xo:perf blocked for 54ms
abr 30 13:07:14 ubuntu16 xo-server[995]: 2018-04-30T11:07:14.133Z xo:perf blocked for 58ms
abr 30 13:07:14 ubuntu16 xo-server[995]: 2018-04-30T11:07:14.598Z xo:perf blocked for 60ms
abr 30 13:07:15 ubuntu16 xo-server[995]: 2018-04-30T11:07:15.469Z xo:perf blocked for 50ms
abr 30 13:07:22 ubuntu16 xo-server[995]: 2018-04-30T11:07:22.643Z xo:perf blocked for 79ms
abr 30 13:07:42 ubuntu16 xo-server[995]: 2018-04-30T11:07:42.086Z xo:perf blocked for 55ms
abr 30 13:07:42 ubuntu16 xo-server[995]: 2018-04-30T11:07:42.683Z xo:perf blocked for 92ms
abr 30 13:07:46 ubuntu16 xo-server[995]: 2018-04-30T11:07:46.760Z xo:perf blocked for 67ms
abr 30 13:07:49 ubuntu16 xo-server[995]: 2018-04-30T11:07:49.109Z xo:perf blocked for 89ms
abr 30 13:07:49 ubuntu16 xo-server[995]: 2018-04-30T11:07:49.901Z xo:perf blocked for 84ms
abr 30 13:07:53 ubuntu16 xo-server[995]: 2018-04-30T11:07:53.662Z xo:perf blocked for 73ms
abr 30 13:07:54 ubuntu16 xo-server[995]: 2018-04-30T11:07:54.773Z xo:perf blocked for 103ms
abr 30 13:07:55 ubuntu16 xo-server[995]: 2018-04-30T11:07:55.257Z xo:perf blocked for 84ms
abr 30 13:12:17 ubuntu16 xo-server[995]: 2018-04-30T11:12:17.998Z xo:perf blocked for 97ms
abr 30 13:12:20 ubuntu16 xo-server[995]: 2018-04-30T11:12:20.209Z xo:perf blocked for 102ms
abr 30 13:13:00 ubuntu16 xo-server[995]: 2018-04-30T11:13:00.220Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:04 ubuntu16 xo-server[995]: acb8d0d6-1b44-3a85-ce8b-032c42c69010
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.081Z xo:xapi Creating VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.310Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:04 ubuntu16 xo-server[995]: Error: missing base VDI (copy of c88d4ade-786f-45a2-a425-0339755459fa)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:1009:16
abr 30 13:13:04 ubuntu16 xo-server[995]:     at Generator.next (<anonymous>)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:477
abr 30 13:13:04 ubuntu16 xo-server[995]:     at new Promise (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:168:7)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:97
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:26
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/src/utils.js:374:18
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseFor.js:17:11
abr 30 13:13:04 ubuntu16 xo-server[995]:     at baseForOwn (/opt/xen-orchestra/node_modules/lodash/_baseForOwn.js:13:20)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseEach.js:17:14
abr 30 13:13:04 ubuntu16 xo-server[995]:     at forEach (/opt/xen-orchestra/node_modules/lodash/forEach.js:38:10)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at map (/opt/xen-orchestra/packages/xo-server/src/utils.js:373:10)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:30
abr 30 13:13:04 ubuntu16 xo-server[995]:     at Generator.next (<anonymous>)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at run (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:66:22)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:83:30
abr 30 13:13:04 ubuntu16 xo-server[995]:     at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at _combinedTickCallback (internal/process/next_tick.js:131:7)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at process._tickCallback (internal/process/next_tick.js:180:9)
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.511Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.701Z xo:xapi Deleting VDI S-XO-deb8
abr 30 13:13:05 ubuntu16 xo-server[995]: 2018-04-30T11:13:05.429Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:05 ubuntu16 xo-server[995]: 2018-04-30T11:13:05.537Z xo:xapi Deleting VDI S-XO-deb8
abr 30 13:17:29 ubuntu16 xo-server[995]: 2018-04-30T11:17:29.345Z xo:main + WebSocket connection (::ffff:192.168.222.8)
abr 30 13:17:29 ubuntu16 xo-server[995]: 2018-04-30T11:17:29.642Z xo:perf blocked for 70ms

Provide answers to these questions:

  • Is this a fresh install or an upgrade?
    fresh install ubuntu 16.04, also tried with debian 8 and 9

  • Version of XOCE server and web interface you're running
    5.18.3

  • What OS is XOCE running on?
    Ubuntu 16.04

  • What method you used to install XOCE (xo_install.sh, manual installation, etc)
    tried this script, and also by hand as the xenorchestra web teachs.

  • What specific XOCE page you're on, and what specific element you're interacting with to trigger the error
    Continous Replication, al the others things I tried work fine.

  • If a stacktrace is provided in the error, include that too.

  • Any errors that appear in your browser's error console.
    in the log i can see a red button with "error" once viewed it , it displays

[{"message":"missing base VDI (copy of c88d4ade-786f-45a2-a425-0339755459fa)","stack":"Error: missing base VDI (copy of c88d4ade-786f-45a2-a425-0339755459fa)\n at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:1009:16\n at Generator.next (<anonymous>)\n at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)\n at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)\n at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:477\n at new Promise (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:168:7)\n at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:97\n at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:26\n at /opt/xen-orchestra/packages/xo-server/src/utils.js:374:18\n at /opt/xen-orchestra/node_modules/lodash/_createBaseFor.js:17:11\n at baseForOwn (/opt/xen-orchestra/node_modules/lodash/_baseForOwn.js:13:20)\n at /opt/xen-orchestra/node_modules/lodash/_createBaseEach.js:17:14\n at forEach (/opt/xen-orchestra/node_modules/lodash/forEach.js:38:10)\n at map (/opt/xen-orchestra/packages/xo-server/src/utils.js:373:10)\n at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:30\n at Generator.next (<anonymous>)\n at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)\n at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)\n at run (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:66:22)\n at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:83:30\n at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)\n at _combinedTickCallback (internal/process/next_tick.js:131:7)\n at process._tickCallback (internal/process/next_tick.js:180:9)"}]```
- Include what you've done so far in the installation, and if you got any error messages along the way.
no, the installation went fine.
- Indicate whether or not you've manually edited any data directly.
i haven't

<!-- Please do not post an issue without answering the related questions above. If you have opened a different issue and already answered these questions, answer them again, once for every ticket. It will be next to impossible for us to help you. -->
@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

Are each of your hypervisor's the same version?

All XS7.1 OR All XCP-ng 7.4

@txsastre

This comment has been minimized.

Copy link
Author

commented Apr 30, 2018

hi, at this moment all of them are XS 7.1

@Danp2

This comment has been minimized.

Copy link
Collaborator

commented Apr 30, 2018

May want to review this thread. It's from several years ago, but may help you diagnose the issue.

How much ram do you have assigned to the XO VM? Have you checked the syslog for these same "process out of memory" errors?

@txsastre

This comment has been minimized.

Copy link
Author

commented Apr 30, 2018

I've setup the new VM XOCE as the XOA, now I added more RAM to the Ubuntu, now its dynamic 8GB-16GB
tried again and fails again.

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

@txsastre does the VM you're attempting to run CR against already have snapshots?

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

@Danp2 based on your link it looks like an unknown issue within XS that caused this issue. Yet what seems odd is that @txsastre is saying that by using XOA the CR job operates without issue.

@Danp2

This comment has been minimized.

Copy link
Collaborator

commented Apr 30, 2018

@Jarli01 There may have been multiple issues back then. One was pre-XS 7.x, so it shouldn't be coming into play here.

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

As a side question, I'm curious if @txsastre has an unhealthy vdi chain (to many replications) like I was encountering a while back.

Here is a link to the issue I created over at XOs GH: vatesfr/xen-orchestra#2047

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

@txsastre Please provide logs from XOA and XOCE running the same settings for a CR job.

We won't be able to troubleshoot otherwise.

@txsastre

This comment has been minimized.

Copy link
Author

commented Apr 30, 2018

hi everyone.
I don't think so about a unhealthy vdi, 1 because on XOA works, 2 because I created new VM from scratch and no, they have no previous snapshots.
Although when trying the test I previously delete them.

Which logs do you want ? where can I take from ?

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

The logs within XOA and XOCE regarding the backup process would likely shed the most light on whatever is occurring here.

@txsastre

This comment has been minimized.

Copy link
Author

commented Apr 30, 2018

yes, but where do I find it ? in the XOA/XOCE WegGUI ? or somewhere in /var/log ?
thanks.

@txsastre

This comment has been minimized.

Copy link
Author

commented Apr 30, 2018

by the way Continuous Replications works for you ?

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

Copy the logging available from within XO and paste it into a code block here.

@txsastre

This comment has been minimized.

Copy link
Author

commented Apr 30, 2018

well I watched /var/log/syslog and I found this. while trying to CR again over a VM (previously had a CR with a full backup)

Apr 30 18:23:34 ubuntu16 xo-server[993]: 2018-04-30T16:23:34.179Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:23:38 ubuntu16 xo-server[993]: 41d170a0-a97e-0ef2-0969-0d6099db5065
Apr 30 18:23:38 ubuntu16 xo-server[993]: 2018-04-30T16:23:38.785Z xo:xapi Creating VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:23:39 ubuntu16 xo-server[993]: 2018-04-30T16:23:39.015Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:23:39 ubuntu16 xo-server[993]: Error: missing base VDI (copy of b6e1f603-4981-4119-922c-e5a2ad67b6ee)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:1009:16
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at Generator.next (<anonymous>)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:477
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at new Promise (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:168:7)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:97
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:26
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/utils.js:374:18
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseFor.js:17:11
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at baseForOwn (/opt/xen-orchestra/node_modules/lodash/_baseForOwn.js:13:20)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseEach.js:17:14
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at forEach (/opt/xen-orchestra/node_modules/lodash/forEach.js:38:10)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at map (/opt/xen-orchestra/packages/xo-server/src/utils.js:373:10)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:30
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at Generator.next (<anonymous>)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at run (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:66:22)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:83:30
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at _combinedTickCallback (internal/process/next_tick.js:131:7)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at process._tickCallback (internal/process/next_tick.js:180:9)
Apr 30 18:23:39 ubuntu16 xo-server[993]: 2018-04-30T16:23:39.214Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:23:39 ubuntu16 xo-server[993]: 2018-04-30T16:23:39.404Z xo:xapi Deleting VDI S-XO-deb8
Apr 30 18:23:40 ubuntu16 xo-server[993]: 2018-04-30T16:23:40.105Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:23:40 ubuntu16 xo-server[993]: 2018-04-30T16:23:40.231Z xo:xapi Deleting VDI S-XO-deb8
@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

Error: missing base VDI (copy of b6e1f603-4981-4119-922c-e5a2ad67b6ee)

Warning
On your target host does this VM ever get created? Does a VM with the same name already exist on the CR target host?

Make sure to have a backup

If so try deleting this copy, and run the job again.

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

What you need to do is watch both your source hypervisor and CR target hypervisor and see if a new VM is created once the CR job is run. If it never gets created this to me would indicate an issue with your hypervisors.

The issue isn't within the installation process, but something outside of it.

@txsastre

This comment has been minimized.

Copy link
Author

commented Apr 30, 2018

ok. repeated with XOA, same initial error. now doing first copy with CR.

Apr 30 12:27:49 xoa xo-server[502]: 2018-04-30T16:27:49.028Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup debianjessi] Debian Jessie 8.0
Apr 30 12:27:52 xoa xo-server[502]: 2018-04-30T16:27:52.931Z xo:xapi Creating VM [XO Backup debianjessi] Debian Jessie 8.0
Apr 30 12:27:53 xoa xo-server[502]: 2018-04-30T16:27:53.177Z xo:xapi Creating VDI S-XO-deb8 on SR-FJ-HBA
Apr 30 12:27:54 xoa xo-server[502]: 2018-04-30T16:27:54.011Z xo:xapi Creating VBD for VDI S-XO-deb8 on VM [Importing…] Debian Jessie 8.0 (20180430T162752Z)
Apr 30 12:27:54 xoa xo-server[502]: 2018-04-30T16:27:54.016Z xo:xapi exporting VDI S-XO-deb8
Apr 30 12:27:54 xoa xo-server[502]: 2018-04-30T16:27:54.022Z xo:xapi Creating VIF for VM [XO Backup debianjessi] Debian Jessie 8.0 on network Pool-wide network associated with eth0
Apr 30 12:28:18 xoa kernel: [  588.708549] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:28:37 xoa xo-server[502]: 2018-04-30T16:28:37.894Z xo:perf blocked for 106ms
Apr 30 12:28:49 xoa kernel: [  618.848779] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:29:19 xoa kernel: [  648.989570] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:29:49 xoa kernel: [  679.124031] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:30:19 xoa kernel: [  709.259781] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:30:37 xoa xo-server[502]: 2018-04-30T16:30:37.311Z xo:perf blocked for 54ms
Apr 30 12:30:49 xoa kernel: [  739.405767] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:31:19 xoa kernel: [  769.540469] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:31:49 xoa kernel: [  799.677714] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 

Doing now a CR delta now. and it works

Apr 30 12:31:59 xoa xo-server[502]: 2018-04-30T16:31:59.405Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup debianjessi] Debian Jessie 8.0
Apr 30 12:32:04 xoa xo-server[502]: fc1a002c-3257-d987-8e53-15953d4c0a62
Apr 30 12:32:04 xoa xo-server[502]: 2018-04-30T16:32:04.021Z xo:xapi Creating VM [XO Backup debianjessi] Debian Jessie 8.0
Apr 30 12:32:04 xoa xo-server[502]: 2018-04-30T16:32:04.238Z xo:xapi Cloning VDI S-XO-deb8
Apr 30 12:32:07 xoa xo-server[502]: 2018-04-30T16:32:07.035Z xo:xapi Creating VBD for VDI S-XO-deb8 on VM [Importing…] Debian Jessie 8.0 (20180430T163204Z)
Apr 30 12:32:07 xoa xo-server[502]: 2018-04-30T16:32:07.038Z xo:xapi exporting VDI S-XO-deb8 (from base S-XO-deb8)
Apr 30 12:32:07 xoa xo-server[502]: 2018-04-30T16:32:07.041Z xo:xapi Creating VIF for VM [XO Backup debianjessi] Debian Jessie 8.0 on network Pool-wide network associated with eth0
Apr 30 12:32:14 xoa xo-server[502]: 2018-04-30T16:32:14.986Z xo:xapi Deleting VM [XO Backup debianjessi] Debian Jessie 8.0
Apr 30 12:32:15 xoa xo-server[502]: 2018-04-30T16:32:15.168Z xo:xapi Deleting VDI S-XO-deb8
Apr 30 12:32:20 xoa kernel: [  829.818490] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:32:50 xoa kernel: [  859.953326] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
@txsastre

This comment has been minimized.

Copy link
Author

commented Apr 30, 2018

now with XOCE
creating firts backup

Apr 30 18:34:29 ubuntu16 xo-server[993]: 2018-04-30T16:34:29.706Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:34:33 ubuntu16 xo-server[993]: 2018-04-30T16:34:33.545Z xo:xapi Creating VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:34:33 ubuntu16 xo-server[993]: 2018-04-30T16:34:33.773Z xo:xapi Creating VDI S-XO-deb8 on SR-FJ-HBA
Apr 30 18:34:34 ubuntu16 xo-server[993]: 2018-04-30T16:34:34.605Z xo:xapi exporting VDI S-XO-deb8
Apr 30 18:34:34 ubuntu16 xo-server[993]: 2018-04-30T16:34:34.608Z xo:xapi Creating VIF for VM [XO Backup cr-debianjessie8] Debian Jessie 8.0 on network Pool-wide network associated with eth0
Apr 30 18:34:42 ubuntu16 xo-server[993]: 2018-04-30T16:34:42.196Z xo:perf blocked for 63ms
Apr 30 18:34:43 ubuntu16 xo-server[993]: 2018-04-30T16:34:43.106Z xo:perf blocked for 68ms
Apr 30 18:34:47 ubuntu16 xo-server[993]: 2018-04-30T16:34:47.138Z xo:perf blocked for 74ms
Apr 30 18:34:48 ubuntu16 xo-server[993]: 2018-04-30T16:34:48.030Z xo:perf blocked for 59ms
... a lot of "xo:perf" here

and now the second copy . Fail

Apr 30 18:38:07 ubuntu16 xo-server[993]: 2018-04-30T16:38:07.388Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:38:11 ubuntu16 xo-server[993]: 1280a1f5-10db-4c6e-42bd-dab8b79924b8
Apr 30 18:38:11 ubuntu16 xo-server[993]: 2018-04-30T16:38:11.878Z xo:xapi Creating VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:38:12 ubuntu16 xo-server[993]: 2018-04-30T16:38:12.108Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:38:12 ubuntu16 xo-server[993]: Error: missing base VDI (copy of 95c17de7-f935-4aa4-873e-dd79b117afbc)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:1009:16
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at Generator.next (<anonymous>)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:477
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at new Promise (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:168:7)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:97
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:26
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/utils.js:374:18
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseFor.js:17:11
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at baseForOwn (/opt/xen-orchestra/node_modules/lodash/_baseForOwn.js:13:20)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseEach.js:17:14
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at forEach (/opt/xen-orchestra/node_modules/lodash/forEach.js:38:10)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at map (/opt/xen-orchestra/packages/xo-server/src/utils.js:373:10)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:30
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at Generator.next (<anonymous>)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at run (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:66:22)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:83:30
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at _combinedTickCallback (internal/process/next_tick.js:131:7)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at process._tickCallback (internal/process/next_tick.js:180:9)
Apr 30 18:38:12 ubuntu16 xo-server[993]: 2018-04-30T16:38:12.311Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:38:12 ubuntu16 xo-server[993]: 2018-04-30T16:38:12.496Z xo:xapi Deleting VDI S-XO-deb8
Apr 30 18:38:13 ubuntu16 xo-server[993]: 2018-04-30T16:38:13.249Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:38:13 ubuntu16 xo-server[993]: 2018-04-30T16:38:13.332Z xo:xapi Deleting VDI S-XO-deb8
@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

I just tested CR between my two systems at my house and everything appears to work just fine for me.

Each host is standalone (so not pooled). Can you provide any other settings you might have configured?

https://i.imgur.com/u373TsM.png

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

Here on purpose I ran the job to fast and the job was stopped to prevent filling the target host (and affecting the VDI chain)

https://i.imgur.com/4sMZvW3.png

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

And here you can see the job ran again, without issue. As well as both of my host (ignore the mess of VM's I have here)

https://i.imgur.com/alO7ZrM.png
https://i.imgur.com/dhhwmhw.png

@Jarli01

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

And here the job is running on it's own every 5 minutes. Stopping automatically where it must to prevent a VDI overrun.

https://i.imgur.com/7SW9rMM.png

@Jarli01

This comment has been minimized.

Copy link
Owner

commented May 1, 2018

And lastly here is a continuous run, without any failed backups.

https://i.imgur.com/Jq35924.png

@txsastre

This comment has been minimized.

Copy link
Author

commented May 1, 2018

First of all, thank you very much for your time.
Second I see that the CR are done from a local storage to another local storage, from one server to another.
I tried and also failed, it has to be the OS of the XOCE, or the VM itself, I think that there is no other explanation.
how is your VM configured ? CPU, RAM, OS , NIC ?
Meanwhile I will create another pool with virtualbox and try again.

@Jarli01

This comment has been minimized.

Copy link
Owner

commented May 1, 2018

My XOCE host host is running on Ubuntu 17.10 with 1Gb of ram and 1 vCPU, with a single NIC (not teamed at the hypervisor - though this is the production approach).

Nothing at all special.

@Jarli01

This comment has been minimized.

Copy link
Owner

commented May 1, 2018

As a side note, I also had this setup and working at a previous client without issue (some version ago) when CR was originally released.

Back then that was on Ubuntu 16.04

@Jarli01

This comment has been minimized.

Copy link
Owner

commented May 1, 2018

And lastly the host I setup last night as the CR target (lazy I know) was installed with XS7.2 and not at all updated last night.

But that shouldn't affect what is occurring here, which is an issue somewhere with the way things are configured.

Can you explain how you have things setup, your Hypervisors, your XOCE installation, switching etc.

@Jarli01

This comment has been minimized.

Copy link
Owner

commented May 1, 2018

What do you mean by "Meanwhile I will create another pool with virtualbox and try again."

XS is a type 1 hypervisor (and by extension so is XCP-ng). Are you creating XS/XCP-ng Hypervisors within VirtualBox and attempting to have XO run on either of these systems and then backup to the other XS/XCP-ng VirtualBox Host?

Attempting to use VirtualBox to act as the hardware for your hypervisor could cause weird issues like this. . .

Please explain how you're hardware is configured.

@txsastre

This comment has been minimized.

Copy link
Author

commented May 1, 2018

hi @Jarli01 again.
I haven't time enough to set up another environment, yes I understand what you saying about virtualbox.
No my pools are a real powerful machines, one set is for producion and the other one intends to be a backup pool.
As I said in the Xen Orchestra forum, I upgrade to 5.19 (its been a few hours since they deployed) with your update script, and now it works fine.
https://xen-orchestra.com/forum/topic/638/error-continous-replications-5-18-0/40

may be they found a bug related with my problem, or they were working in another that solved my situation.

thank you very much for your time and support. :)

@txsastre txsastre closed this May 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.