Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal Server Error: Can't import exported XVA via xo-cli #6154

Closed
orangebutblue opened this issue Mar 23, 2022 · 15 comments
Closed

Internal Server Error: Can't import exported XVA via xo-cli #6154

orangebutblue opened this issue Mar 23, 2022 · 15 comments

Comments

@orangebutblue
Copy link

I'm able to export VMs via xo-cli, but whenever I try to import them again, I get an 'internal server error'

XCP-ng 8.2.0
xo-server 5.73.0
xo-cli 0.13

I use the web-ui to Import/Export smaller VMs, but I have some VMs with 100-200GB drives, which isn't really feasible to import/export via web-browser.
That's why I'm trying to use xo-cli instead.

Exporting works fine:

root@myserver:~# xo-cli vm.export vm=my-vm-uuid @=/media/export/vm.xva
2.83 MiB @ 1.88 MiB/s
19.32 MiB @ 7.73 MiB/s
...

Importing doesn't work:

root@myserver:~# xo-cli vm.import sr=my-sr-uuid @=/media/export/vm.xva
✖ Internal Server Error
Error: Internal Server Error
    at onResponse (/usr/local/share/.config/yarn/global/node_modules/http-request-plus/index.js:255:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

One time, I've gotten this error trying to import, but I'm not able to reproduce it:

✖ write EPIPE
Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:817:11)
    at Socket._write (node:net:829:8)
    at doWrite (node:internal/streams/writable:409:12)
    at clearBuffer (node:internal/streams/writable:570:7)
    at Socket.Writable.uncork (node:internal/streams/writable:349:7)
    at connectionCorkNT (node:_http_outgoing:797:8)
    at processTicksAndRejections (node:internal/process/task_queues:82:21)

Here are the xo-server logs when I try to import:

Mar 23 10:27:17 myserver xo-server[849]: 2022-03-23T09:27:17.310Z xo:main INFO + WebSocket connection (::ffff:127.0.0.1)
Mar 23 10:27:17 myserver xo-server[849]: 2022-03-23T09:27:17.357Z xo:main INFO - WebSocket connection (::ffff:127.0.0.1)
Mar 23 10:27:22 myserver xo-server[849]: 2022-03-23T09:27:22.566Z xo:main INFO + WebSocket connection (::ffff:127.0.0.1)
Mar 23 10:27:22 myserver xo-server[849]: 2022-03-23T09:27:22.611Z xo:xo ERROR HTTP request error {
Mar 23 10:27:22 myserver xo-server[849]:   error: UnsupportedMediaTypeError: missing content-type header
Mar 23 10:27:22 myserver xo-server[849]:       at Form.parse (/opt/xen-orchestra/node_modules/multiparty/index.js:162:21)
Mar 23 10:27:22 myserver xo-server[849]:       at /opt/xen-orchestra/packages/xo-server/src/api/vm.js:1262:10
Mar 23 10:27:22 myserver xo-server[849]:       at Promise._execute (/opt/xen-orchestra/node_modules/bluebird/js/release/debuggability.js:384:9)
Mar 23 10:27:22 myserver xo-server[849]:       at Promise._resolveFromExecutor (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:518:18)
Mar 23 10:27:22 myserver xo-server[849]:       at new Promise (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:103:10)
Mar 23 10:27:22 myserver xo-server[849]:       at Xo.call (/opt/xen-orchestra/packages/xo-server/src/api/vm.js:1232:9)
Mar 23 10:27:22 myserver xo-server[849]:       at /opt/xen-orchestra/packages/xo-server/src/xo.js:136:18
Mar 23 10:27:22 myserver xo-server[849]:       at Promise._execute (/opt/xen-orchestra/node_modules/bluebird/js/release/debuggability.js:384:9)
Mar 23 10:27:22 myserver xo-server[849]:       at Promise._resolveFromExecutor (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:518:18)
Mar 23 10:27:22 myserver xo-server[849]:       at new Promise (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:103:10)
Mar 23 10:27:22 myserver xo-server[849]:       at Xo._handleHttpRequest (/opt/xen-orchestra/packages/xo-server/src/xo.js:135:5)
Mar 23 10:27:22 myserver xo-server[849]:       at Layer.handle [as handle_request] (/opt/xen-orchestra/node_modules/express/lib/router/layer.js:95:5)
Mar 23 10:27:22 myserver xo-server[849]:       at trim_prefix (/opt/xen-orchestra/node_modules/express/lib/router/index.js:317:13)
Mar 23 10:27:22 myserver xo-server[849]:       at /opt/xen-orchestra/node_modules/express/lib/router/index.js:284:7
Mar 23 10:27:22 myserver xo-server[849]:       at Function.process_params (/opt/xen-orchestra/node_modules/express/lib/router/index.js:335:12)
Mar 23 10:27:22 myserver xo-server[849]:       at next (/opt/xen-orchestra/node_modules/express/lib/router/index.js:275:10)
Mar 23 10:27:22 myserver xo-server[849]:       at initialize (/opt/xen-orchestra/node_modules/passport/lib/middleware/initialize.js:53:5)
Mar 23 10:27:22 myserver xo-server[849]:       at Layer.handle [as handle_request] (/opt/xen-orchestra/node_modules/express/lib/router/layer.js:95:5)
Mar 23 10:27:22 myserver xo-server[849]:       at trim_prefix (/opt/xen-orchestra/node_modules/express/lib/router/index.js:317:13)
Mar 23 10:27:22 myserver xo-server[849]:       at /opt/xen-orchestra/node_modules/express/lib/router/index.js:284:7
Mar 23 10:27:22 myserver xo-server[849]:       at Function.process_params (/opt/xen-orchestra/node_modules/express/lib/router/index.js:335:12)
Mar 23 10:27:22 myserver xo-server[849]:       at next (/opt/xen-orchestra/node_modules/express/lib/router/index.js:275:10)
Mar 23 10:27:22 myserver xo-server[849]: }
Mar 23 10:27:22 myserver xo-server[849]: 2022-03-23T09:27:22.624Z xo:main INFO - WebSocket connection (::ffff:127.0.0.1)
Mar 23 10:28:00 myserver xo-server[849]: [load-balancer]Execute plans!
Mar 23 10:29:00 myserver xo-server[849]: [load-balancer]Execute plans!
Mar 23 10:30:00 myserver xo-server[849]: [load-balancer]Execute plans!

xo-cli is installed on the xo machine

@orangebutblue
Copy link
Author

Sorry to butt in, but has anyone any guesses about what's wrong or any idea how to circumvent it?

@fbeauchamp
Copy link
Collaborator

fbeauchamp commented Apr 1, 2022

hi , it looks like quite an old version (released on 2020 - 12 29 if I'm correct) , is it possible to upgrade it ?

regards,
Florent

@olivierlambert
Copy link
Member

Yes, current xo-server version is 5.90.1. It's important to check you are fully up to date before reporting an issue (see https://xen-orchestra.com/docs/community.html#report-a-bug )

@4censord
Copy link

4censord commented Apr 3, 2022

I have a very similiar expirience, but while importing disks, not full vms.

If this should be a new issus please tell me and i will make a new one.

System info

  • node: 16.13.2
  • npm: 8.4.0
  • xen-orchestra-upload-ova: 0.1.4
  • xo-server: 5.89.0
  • xo-server-telemetry: 0.5.0
  • xo-server-xoa: 0.12.0
  • xo-web-free: 5.94.0
  • xoa-cli: 0.29.0
  • xoa-updater: 0.40.1

This is the appliance as deployed with the quick deploy option on the xcp-ng hosts.
This is the stable channel.

  • xo-cli v0.13.0
  • node v17.8.0
  • npm 8.5.5
  • Archlinux with kernel 5.17.1

What i am doing

I am trying to import vm disks using xo-cli

xo-cli disk.create name=testdisk size=1G sr=$SR_UUID bootable=true
xo-cli disk.importContent id=$DSIK_UUID @=/path/to/disk_image

This should be essentially the same as with xe:
This is running directly on a host.

xe vdi-create sr-uuid=$SR_UUID name-label=testdisk2 virtual-size=1Gib
xe vdi-import uuid=$VDI_UUID filename=/path/to/disk_image

With xe the import succeeds, and i can add the disk to a vm and everything works.

with xo-cli i get errors, and not always the same.

Most of the time I get this error:

xo-cli disk.importContent id=$DSIK_UUID @=/path/to/disk_image
✖ socket hang up
Error: socket hang up
    at connResetException (node:internal/errors:692:14)
    at Socket.socketOnEnd (node:_http_client:478:23)
    at Socket.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

Sometimes it seems to start, and fail immediately after.

xo-cli disk.importContent id=$DSIK_UUID @=/path/to/disk_image
0% of 578.32 MiB @ 365.71 KiB/s - ETA 26m 58s
✖ write EPIPE
Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:817:11)
    at Socket._write (node:net:829:8)
    at doWrite (node:internal/streams/writable:409:12)
    at clearBuffer (node:internal/streams/writable:570:7)
    at Socket.Writable.uncork (node:internal/streams/writable:349:7)
    at connectionCorkNT (node:_http_outgoing:797:8)
    at processTicksAndRejections (node:internal/process/task_queues:82:21)

And sometimes it looks like this:

xo-cli disk.importContent id=$DSIK_UUID @=/path/to/disk_image
0% of 578.32 MiB @ 426.67 KiB/s - ETA 23m 6s
✖ socket hang up
Error: socket hang up
    at connResetException (node:internal/errors:692:14)
    at Socket.socketOnEnd (node:_http_client:478:23)
    at Socket.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

@4censord
Copy link

4censord commented Apr 3, 2022

For exporting disk it looks like this:

xo-cli disk.exportContent id=$VDI_ID @=testfile
8.45 MiB @ 5.63 MiB/s
**snip**
785.7 MiB @ 4.99 MiB/s
✖ HTTP connection abruptly closed
Error: HTTP connection abruptly closed
    at IncomingMessage.emitAbortedError (/home/daniel/.local/lib/node_modules/xo-cli/node_modules/http-request-plus/index.js:83:19)
    at Object.onceWrapper (node:events:641:28)
    at IncomingMessage.emit (node:events:527:28)
    at IncomingMessage._destroy (node:_http_incoming:179:10)
    at _destroy (node:internal/streams/destroy:109:25)
    at IncomingMessage.destroy (node:internal/streams/destroy:71:5)
    at Socket.socketCloseListener (node:_http_client:414:11)
    at Socket.emit (node:events:539:35)
    at TCP.<anonymous> (node:net:709:12)

The disk size is almost there, and in xoa there is a task visible

[XO] VDI Export $VDI_NAME 99%

tats stuck on 99%

@olivierlambert
Copy link
Member

Same remark than before. Please read carefully my previous post.

@4censord
Copy link

4censord commented Apr 3, 2022

Yes, you are right, I was up to date on stable.
Switching to latest, my versions are now:

  • node: 16.14.2
  • npm: 8.4.0
  • xen-orchestra-upload-ova: 0.1.4
  • xo-server: 5.90.2
  • xo-server-telemetry: 0.5.0
  • xo-server-xoa: 0.13.0
  • xo-web-free: 5.95.0
  • xoa-cli: 0.30.0
  • xoa-updater: 0.42.0

But, it still reproduces

xo-cli disk.create name=testdisk size=1G sr=$SR_UUID bootable=true
xo-cli disk.importContent id=$DSIK_UUID @=/path/to/disk_image

Most of the time i get:

✖ socket hang up
Error: socket hang up
    at connResetException (node:internal/errors:692:14)
    at Socket.socketOnEnd (node:_http_client:478:23)
    at Socket.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

once i got:

✖ read ECONNRESET
Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)

But i didnt get the EPIPE error anymore.

@4censord
Copy link

4censord commented Apr 3, 2022

For exports:

xo-cli disk.exportContent id=$VDI_UUID @=testfile
3.03 MiB @ 1.21 MiB/s
*snip*
782.04 MiB @ 5.15 MiB/s
✖ HTTP connection abruptly closed
Error: HTTP connection abruptly closed
    at IncomingMessage.emitAbortedError (/home/daniel/.local/lib/node_modules/xo-cli/node_modules/http-request-plus/index.js:83:19)
    at Object.onceWrapper (node:events:641:28)
    at IncomingMessage.emit (node:events:527:28)
    at IncomingMessage._destroy (node:_http_incoming:179:10)
    at _destroy (node:internal/streams/destroy:109:25)
    at IncomingMessage.destroy (node:internal/streams/destroy:71:5)
    at Socket.socketCloseListener (node:_http_client:414:11)
    at Socket.emit (node:events:539:35)
    at TCP.<anonymous> (node:net:709:12)

The xoa task is still the same

[XO] VDI Export (grml on xcp-ng1) 99%

@4censord
Copy link

4censord commented Apr 3, 2022

As for trying the latest master, I don't currently have a from the sources setup, but could try to set one up if required.

@olivierlambert
Copy link
Member

If you are on XOA latest, that's enough to me 👍

@fbeauchamp so we can try to reproduce it directly :)

@julien-f
Copy link
Member

julien-f commented Apr 4, 2022

I don't have issue with the following:

  • vm.export
  • vm.import
  • disk.importContent

@4censord
Copy link

4censord commented Apr 4, 2022

I have been testing:

tldr:

vms work for me.

disk exports work, import not.

My setup

2 xcp-ng nodes, xcp-ng1 (pool primary) and xcp-ng2
xcp-ng version is 8.2.1, build release/yangtze/master/58
1 Truenas node for shared storage

1 shared iscsi sr (dba5d73e-5638-96c8-4cb2-3c7fff5687eb), this is the pool default.
1 shared iso nfs sr (191389af-9767-e63d-8a1c-0bc7762fd3a8)

xoa, as deployed with the xcp-ng quick deploy.

all xe commands are run on xcp-ng1
all xo-cli commands are run on my workstation

Versions
  • node: 16.14.2
  • npm: 8.4.0
  • xen-orchestra-upload-ova: 0.1.4
  • xo-server: 5.90.2
  • xo-server-telemetry: 0.5.0
  • xo-server-xoa: 0.13.0
  • xo-web-free: 5.95.0
  • xoa-cli: 0.30.0
  • xoa-updater: 0.42.0

Running the free version.

xo-cli:

  • xo-cli v0.13.0
  • node v17.8.0
  • npm 8.5.5
  • on Archlinux, 5.17.1
  • connected to the hosts via wireguard vpn

Preparation

Creating a zeroed disk.

truncate --size 1G zerodisk
sha256sum zerodisk
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  zerodisk

Creating a vdi in both storages and importing the zerodisk.

# iscsi sr
$ xe vdi-create name-label=zerodisk virtual-size=1073741824 sr-uuid=dba5d73e-5638-96c8-4cb2-3c7fff5687eb 
6393dba6-768d-41c1-8dc2-8f996ef310c1
$ xe vdi-import uuid=6393dba6-768d-41c1-8dc2-8f996ef310c1 filename=zerodisk

# nfs sr
$ xe vdi-create name-label=zerodisk virtual-size=1073741824 sr-uuid=191389af-9767-e63d-8a1c-0bc7762fd3a8 
4e332d42-29e1-4d5e-be2f-8ed0c755bdb7
$ xe vdi-import uuid=4e332d42-29e1-4d5e-be2f-8ed0c755bdb7 filename=zerodisk

Creating a blank vm for testing:

xe vm-install template=Debian\ Bullseye\ 11 new-name-label=testvm sr-uuid=dba5d73e-5638-96c8-4cb2-3c7fff5687eb 
b03c1418-aff1-ffc6-54b5-982fb54097ea

VM tests

VM export

$ xo-cli vm.export vm=b03c1418-aff1-ffc6-54b5-982fb54097ea @=vm-export 
224.5 KiB @ 128.29 KiB/s
1.02 MiB @ 166.88 KiB/s
1.03 MiB @ 100.1 KiB/s
2.02 MiB @ 172.46 KiB/s
2.02 MiB @ 172.46 KiB/s

works.

VM import, iscsi sr

$ xo-cli vm.import sr=dba5d73e-5638-96c8-4cb2-3c7fff5687eb @=vm-export
53% of 2.02 MiB @ 725.33 KiB/s - ETA 1s
100% of 2.02 MiB @ 827.8 KiB/s - ETA 0ms
{"id":0,"jsonrpc":"2.0","result":"9753e5cb-1db8-dee3-8fce-8941f9b53b91"}

works

VM import, iso sr

doesn't sound smart, but to be thorough

xo-cli vm.import sr=191389af-9767-e63d-8a1c-0bc7762fd3a8 @=vm-export
✖ Internal Server Error
Error: Internal Server Error
    at onResponse (/home/daniel/.local/lib/node_modules/xo-cli/node_modules/http-request-plus/index.js:255:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

This Internal Server Error seems expected, but could maybe have a nicer error message

disk tests

disk export, iscsi sr

$ xo-cli disk.exportContent id=6393dba6-768d-41c1-8dc2-8f996ef310c1 @=zerodisk_from_iscsi_sr
2.63 MiB @ 1.75 MiB/s
*snip*
1 GiB @ 5.54 MiB/s

$ sha256sum zerodisk_from_iscsi_sr
8f67afc1e16fbaca0ab0a98c7064c0da0ceffd8f063d8d1601d5d069403c574a zerodisk_from_iscsi_sr

works.

But the sha256sum doesn't match the zerodisk, but maybe i am missing something about the disk format?

disk export, nfs iso sr

$ xo-cli disk.exportContent id=4e332d42-29e1-4d5e-be2f-8ed0c755bdb7 @=zerodisk_from_nfs_iso_sr
1.21 MiB @ 829.09 KiB/s
*snip*
1 GiB @ 3.53 MiB/s
$ sha256sum zerodisk_from_nfs_iso_sr
0af98ff8aa0a1e01114865a81cd10f06e8cb1fb40278bc04c7b5f4728b6c2de0  zerodisk_from_nfs_iso_sr

works.

But the sha256sum doesn't match the zerodisk, and is different from the other one?

disk import

disk import, zerodisk, iscsi sr

$ truncate --size 1GiB zerodisk
$ xo-cli disk.import name=zerodisk_import_xo-cli sr=dba5d73e-5638-96c8-4cb2-3c7fff5687eb type=raw @=zerodisk
✖ Internal Server Error
Error: Internal Server Error
    at onResponse (/home/daniel/.local/lib/node_modules/xo-cli/node_modules/http-request-plus/index.js:255:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

unsure whether thats working as expected

disk import, zerodisk, nfs iso sr

$ xo-cli disk.import name=zerodisk_import_xo-cli sr=191389af-9767-e63d-8a1c-0bc7762fd3a8 type=raw @=zerodisk
✖ Internal Server Error
Error: Internal Server Error
    at onResponse (/home/daniel/.local/lib/node_modules/xo-cli/node_modules/http-request-plus/index.js:255:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

unsure whether thats working as expected

disk import, disk exported from iscsi sr

$ xo-cli disk.import name=zerodisk_import_from_iscsi_sr sr=dba5d73e-5638-96c8-4cb2-3c7fff5687eb type=raw @=zerodisk_from_iscsi_sr
✖ Internal Server Error
Error: Internal Server Error
    at onResponse (/home/daniel/.local/lib/node_modules/xo-cli/node_modules/http-request-plus/index.js:255:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

I'm not sure what the actual format is of those disk is, i didn't find any documentation about that.

disk import, disk exported from nfs iso sr

$ xo-cli disk.import name=zerodisk_import_from_iscsi_sr sr=191389af-9767-e63d-8a1c-0bc7762fd3a8 type=raw @=zerodisk_from_nfs_iso_sr
✖ Internal Server Error
Error: Internal Server Error
    at onResponse (/home/daniel/.local/lib/node_modules/xo-cli/node_modules/http-request-plus/index.js:255:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

I'm not sure what the actual format is of those disk is, i didn't find any documentation about that.

@4censord
Copy link

4censord commented Apr 4, 2022

I will be testing disk.importContent more tomorrow.

@orangebutblue
Copy link
Author

Yes, current xo-server version is 5.90.1. It's important to check you are fully up to date before reporting an issue (see https://xen-orchestra.com/docs/community.html#report-a-bug )

Thanks, I was finally able to upgrade to the most recent version and now I'm able to import without any problems. This task was kind of hijacked, but at least my issue has been resolved. Thank you

@olivierlambert
Copy link
Member

Okay so since we fixed OP's issue, we should probably close 👍

@4censord if the problem persist on your side, please open a new issue, your feedback is obviously welcome 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants