Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upPCI passthrough not working for HVM domains #1659
Comments
marmarek
added this to the Release 3.1 milestone
Jan 19, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Jan 19, 2016
Member
@rootkovska @mfc I've assigned priority "major" but maybe it deserve some higher?
|
@rootkovska @mfc I've assigned priority "major" but maybe it deserve some higher? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rootkovska
Jan 19, 2016
Member
... or lower, rather? (as we currently we don't support HVM-based net/usb VMs, so this affects very few users)
|
... or lower, rather? (as we currently we don't support HVM-based net/usb VMs, so this affects very few users) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
esheltone
Jan 19, 2016
I would say the affected users fall into two main categories: (1) users trying to get GPU passthrough working for their Windows HVMs, to be able to do real 3D graphics applications, etc., and (2) users wanting to pass through a USB controller, so they can use a webcam or other devices with Windows. I would like to be able to do things like pass through a storage adapter and network adapter directly to a FreeBSD HVM, but that is admittedly an even more specialized use case affecting almost no one else.
I think it would be good to clearly document that passthrough is not available for HVM domains and perhaps also remove the capability from Qubes Manager for HVM domains until this issue is resolved.
esheltone
commented
Jan 19, 2016
|
I would say the affected users fall into two main categories: (1) users trying to get GPU passthrough working for their Windows HVMs, to be able to do real 3D graphics applications, etc., and (2) users wanting to pass through a USB controller, so they can use a webcam or other devices with Windows. I would like to be able to do things like pass through a storage adapter and network adapter directly to a FreeBSD HVM, but that is admittedly an even more specialized use case affecting almost no one else. I think it would be good to clearly document that passthrough is not available for HVM domains and perhaps also remove the capability from Qubes Manager for HVM domains until this issue is resolved. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
esheltone
Jan 25, 2016
Another use case: wanting to be able to have sound output in Windows (currently unavailable by any means): https://groups.google.com/forum/#!topic/qubes-users/BBWF9wguP-0
esheltone
commented
Jan 25, 2016
|
Another use case: wanting to be able to have sound output in Windows (currently unavailable by any means): https://groups.google.com/forum/#!topic/qubes-users/BBWF9wguP-0 |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ideologysec
Feb 1, 2016
Would be great to have PCI passthrough on a dual-GPU laptop, for gaming purposes in Windows without sacrificing the isolation that Qubes provides. (audio via device passthrough is nice but really a working Qubes Windows Tools audio driver for better interaction with the rest of the system and not specialized applications, would be great).
ideologysec
commented
Feb 1, 2016
|
Would be great to have PCI passthrough on a dual-GPU laptop, for gaming purposes in Windows without sacrificing the isolation that Qubes provides. (audio via device passthrough is nice but really a working Qubes Windows Tools audio driver for better interaction with the rest of the system and not specialized applications, would be great). |
rootkovska
modified the milestones:
Release 3.1 updates,
Release 3.1
Feb 12, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
wphowell
Feb 16, 2016
There is actually a 3rd category: disk controllers. It isn't possible to rip disks without the raw device being available to the VM.
wphowell
commented
Feb 16, 2016
|
There is actually a 3rd category: disk controllers. It isn't possible to rip disks without the raw device being available to the VM. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Feb 21, 2016
Member
Some tests reveals that the problem is in stubdomain - the very same config (xl create directly) with device_model_version = 'qemu-xen-traditional' does work, but with device_model_stubdomain_override=1 does not.
Next thing to check: unpatched libxl to rule out our patches breaking it.
|
Some tests reveals that the problem is in stubdomain - the very same config ( |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Feb 22, 2016
Member
Using libxl (xen packages 4.6.1) from Fedora 24 it does work, even with stubdomain...
|
Using libxl (xen packages 4.6.1) from Fedora 24 it does work, even with stubdomain... |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
tirrorex
Feb 25, 2016
Though fedora 24 wasn't due until april?
When you say it is working, you mean flawless with every device just like kvm?
tirrorex
commented
Feb 25, 2016
|
Though fedora 24 wasn't due until april? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Feb 25, 2016
Member
Though fedora 24 wasn't due until april?
Yes, I've used packages from rawhide - to have the same version (F23 has Xen 4.5).
When you say it is working, you mean flawless with every device just like kvm?
I've just tried one sample device and it is properly discovered in the VM. With our libxl/stubdom packages it doesn't show at all.
Yes, I've used packages from rawhide - to have the same version (F23 has Xen 4.5).
I've just tried one sample device and it is properly discovered in the VM. With our libxl/stubdom packages it doesn't show at all. |
added a commit
to marmarek/old-qubes-core-admin
that referenced
this issue
Feb 26, 2016
added a commit
to marmarek/old-qubes-core-admin
that referenced
this issue
Feb 26, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Feb 26, 2016
Member
Made automated test for this issue (annotated with "expected failure" for now). Should ease debugging (for example bisection).
Also, I'm unable to reproduce the success with unmodified xen-4.6.0 toolstack (+qemu) compiled manually. Maybe that success was previous because of some Fedora patch. Or it is some race condition (Fedora was running from USB stick, while Qubes from fast SSD disk). Or something totally different...
|
Made automated test for this issue (annotated with "expected failure" for now). Should ease debugging (for example bisection). |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
esheltone
Feb 26, 2016
The race condition idea seems unlikely, since the problem we are chasing is seen across all systems on Qubes.
There is a surprising number of patches in Fedora for Xen. However, at a glance, the only patches that seem to touch on code that would relate to this kind of problem are the patches for XSAs 154, 164, and 170 - assuming one of these patches is responsible.
esheltone
commented
Feb 26, 2016
|
The race condition idea seems unlikely, since the problem we are chasing is seen across all systems on Qubes. There is a surprising number of patches in Fedora for Xen. However, at a glance, the only patches that seem to touch on code that would relate to this kind of problem are the patches for XSAs 154, 164, and 170 - assuming one of these patches is responsible. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jaspertron
May 22, 2016
@marmarek, is this an accurate summary of your testing?
| stubdomain | 'qemu-xen-traditional' | |
|---|---|---|
| Xen 4.6.0 with Qubes patches | broken | working |
| Xen 4.6.0 (unmodified) | broken | working |
| Xen 4.6.1 with Fedora patches | working | working |
Also, how did you create the xl config file for testing? I tried doing
virsh -c xen:/// domxml-to-native xen-xl /etc/libvirt/libxl/my-test-hvm.xml
but that gives me
error: Disconnected from xen:/// due to I/O error
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
jaspertron
commented
May 22, 2016
|
@marmarek, is this an accurate summary of your testing?
Also, how did you create the xl config file for testing? I tried doing |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
May 22, 2016
Member
I would rather say "Fedora 24" instead of "Xen 4.6.1 with Fedora patches" - this may be broken by some other package than Xen (some library used or so). Additionally, I wasn't able to reproduce the success when running Qubes but launching domain using Fedora 24 binaries (from chroot). Which is another hint it isn't about just toolstack/qemu.
As for config file - something like this. And indeed it seems to be crashing libvirtd... It looks like the bug is triggered by lack of <graphics type='vnc'/> entry (which is intentional on Qubes). Anyway adding it produced some config file. Then, to enable stubdomain you need to add device_model_stubdomain_override=1. And probably set vnc = 0 ;)
I guess the whole problem may have something to do with disabled qemu in dom0 (in addition to stubdomain). This isn't fully consistent with test results, but there may be some other factors.
|
I would rather say "Fedora 24" instead of "Xen 4.6.1 with Fedora patches" - this may be broken by some other package than Xen (some library used or so). Additionally, I wasn't able to reproduce the success when running Qubes but launching domain using Fedora 24 binaries (from chroot). Which is another hint it isn't about just toolstack/qemu. As for config file - something like this. And indeed it seems to be crashing libvirtd... It looks like the bug is triggered by lack of I guess the whole problem may have something to do with disabled qemu in dom0 (in addition to stubdomain). This isn't fully consistent with test results, but there may be some other factors. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
May 22, 2016
Member
Found configs from those tests: https://gist.github.com/marmarek/794305496557cc679fced21e252e05b4
May contain some later changes though...
|
Found configs from those tests: https://gist.github.com/marmarek/794305496557cc679fced21e252e05b4 |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jaspertron
May 30, 2016
It looks like the bug is triggered by lack of entry (which is intentional on Qubes). Anyway adding it produced some config file.
Thanks, that did the trick.
I'm getting the same results as you; it only works without a stubdomain.
Additionally, I wasn't able to reproduce the success when running Qubes but launching domain using Fedora 24 binaries (from chroot). Which is another hint it isn't about just toolstack/qemu.
Could the xen-pciback kernel module be to blame? Does Qubes make any modifications to it?
jaspertron
commented
May 30, 2016
Thanks, that did the trick.
Could the xen-pciback kernel module be to blame? Does Qubes make any modifications to it? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
May 30, 2016
Member
Could the xen-pciback kernel module be to blame? Does Qubes make any modifications to it?
No, we don't have any modifications there.
It was working in Qubes R2, but there are a lot of differences:
- Xen version (was 4.1) - all the parts: hypervisor, qemu, toolstack
- Kernel version (was 3.12, probably irrelevant)
- Libvirt usage vs xl directly (this should be excluded by above tests)
Next thing I'd check is qemu in stubdomain - simply get stubdomain binary from R2 and try it on R3.x. It is in /usr/lib/xen/boot/ioemu-stubdom.gz, which is shipped in xen-hvm rpm.
No, we don't have any modifications there. It was working in Qubes R2, but there are a lot of differences:
Next thing I'd check is qemu in stubdomain - simply get stubdomain binary from R2 and try it on R3.x. It is in |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jaspertron
May 30, 2016
Next thing I'd check is qemu in stubdomain - simply get stubdomain binary from R2 and try it on R3.x. It is in /usr/lib/xen/boot/ioemu-stubdom.gz, which is shipped in xen-hvm rpm.
Ok, I replaced /usr/lib/xen/boot/ioemu-stubdom.gz with the ioemu-stubdom.gz from Qubes-R2-x86_64-DVD.iso. Unfortunately it doesn't want to start:
[user@dom0 ~]$ sudo xl create pcihvm.xl
Parsing config from pcihvm.xl
libxl: error: libxl_dm.c:1671:stubdom_xswait_cb: Stubdom 13 for 12 startup: startup timed out
libxl: error: libxl_create.c:1339:domcreate_devmodel_started: device model did not start: -9
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [10199] exited with error status 1
libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [10197] exited with error status 1
libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
libxl: error: libxl.c:1606:libxl__destroy_domid: non-existant domain 12
libxl: error: libxl.c:1564:domain_destroy_callback: unable to destroy guest with domid 12
libxl: error: libxl.c:1491:domain_destroy_cb: destruction of domain 12 failed
Here's some of the output from /var/log/xen/qemu-dm-pcihvm.log:
---snip---
Register xen platform.
Done register platform.
xs_watch(/local/domain/12/log-throttling, /local/domain/12/log-throttling)
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
qubes_gui/init: 660
qubes_gui/init: 669
qubes_gui/init: 672
qubes_gui/init: 681
xs_daemon_open -> 9, 0x1609f8
evtchn_open() -> 10
xc_evtchn_bind_unbound_port(0) = 0
xs_write(device/vchan/6000/ring-ref): EACCES
close(10)
libvchan_server_init:
close(0)
GPF rip: 0xfc514, error_code=0
Thread: main
RIP: e030:[<00000000000fc514>]
RSP: e02b:00000000005ef8a8 EFLAGS: 00010202
RAX: 2f302f6e69616d6f RBX: 0000002002c087c0 RCX: 0000000000001055
RDX: 000000000000000a RSI: 00000000005ef798 RDI: 2f302f6e69616d6f
RBP: 00000000005ef8a8 R08: 000000000000000a R09: 0000000000576000
R10: 000000000000104b R11: 0000000000000ffa R12: 0000000000000000
R13: 00000000001635f0 R14: 0000000000000000 R15: 0000000000163558
base is 0x5ef8a8 caller is 0xe2a78
base is 0x5ef908 caller is 0xe2635
base is 0x5ef918 caller is 0xddc61
base is 0x5ef938 caller is 0x1047ed
base is 0x5ef958 caller is 0xfad7d
base is 0x5ef968 caller is 0xf4510
base is 0x5ef998 caller is 0xf45ac
base is 0x5ef9b8 caller is 0xf6379
base is 0x5efa08 caller is 0xf4b2e
base is 0x5efa18 caller is 0xf4474
base is 0x5efa38 caller is 0x24790
base is 0x5efa58 caller is 0x24002
base is 0x5efa78 caller is 0x8ea6
base is 0x5efe08 caller is 0xd7129
base is 0x5effe8 caller is 0x33da
5ef890: a8 f8 5e 00 00 00 00 00 2b e0 00 00 00 00 00 00
5ef8a0: 01 00 00 00 00 00 00 00 08 f9 5e 00 00 00 00 00
5ef8b0: 78 2a 0e 00 00 00 00 00 01 00 00 00 00 00 00 00
5ef8c0: 1a 00 00 00 00 00 00 00 f8 f8 5e 00 00 00 00 00
5ef890: a8 f8 5e 00 00 00 00 00 2b e0 00 00 00 00 00 00
5ef8a0: 01 00 00 00 00 00 00 00 08 f9 5e 00 00 00 00 00
5ef8b0: 78 2a 0e 00 00 00 00 00 01 00 00 00 00 00 00 00
5ef8c0: 1a 00 00 00 00 00 00 00 f8 f8 5e 00 00 00 00 00
fc500: ca 48 85 f2 74 ea eb 0c 0f 1f 84 00 00 00 00 00
fc510: 48 83 c0 01 80 38 00 75 f7 48 29 f8 5d c3 66 90
fc520: 55 48 89 f8 48 89 f9 a8 07 48 89 e5 75 56 48 8b
fc530: 0f 49 ba ff fe fe fe fe fe fe fe 49 89 c8 4c 01
Any ideas?
jaspertron
commented
May 30, 2016
Ok, I replaced /usr/lib/xen/boot/ioemu-stubdom.gz with the ioemu-stubdom.gz from Qubes-R2-x86_64-DVD.iso. Unfortunately it doesn't want to start: Here's some of the output from /var/log/xen/qemu-dm-pcihvm.log: Any ideas? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
May 30, 2016
Member
vchan library is different in R2 than in R3.x... You can try to fake the old one, execute:
xenstore-write /local/domain/`xl domid pcihvm-dm`/device/vchan ''
xenstore-chmod /local/domain/`xl domid pcihvm-dm`/device/vchan n`xl domid pcihvm-dm`
But you need to be very fast with this, as you need to make it before stubdom reach this GUI initialization. Maybe xl create -p will help, but AFAIR it only keep the target domain paused, not stubdomain.
|
vchan library is different in R2 than in R3.x... You can try to fake the old one, execute: But you need to be very fast with this, as you need to make it before stubdom reach this GUI initialization. Maybe |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jaspertron
May 31, 2016
You can try to fake the old one, execute:
xenstore-write /local/domain/`xl domid pcihvm-dm`/device/vchan '' xenstore-chmod /local/domain/`xl domid pcihvm-dm`/device/vchan n`xl domid pcihvm-dm`
That seems to help it go a little further before running into a different error:
---snip---
Register xen platform.
Done register platform.
xs_watch(/local/domain/42/log-throttling, /local/domain/42/log-throttling)
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
qubes_gui/init: 660
qubes_gui/init: 669
qubes_gui/init: 672
qubes_gui/init: 681
xs_daemon_open -> 7, 0x1606d8
evtchn_open() -> 8
xc_evtchn_bind_unbound_port(0) = 0
qubes gui initialized
resize to 640x480@32, 2560 required
xs_write(/local/domain/0/device-model/42/state): EACCES
error recording dm
xs_read_watch() -> /local/domain/42/log-throttling /local/domain/42/log-throttling
xs_read(/local/domain/42/log-throttling): ENOENT
xs_read(/local/domain/42/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/42/log-throttling'
medium change watch on `/local/domain/42/log-throttling' - unknown device, ignored
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
resize to 720x400@32, 2880 required
xs_read_watch() -> /local/domain/42/cpu vcpu-set
vcpu-set: watch node error.
[xenstore_process_vcpu_set_event]: /local/domain/42/cpu has no CPU!
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
xs_read_watch() -> /local/domain/0/device-model/42/command dm-command
xs_read(/local/domain/0/device-model/42/command): EACCES
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
xs_read_watch() -> /local/domain/0/device-model/42/logdirty/cmd logdirty
xs_read(/local/domain/0/device-model/42/logdirty/cmd): EACCES
Log-dirty: no command yet.
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
xs_read_watch() -> /local/domain/10/backend/vbd/43/51760/params xvdd
Using xvdd for guest's hdd
medium change watch on `xvdd' (index: 0): /home/user/Downloads/Fedora-Live-Workstation-x86_64-23-10.iso
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
jaspertron
commented
May 31, 2016
That seems to help it go a little further before running into a different error: |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
May 31, 2016
Member
Are you sure about path? Is xen-blkback module loaded in domain holding this image?
|
Are you sure about path? Is |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jaspertron
May 31, 2016
Are you sure about path? Is xen-blkback module loaded in domain holding this image?
Yes to both, and it works with the original (R3.1) stubdomain binary:
---snip---
Register xen platform.
Done register platform.
xs_watch(/local/domain/21/log-throttling, /local/domain/21/log-throttling)
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
qubes_gui/init: 657
qubes_gui/init: 666
qubes_gui/init: 669
qubes_gui/init: 678
evtchn_open() -> 8
xc_evtchn_bind_unbound_port(0) = 0
xs_daemon_open -> 9, 0x15cec8
qubes_gui/init[708]: version sent, waiting for xorg conf
qubes gui initialized
resize to 640x480@32, 2560 required
xs_read_watch() -> /local/domain/21/log-throttling /local/domain/21/log-throttling
xs_read(/local/domain/21/log-throttling): ENOENT
xs_read(/local/domain/21/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/21/log-throttling'
medium change watch on `/local/domain/21/log-throttling' - unknown device, ignored
resize to 720x400@32, 2880 required
xs_read_watch() -> /local/domain/21/cpu vcpu-set
vcpu-set: watch node error.
[xenstore_process_vcpu_set_event]: /local/domain/21/cpu has no CPU!
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
xs_read_watch() -> device-model/21/command dm-command
xs_read(device-model/21/command): ENOENT
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
xs_read_watch() -> device-model/21/logdirty/cmd logdirty
xs_read(device-model/21/logdirty/cmd): ENOENT
Log-dirty: no command yet.
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
xs_read_watch() -> /local/domain/6/backend/vbd/22/51760/params xvdd
Using xvdd for guest's hdd
medium change watch on `xvdd' (index: 0): /home/user/Downloads/Fedora-Live-Workstation-x86_64-23-10.iso
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
vga s->lfb_addr = f0000000 s->lfb_end = f1000000
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
mapping vram to f0000000 - f1000000
resize to 640x480@32, 2560 required
resize to 720x400@32, 2880 required
Unknown PV product 3 loaded in guest
PV driver build 1
vga s->lfb_addr = f0000000 s->lfb_end = f1000000
vga s->lfb_addr = f0000000 s->lfb_end = f1000000
vga s->lfb_addr = f0000000 s->lfb_end = f1000000
vga s->lfb_addr = f0000000 s->lfb_end = f1000000
vga s->lfb_addr = f0000000 s->lfb_end = f1000000
vga s->lfb_addr = f0000000 s->lfb_end = f1000000
vga s->lfb_addr = f0000000 s->lfb_end = f1000000
resize to 1024x768@32, 4096 required
xs_daemon_open -> 10, 0x15cee8
qubes_gui/init[719]: got xorg conf, creating window
qubes_gui/init: 726
dumping mfns: n=768, w=1024, h=768, bpp=32
configure msg, x/y 1682 18 (was 0 0), w/h 236 1041
jaspertron
commented
May 31, 2016
Yes to both, and it works with the original (R3.1) stubdomain binary: |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
May 31, 2016
Member
Hmm, where is actual error? I see I/O request not ready in both working and not-working setups... Maybe the real problem is:
xs_write(/local/domain/0/device-model/42/state): EACCES
error recording dm
?
This looks strange, as the entry should be inside stubdomain directory. Maybe this have changed from Xen 4.1 time...
You can emulate proper behavior with:
xenstore-write /local/domain/`xl domid pcihvm-dm`/device-model/`xl domid pcihvm`/state running
(wait a sec or two with this, to make sure stubdomain really have started, or simply watch its log)
|
Hmm, where is actual error? I see ? (wait a sec or two with this, to make sure stubdomain really have started, or simply watch its log) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jaspertron
May 31, 2016
You can emulate proper behavior with:
xenstore-write /local/domain/`xl domid pcihvm-dm`/device-model/`xl domid pcihvm`/state running
That did the trick! The vm gets created without any errors.
However, it won't let me connect to the GUI agent. When I do qubes-guid -dxl domid pcihvm-dm-N pcihvm, it times out.
jaspertron
commented
May 31, 2016
That did the trick! The vm gets created without any errors. However, it won't let me connect to the GUI agent. When I do |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
May 31, 2016
Member
As vchan library is different, it will not work. You may try with xl console pcihvm - if that system expose login prompt at Xen console. If not, try booting something which will allow you to access through network. You'll probably need to force static IP address, or launch DHCP server in directly connected ProxyVM/NetVM.
|
As vchan library is different, it will not work. You may try with |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
May 31, 2016
Member
Also, you may take a look at automated test for this:
https://github.com/QubesOS/qubes-core-admin/blob/master/tests/hardware.py
https://github.com/QubesOS/qubes-core-admin/blob/master/tests/__init__.py#L511-L580
It creates small initramfs (using dracut) with the only purpose to dump lspci output to private.img. Then setup it to really start there (install grub etc).
You may even try to launch this test, using:
python -m qubes.tests.run hardware/TC_00_HVM/test_000_pci_passthrough_presence
Documentation: https://www.qubes-os.org/doc/automated-tests/
And a warning repeated here:
Integration tests are written with assumption to be called on dedicated hardware. Do not run those test on machine where you have important data, you can loose it. Especially all the VMs with name starting with test- are removed.
|
Also, you may take a look at automated test for this: It creates small initramfs (using You may even try to launch this test, using: Documentation: https://www.qubes-os.org/doc/automated-tests/ |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jaspertron
Jun 2, 2016
You may even try to launch this test, using:
python -m qubes.tests.run hardware/TC_00_HVM/test_000_pci_passthrough_presence
When I run this test I get an error after about 30 seconds:
ERROR (libvirtError: internal error: libxenlight failed to create new domain 'test-inst-vm1')
======================================================================
ERROR: hardware/TC_00_HVM/test_000_pci_passthrough_presence
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/qubes/tests/hardware.py", line 59, in test_000_pci_passthrough_presence
self.vm.start()
File "/usr/lib64/python2.7/site-packages/qubes/modules/01QubesHVm.py", line 326, in start
return super(QubesHVm, self).start(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py", line 1901, in start
self.libvirt_domain.createWithFlags(libvirt.VIR_DOMAIN_START_PAUSED)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: internal error: libxenlight failed to create new domain 'test-inst-vm1'
----------------------------------------------------------------------
Ran 1 test in 31.973s
FAILED (errors=1)
And here's what gets written to /var/log/libvirt/libxl/libxl-driver.log:
2016-06-02 01:17:19 CDT xc: error: panic: xc_dom_core.c:207: failed to open file: No such file or directory: Internal error
2016-06-02 01:17:19 CDT libxl: error: libxl_dom.c:644:libxl__build_pv: xc_dom_kernel_file failed: No such file or directory
2016-06-02 01:17:19 CDT libxl: error: libxl_dm.c:1639:stubdom_pvqemu_cb: error connecting nics devices: No such file or directory
2016-06-02 01:17:19 CDT libxl: error: libxl_create.c:1339:domcreate_devmodel_started: device model did not start: -3
2016-06-02 01:17:19 CDT libxl: error: libxl_dm.c:2031:libxl__destroy_device_model: xs_rm failed for /local/domain/0/device-model/5
2016-06-02 01:17:19 CDT libxl: error: libxl_dm.c:1983:kill_device_model: unable to find device model pid in /local/domain/5/image/device-model-pid
2016-06-02 01:17:19 CDT libxl: error: libxl.c:1643:libxl__destroy_domid: libxl__destroy_device_model failed for 5
jaspertron
commented
Jun 2, 2016
When I run this test I get an error after about 30 seconds: And here's what gets written to |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Jun 2, 2016
Member
xc_dom_kernel_file failed: No such file or directory? In HVM?
Anyway, you can add --do-not-clean option to leave all the files present and try to start it manually. Or even get only root.img from there and use it in your previous test VM. Script there is supposed to write lspci output directly to private.img (without any filesystem).
|
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jaspertron
Jun 3, 2016
Oops, ignore that last error. I had /usr/lib/xen/boot/ioemu-stubdom.gz symlinked to a non-existent file.
Using root.img and private.img from that Qubes test with my previous test VM:
[user@dom0 ~]$ sudo xl create pcihvm.xl
Parsing config from pcihvm.xl
libxl: error: libxl_pci.c:1041:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.0
--- waits here for about 60 seconds ---
libxl: error: libxl_exec.c:227:libxl__xenstore_child_wait_deprecated: Device Model not ready
libxl: error: libxl_pci.c:879:qemu_pci_add_xenstore: qemu refused to add device: 0000:01:00.0,msitranslate=0,power_mgmt=0
libxl: error: libxl_create.c:1422:domcreate_attach_pci: libxl_device_pci_add failed: -3
libxl: error: libxl_pci.c:1041:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.0
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [31290] exited with error status 1
libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [31288] exited with error status 1
libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
libxl: error: libxl.c:1606:libxl__destroy_domid: non-existant domain 33
libxl: error: libxl.c:1564:domain_destroy_callback: unable to destroy guest with domid 33
libxl: error: libxl.c:1491:domain_destroy_cb: destruction of domain 33 failed
/var/log/xen/qemu-dm-pcihvm.log:
---snip---
qubes gui initialized
resize to 640x480@32, 2560 required
pcifront_watches: waiting for backend to get into the right state /local/domain/0/backend/
pci/34/0
xs_write(/local/domain/0/device-model/33/state): EACCES
error recording dm
xs_read_watch() -> /local/domain/33/log-throttling /local/domain/33/log-throttling
xs_read(/local/domain/33/log-throttling): ENOENT
xs_read(/local/domain/33/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/33/log-throttling'
medium change watch on `/local/domain/33/log-throttling' - unknown device, ignored
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
resize to 720x400@32, 2880 required
xs_read_watch() -> /local/domain/33/cpu vcpu-set
vcpu-set: watch node error.
[xenstore_process_vcpu_set_event]: /local/domain/33/cpu has no CPU!
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
xs_read_watch() -> /local/domain/0/device-model/33/command dm-command
******************* PCIFRONT for device/pci/0 **********
xs_read(/local/domain/0/device-model/33/command): EACCES
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
xs_read_watch() -> /local/domain/0/device-model/33/logdirty/cmd logdirty
xs_read(/local/domain/0/device-model/33/logdirty/cmd): EACCES
Log-dirty: no command yet.
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
backend at /local/domain/0/backend/pci/34/0
**************************
pcifront_watches: waiting for backend events /local/domain/0/backend/pci/34/0/state
--- waits here for about 60 seconds ---
pcifront_watches: backend state changed: /local/domain/0/backend/pci/34/0/state 7
pcifront_watches: writing device/pci/0/state 7
pcifront_watches: backend state changed: /local/domain/0/backend/pci/34/0/state 8
pcifront_watches: writing device/pci/0/state 4
pcifront_watches: changing state to 4
pcifront_watches: backend state changed: /local/domain/0/backend/pci/34/0/state 4
jaspertron
commented
Jun 3, 2016
|
Oops, ignore that last error. I had Using /var/log/xen/qemu-dm-pcihvm.log: |
marmarek
added
the
r3.2-dom0-stable
label
Nov 17, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Nov 22, 2016
Member
Automated announcement from builder-github
The package xen-4.6.3-22.fc21 has been pushed to the r3.1 testing repository for the Fedora fc21 template.
To test this update, please install it with the following command:
sudo yum update --enablerepo=qubes-vm-r3.1-current-testing
|
Automated announcement from builder-github The package |
marmarek
added
the
r3.1-fc21-cur-test
label
Nov 22, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Nov 22, 2016
Member
Automated announcement from builder-github
The package xen-4.6.3-22.fc22 has been pushed to the r3.1 testing repository for the Fedora fc22 template.
To test this update, please install it with the following command:
sudo yum update --enablerepo=qubes-vm-r3.1-current-testing
|
Automated announcement from builder-github The package |
marmarek
added
the
r3.1-fc22-cur-test
label
Nov 22, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Nov 22, 2016
Member
Automated announcement from builder-github
The package xen-4.6.3-22.fc23 has been pushed to the r3.1 testing repository for the Fedora fc23 template.
To test this update, please install it with the following command:
sudo yum update --enablerepo=qubes-vm-r3.1-current-testing
|
Automated announcement from builder-github The package |
marmarek
added
the
r3.1-fc23-cur-test
label
Nov 22, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Nov 22, 2016
Member
Automated announcement from builder-github
The package xen-4.6.3-22.fc24 has been pushed to the r3.1 testing repository for the Fedora fc24 template.
To test this update, please install it with the following command:
sudo yum update --enablerepo=qubes-vm-r3.1-current-testing
|
Automated announcement from builder-github The package |
marmarek
added
the
r3.1-fc24-cur-test
label
Nov 22, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Nov 22, 2016
Member
Automated announcement from builder-github
The package xen-4.6.3-22.fc20 has been pushed to the r3.1 testing repository for dom0.
To test this update, please install it with the following command:
sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing
|
Automated announcement from builder-github The package |
marmarek
added
the
r3.1-dom0-cur-test
label
Nov 22, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Nov 26, 2016
Member
Automated announcement from builder-github
The package xen_4.6.3-24+deb8u1 has been pushed to the r3.2 testing repository for the Debian jessie template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing jessie-testing, then use the standard update command:
sudo apt-get update && sudo apt-get dist-upgrade
|
Automated announcement from builder-github The package |
marmarek
added
the
r3.2-jessie-cur-test
label
Nov 26, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Nov 26, 2016
Member
Automated announcement from builder-github
The package xen_4.6.3-24+deb9u1 has been pushed to the r3.2 testing repository for the Debian stretch template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing, then use the standard update command:
sudo apt-get update && sudo apt-get dist-upgrade
|
Automated announcement from builder-github The package |
marmarek
added
the
r3.2-stretch-cur-test
label
Nov 26, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Nov 29, 2016
Member
Automated announcement from builder-github
The package xen-4.6.3-24.fc20 has been pushed to the r3.1 stable repository for dom0.
To install this update, please use the standard update command:
sudo qubes-dom0-update
Or update dom0 via Qubes Manager.
|
Automated announcement from builder-github The package Or update dom0 via Qubes Manager. |
marmarek
added
r3.1-dom0-stable
and removed
r3.1-dom0-cur-test
labels
Nov 29, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Dec 4, 2016
Member
Automated announcement from builder-github
The package xen-4.6.3-24.fc21 has been pushed to the r3.1 stable repository for the Fedora fc21 template.
To install this update, please use the standard update command:
sudo yum update
|
Automated announcement from builder-github The package |
marmarek
added
r3.1-fc21-stable
and removed
r3.1-fc21-cur-test
labels
Dec 4, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Dec 4, 2016
Member
Automated announcement from builder-github
The package xen_2001:4.6.3-24+deb8u1 has been pushed to the r3.2 stable repository for the Debian jessie template.
To install this update, please use the standard update command:
sudo apt-get update && sudo apt-get dist-upgrade
|
Automated announcement from builder-github The package |
marmarek
added
r3.2-jessie-stable
and removed
r3.2-jessie-cur-test
labels
Dec 4, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Dec 4, 2016
Member
Automated announcement from builder-github
The package xen-4.6.3-24.fc22 has been pushed to the r3.1 stable repository for the Fedora fc22 template.
To install this update, please use the standard update command:
sudo yum update
|
Automated announcement from builder-github The package |
marmarek
added
r3.1-fc22-stable
and removed
r3.1-fc22-cur-test
labels
Dec 4, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Dec 4, 2016
Member
Automated announcement from builder-github
The package xen-4.6.3-24.fc23 has been pushed to the r3.1 stable repository for the Fedora fc23 template.
To install this update, please use the standard update command:
sudo yum update
|
Automated announcement from builder-github The package |
marmarek
added
r3.1-fc23-stable
and removed
r3.1-fc23-cur-test
labels
Dec 4, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Dec 5, 2016
Member
Automated announcement from builder-github
The package xen_4.6.3-24+deb8u1 has been pushed to the r3.1 testing repository for the Debian jessie template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing jessie-testing, then use the standard update command:
sudo apt-get update && sudo apt-get dist-upgrade
|
Automated announcement from builder-github The package |
marmarek
added
the
r3.1-jessie-cur-test
label
Dec 5, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Dec 5, 2016
Member
Automated announcement from builder-github
The package xen_4.6.3-24+deb9u1 has been pushed to the r3.1 testing repository for the Debian stretch template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing, then use the standard update command:
sudo apt-get update && sudo apt-get dist-upgrade
|
Automated announcement from builder-github The package |
marmarek
added
the
r3.1-stretch-cur-test
label
Dec 5, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Dec 5, 2016
Member
Automated announcement from builder-github
The package xen_4.6.3-24+deb7u1 has been pushed to the r3.1 testing repository for the Debian wheezy template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing wheezy-testing, then use the standard update command:
sudo apt-get update && sudo apt-get dist-upgrade
|
Automated announcement from builder-github The package |
marmarek
added
the
r3.1-wheezy-cur-test
label
Dec 5, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
qubesos-bot
Dec 19, 2016
Automated announcement from builder-github
The package xen_2001:4.6.3-24+deb9u1 has been pushed to the r3.2 stable repository for the Debian stretch template.
To install this update, please use the standard update command:
sudo apt-get update && sudo apt-get dist-upgrade
qubesos-bot
commented
Dec 19, 2016
|
Automated announcement from builder-github The package |
esheltone commentedJan 19, 2016
There have been multiple reports that PCI passthrough does not work for HVM domains using the qubes software:
https://groups.google.com/d/msg/qubes-users/cmPRMOkxkdA/gIV68O0-CQAJ (reporting passthrough not working via libvirt, but that passthrough still could be done using Xen xl)
https://groups.google.com/d/msg/qubes-users/ExMvykCyYiY/M3nHxweRFAAJ (confirmation by Marek that passthrough was not working on R3)
https://groups.google.com/d/msg/qubes-users/ppKj_YWqr94/l2gHv6uJAgAJ
This issue appears to have started with use of the HAL in Qubes R3. PCI passthrough continues to work fine for PV-based Qubes VMs, such as sys-net.
Marek guessed that it could be a qemu issue (see second linked post). However, in the first linked post, PCI passthrough was done to an HVM domain via 'xl' using "device_model_version = 'qemu-xen-traditional'", so this may rule out qemu as the culprit.