Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upnetwork attach/detach problem with kernel 4.14.13 or above. #3657
Comments
andrewdavidwong
added
bug
C: kernel
labels
Mar 6, 2018
andrewdavidwong
added this to the Release 3.2 updates milestone
Mar 6, 2018
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Zrubi
Mar 29, 2018
Member
This issue is affecting the just released 4.0 (with kernel 4.14.18)
edit: downgrade to 4.14.12 works in case of 4.0 too:
sudo qubes-dom0-update --action=downgrade kernel-latest-4.14.12 kernel-latest-qubes-vm-4.14.12
I do not believe if this is issue is model specific, but my hardware is a Lenovo T450
|
This issue is affecting the just released 4.0 (with kernel 4.14.18) I do not believe if this is issue is model specific, but my hardware is a Lenovo T450 |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
sorandom
Apr 2, 2018
In qubes 4.0 (rc4 but fully upgraded, kernel 4.14.18-1) I am getting the same issue. VMs will exhibit this behaviour intermittently after a resume from suspend. Hardware is a Librem 13v2.
Network traffic will not pass through them. In this case, I had the particular VM acting as a VPN firewall and its xterm was usable but would not pass traffic. (appVMs -> appfw -> vpn -> fw (buggy) -> sys-net) Thus, I attempted to set my VPN vm's netvm property to "" so I could restart the firewall.
It manifests as "qvm-prefs: error: no such property: 'netvm'". Journald below:
<time> dom0 qubesd[17004]: unhandled exception while calling src=b'dom0' meth=b'admin.vm.property.Set' dest=b'sys-vpn' arg=b'netvm' len(untrusted_payload)=0
<time> dom0 qubesd[17004]: Traceback (most recent call last):
<time> dom0 qubesd[17004]: File "/usr/lib/python3.5/site-packages/qubes/api/__init__.py", line 262, in respond
<time> dom0 qubesd[17004]: untrusted_payload=untrusted_payload)
<time> dom0 qubesd[17004]: File "/usr/lib64/python3.5/asyncio/futures.py", line 381, in __iter__
<time> dom0 qubesd[17004]: yield self # This tells Task to wait for completion.
<time> dom0 qubesd[17004]: File "/usr/lib64/python3.5/asyncio/tasks.py", line 310, in _wakeup
<time> dom0 qubesd[17004]: future.result()
<time> dom0 qubesd[17004]: File "/usr/lib64/python3.5/asyncio/futures.py", line 294, in result
<time> dom0 qubesd[17004]: raise self._exception
<time> dom0 qubesd[17004]: File "/usr/lib64/python3.5/asyncio/tasks.py", line 240, in _step
<time> dom0 qubesd[17004]: result = coro.send(None)
<time> dom0 qubesd[17004]: File "/usr/lib64/python3.5/asyncio/coroutines.py", line 210, in coro
<time> dom0 qubesd[17004]: res = func(*args, **kw)
<time> dom0 qubesd[17004]: File "/usr/lib/python3.5/site-packages/qubes/api/admin.py", line 243, in vm_property_set
<time> dom0 qubesd[17004]: untrusted_payload=untrusted_payload)
<time> dom0 qubesd[17004]: File "/usr/lib/python3.5/site-packages/qubes/api/admin.py", line 263, in _property_set
<time> dom0 qubesd[17004]: setattr(dest, self.arg, newvalue)
<time> dom0 qubesd[17004]: File "/usr/lib/python3.5/site-packages/qubes/vm/__init__.py", line 569, in __set__
<time> dom0 qubesd[17004]: super(VMProperty, self).__set__(instance, value)
<time> dom0 qubesd[17004]: File "/usr/lib/python3.5/site-packages/qubes/__init__.py", line 260, in __set__
<time> dom0 qubesd[17004]: name=self.__name__, newvalue=value, oldvalue=oldvalue)
<time> dom0 qubesd[17004]: File "/usr/lib/python3.5/site-packages/qubes/events.py", line 198, in fire_event
<time> dom0 qubesd[17004]: pre_event=pre_event)
<time> dom0 qubesd[17004]: File "/usr/lib/python3.5/site-packages/qubes/events.py", line 166, in _fire_event
<time> dom0 qubesd[17004]: effect = func(self, event, **kwargs)
<time> dom0 qubesd[17004]: File "/usr/lib/python3.5/site-packages/qubes/vm/mix/net.py", line 418, in on_property_pre_set_netvm
<time> dom0 qubesd[17004]: self.detach_network()
<time> dom0 qubesd[17004]: File "/usr/lib/python3.5/site-packages/qubes/vm/mix/net.py", line 326, in detach_network
<time> dom0 qubesd[17004]: vm=self))
<time> dom0 qubesd[17004]: File "/usr/lib/python3.5/site-packages/qubes/app.py", line 94, in wrapper
<time> dom0 qubesd[17004]: return attr(*args, **kwargs)
<time> dom0 qubesd[17004]: File "/usr/lib64/python3.5/site-packages/libvirt.py", line 1170, in detachDevice
<time> dom0 qubesd[17004]: if ret == -1: raise libvirtError ('virDomainDetachDevice() failed', dom=self)
<time> dom0 qubesd[17004]: libvirt.libvirtError: internal error: libxenlight failed to detach network device
<time> dom0 qrexec[14428]: qubes.NotifyUpdates: sys-net -> dom0: allowed to dom0
sorandom
commented
Apr 2, 2018
•
edited
Edited 3 times
-
sorandom
edited Apr 2, 2018 (most recent)
-
sorandom
edited
Apr 2, 2018
-
sorandom
edited
Apr 2, 2018
edited
-
Apr 2, 2018 (most recent) -
Apr 2, 2018 -
Apr 2, 2018
|
In qubes 4.0 (rc4 but fully upgraded, kernel 4.14.18-1) I am getting the same issue. VMs will exhibit this behaviour intermittently after a resume from suspend. Hardware is a Librem 13v2. Network traffic will not pass through them. In this case, I had the particular VM acting as a VPN firewall and its xterm was usable but would not pass traffic. (appVMs -> appfw -> vpn -> fw (buggy) -> sys-net) Thus, I attempted to set my VPN vm's netvm property to "" so I could restart the firewall. It manifests as "qvm-prefs: error: no such property: 'netvm'". Journald below: |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
sorandom
Apr 2, 2018
sudo qubes-dom0-update --action=downgrade kernel-latest-4.14.12 kernel-latest-qubes-vm-4.14.12
This did not work for me.
[user@dom0 ~]$ sudo qubes-dom0-update --action=downgrade kernel-latest-4.14.12 kernel-latest-qubes-vm-4.14.12
Using <vm> as UpdateVM to download updates for Dom0; this may take some time...
Last metadata expiration check: 0:00:52 ago on Mon Apr 2 11:00:45 2018.
Packages for argument kernel-latest-4.14.12 available, but not installed.
Packages for argument kernel-latest-qubes-vm-4.14.12 available, but not installed.
Error: No packages marked for downgrade.
I can get it to download but not install:
[user@dom0 ~]$ sudo qubes-dom0-update kernel-latest-4.14.12 kernel-latest-qubes-vm-4.14.12
Using <vm> as UpdateVM to download updates for Dom0; this may take some time...
Fedora 25 - x86_64 - Updates 1.3 MB/s | 24 MB 00:19
Fedora 25 - x86_64 1.8 MB/s | 50 MB 00:28
Qubes Dom0 Repository (updates) 513 kB/s | 3.0 MB 00:06
Qubes Templates repository 19 kB/s | 8.3 kB 00:00
Last metadata expiration check: 0:00:00 ago on Mon Apr 2 11:55:59 2018.
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
kernel-latest x86_64 1000:4.14.12-1.pvops.qubes qubes-dom0-current 46 M
kernel-latest-qubes-vm
x86_64 1000:4.14.12-1.pvops.qubes qubes-dom0-current 63 M
Transaction Summary
================================================================================
Install 2 Packages
Total download size: 108 M
Installed size: 108 M
DNF will only download packages for the transaction.
Downloading Packages:
(1/2): kernel-latest-4.14.12-1.pvops.qubes.x86_ 881 kB/s | 46 MB 00:53
(2/2): kernel-latest-qubes-vm-4.14.12-1.pvops.q 753 kB/s | 63 MB 01:25
--------------------------------------------------------------------------------
Total 1.3 MB/s | 108 MB 01:25
Complete!
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Qubes OS Repository for Dom0 81 MB/s | 209 kB 00:00
No package kernel-latest-4.14.12 available.
No package kernel-latest-qubes-vm-4.14.12 available.
Error: Unable to find a match.
sorandom
commented
Apr 2, 2018
•
edited
Edited 2 times
-
sorandom
edited
Apr 2, 2018 (most recent)
-
sorandom
edited
Apr 2, 2018
edited
-
Apr 2, 2018 (most recent) -
Apr 2, 2018
|
This did not work for me. I can get it to download but not install: |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Zrubi
Apr 3, 2018
Member
I do not know what is the real problem here, but as a workaround, you can try:
sudo qubes-dom0-update kernel-latest kernel-latest-qubes-vm
Then you can downgrade to the right one:
sudo qubes-dom0-update --action=downgrade kernel-latest-4.14.12 kernel-latest-qubes-vm-4.14.12
|
I do not know what is the real problem here, but as a workaround, you can try: Then you can downgrade to the right one: |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
sorandom
Apr 9, 2018
I was able to downgrade the VM kernel to 4.14.12, and now sys-net reliably freezes after every suspend, it's not intermittent as before. However, network attach/detach failure does not seem to be quite as big of an issue anymore, but still does happen. I'll keep testing.
sorandom
commented
Apr 9, 2018
•
edited
Edited 2 times
-
sorandom
edited
Apr 9, 2018 (most recent)
-
sorandom
edited
Apr 9, 2018
edited
-
Apr 9, 2018 (most recent) -
Apr 9, 2018
|
I was able to downgrade the VM kernel to 4.14.12, and now sys-net reliably freezes after every suspend, it's not intermittent as before. However, network attach/detach failure does not seem to be quite as big of an issue anymore, but still does happen. I'll keep testing. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Apr 19, 2018
Member
With fully updated 4.0 (as of today), and 4.14.34 kernel in VM, this still doesn't work. Minimal test case I have is simple: detach network from a VM (qvm-prefs testvm netvm '').
This supposedly complete successfully, but in fact only backend side is cleaned up, on fronted side there is still (not working now) eth0. Additionally, xenstored process use 100% CPU for some time (a minute or so).
Using sysrq+t I've found xenwatch kernel thread waiting on something during device removal:
[ 168.361621] xenbus S 0 21 2 0x80000000
[ 168.361630] Call Trace:
[ 168.361636] ? __schedule+0x3df/0x880
[ 168.361642] schedule+0x32/0x80
[ 168.361653] xenbus_thread+0x5d5/0x9b0
[ 168.361661] ? __wake_up_common+0x96/0x180
[ 168.361668] ? remove_wait_queue+0x60/0x60
[ 168.361675] kthread+0xff/0x130
[ 168.361686] ? xb_read+0x1b0/0x1b0
[ 168.361694] ? kthread_create_on_node+0x70/0x70
[ 168.361703] ret_from_fork+0x35/0x40
[ 168.361714] xenwatch D 0 22 2 0x80000000
[ 168.361722] Call Trace:
[ 168.361727] ? __schedule+0x3df/0x880
[ 168.361735] schedule+0x32/0x80
[ 168.361743] xennet_remove+0x7b/0x1b0 [xen_netfront]
[ 168.361756] ? remove_wait_queue+0x60/0x60
[ 168.361763] xenbus_dev_remove+0x4f/0xa0
[ 168.361771] device_release_driver_internal+0x157/0x220
[ 168.361782] bus_remove_device+0xe5/0x150
[ 168.361793] device_del+0x1cf/0x300
[ 168.361802] ? xenbus_dev_remove+0xa0/0xa0
[ 168.361811] device_unregister+0x16/0x60
[ 168.361821] xenbus_dev_changed+0x9f/0x1d0
[ 168.361829] ? finish_wait+0x3c/0x80
[ 168.361838] xenwatch_thread+0xcf/0x170
[ 168.362019] ? remove_wait_queue+0x60/0x60
[ 168.362026] kthread+0xff/0x130
[ 168.362032] ? find_watch+0x40/0x40
[ 168.362039] ? kthread_create_on_node+0x70/0x70
[ 168.362047] ret_from_fork+0x35/0x40
|
With fully updated 4.0 (as of today), and 4.14.34 kernel in VM, this still doesn't work. Minimal test case I have is simple: detach network from a VM ( |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
HW42
Apr 19, 2018
@marmarek: Can you confirm that this does not happen with 4.16.2? (At least for me that's the case).
HW42
commented
Apr 19, 2018
|
@marmarek: Can you confirm that this does not happen with 4.16.2? (At least for me that's the case). |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
HW42
commented
Apr 19, 2018
|
Ah, it seems this commit missed stable. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
marmarek
Apr 19, 2018
Member
@marmarek: Can you confirm that this does not happen with 4.16.2? (At least for me that's the case).
Yes, on 4.16.2 it works correctly.
Yes, on 4.16.2 it works correctly. |
added a commit
to HW42/qubes-linux-kernel
that referenced
this issue
Apr 19, 2018
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
qubesos-bot
Apr 21, 2018
Automated announcement from builder-github
The package kernel-4.14.35-1.pvops.qubes has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:
sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing
qubesos-bot
commented
Apr 21, 2018
|
Automated announcement from builder-github The package |
qubesos-bot
added
the
r4.0-dom0-cur-test
label
Apr 21, 2018
qubesos-bot
referenced this issue
in QubesOS/updates-status
Apr 21, 2018
Closed
linux-kernel v4.14.35-1 (r4.0) #489
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
qubesos-bot
Apr 21, 2018
Automated announcement from builder-github
The package kernel-4.14.35-1.pvops.qubes has been pushed to the r3.2 testing repository for dom0.
To test this update, please install it with the following command:
sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing
qubesos-bot
commented
Apr 21, 2018
|
Automated announcement from builder-github The package |
qubesos-bot
added
the
r3.2-dom0-cur-test
label
Apr 21, 2018
qubesos-bot
referenced this issue
in QubesOS/updates-status
Apr 21, 2018
Closed
linux-kernel v4.14.35-1 (r3.2) #490
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
qubesos-bot
May 14, 2018
Automated announcement from builder-github
The package kernel-4.14.35-1.pvops.qubes has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:
sudo qubes-dom0-update
Or update dom0 via Qubes Manager.
qubesos-bot
commented
May 14, 2018
|
Automated announcement from builder-github The package Or update dom0 via Qubes Manager. |
Zrubi commentedMar 5, 2018
Qubes OS version:
R3.2
(and probably latest 4.0-rc)
Steps to reproduce the behavior:
Changing the NetVM of a proxyVM, takes longer time than before, but
succeed - at least according to Qubes Manager (and qvm tools)
But after the change, no traffic visible on the virtual interfaces, so
no networking after that move.
There is no error messages related this issue.
But If I want to detach this bugged ProxyVM from the connected AppVM
(means: If try to change the connected AppVM's netVM) it is failing
with the following error message:
Internal error: libxenlight failed to detach network device
From this point you are not able to start any new apps from the bugged VM,
(Restarting the affected VMs helps, unbtil you not try to change the netVM)
It seem that in case of using 4.14.12 kernel in dom0 and in VM's
everything is working fine.
ANY newer kernel causing several xen related issues including the
problem I described before.