Qubes Manager unusable (no new VMs, stale state) after Whoops

[qjoo]

when closing firefox in a dispvm with ctrl-w -> window closes but dispvm is still running in qubes manager. Starting new dispvms works but they do no longer show up in QM after this bug has been triggered (dispvm windows themselfs show up and work fine).

In case this is relevant:

• my dispvms have the disable-default-route and disable-dns-server services enabled.

    Whoops. A critical error has occured. This is most likely a bug in Qubes Manager.

    libvirtError: Cannot recv data: Connection reset by peer
    at line 680
    of file /usr/lib64/python2.7/site-packages/qubesmanager/main.py.


    ----
    line: if ret == -1: raise libvirtError ('virDomainIsActive() failed', dom=self)
    func: isActive
    line no.: 1268
    file: /usr/lib64/python2.7/site-packages/libvirt.py
    ----
    line: if self.libvirt_domain.isActive():
    func: is_running
    line no.: 852
    file: /usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py
    ----
    line: if not vm.is_running():
    func: block_list_vm
    line no.: 241
    file: /usr/lib64/python2.7/site-packages/qubes/qubesutils.py
    ----
    line: devices_list.update(block_list_vm(vm, system_disks))
    func: block_list
    line no.: 324
    file: /usr/lib64/python2.7/site-packages/qubes/qubesutils.py
    ----
    line: blk = qubesutils.block_list(self.qvm_collection)
    func: update
    line no.: 81
    file: /usr/lib64/python2.7/site-packages/qubesmanager/block.py
    ----
    line: self.update()
    func: check_for_updates
    line no.: 67
    file: /usr/lib64/python2.7/site-packages/qubesmanager/block.py
    ----
    line: res, msg = self.blk_manager.check_for_updates()
    func: update_block_devices
    line no.: 808
    file: /usr/lib64/python2.7/site-packages/qubesmanager/main.py
    ----
    line: update_devs = self.update_block_devices() or out_of_schedule
    func: update_table
    line no.: 680
    file: /usr/lib64/python2.7/site-packages/qubesmanager/main.py

[marmarek]

This actually looks like some libvirt crash. Check /var/log/messages in
dom0, I guess you'll find there something like this:

libvirtd: libxl_internal.h:2784: libxl__ctx_unlock: Assertion `!r'
failed

After this happens, libvirtd is restarted automatically, but Qubes
Manager do not reconnect to the new instance. Restarting Qubes Manager
should help.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

This actually looks like some libvirt crash. Check /var/log/messages in
dom0, I guess you'll find there something like this:

libvirtd: libxl_internal.h:2784: libxl__ctx_unlock: Assertion `!r'
failed

After this happens, libvirtd is restarted automatically, but Qubes
Manager do not reconnect to the new instance. Restarting Qubes Manager
should help.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[qjoo]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Marek Marczykowski-Górecki:

This actually looks like some libvirt crash. Check
/var/log/messages in dom0, I guess you'll find there something
like this: libvirtd: libxl_internal.h:2784: libxl__ctx_unlock: Assertion!r' failed`

I can confirm having such a line in my logfile.
(Can not say whether that was generated at the time of the described
error, but probably.)

After this happens, libvirtd is restarted automatically, but Qubes
Manager do not reconnect to the new instance. Restarting Qubes
Manager should help.

Should this be reported upstream or how do you want to handle such
problems? Does Qubes 3.0 use an unmodified libvirtd?
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJVSR+HAAoJENGIB/ssoMC2lwkQAKr2t2mXDDjw2EmSkJB/ccXH
IxD0dxAyFxhFrgKp4Yijabr5cbjytkyHY98rLkzZz8vLNp8TU5CKKCwQYchrR3OA
gN/34DZIw22SVNBqcURWMckxL66l+Ruau9/EZQDBYgY+4DRXB9oIsutRMjvtuCoI
75YPcrMXXWZwWudauAkEPzsKHFrTIhvQzbRzlrX3M4Jp8McrDlfXU+AqupqagZV2
+GK6jzpWMmXPISV4eaWXAbawaAU/Z/Q8fp+DFPHrtQL9VLNNIJCHOzL7ch3GtqeA
/QfmoqbiLu0SIDl6o2wr0axYcj2sNPMBVFqoTzVATjuyljCyC4I/S/9hdN2jvsGC
DZPR4pzskzr/z1KSioqaZXkN6AJaxwGCJk6Tn6kqXVC50GlaZAzNMC7RT2PxkzTc
ORhcJDcpgEVs9qCSR83NPWGknbRQ3rA7PGhbFhiUMTSgGADjjEQEiKN8J9lc3ekP
XO6GuJDGxx5Mur7OzNYAweffL6x6aD+LZlYc9+tueprUIexdhjvoLAoqOuBRNXKq
1jx410s4PZtAMaHFXtpLy+TaNAV2rZwHlFFStPPmEHDpCF9kTpGGo9++HPSh1ojj
qc89wN4THp1qK5DekK2d87TN1fKaf8J0fjX7Cyh5l++YHAff1BN2qQaOgr/y+oqB
pykbjjC2545cAfiW627X
=B3De
-----END PGP SIGNATURE-----

[qjoo]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Marek Marczykowski-Górecki:

This actually looks like some libvirt crash. Check
/var/log/messages in dom0, I guess you'll find there something
like this: libvirtd: libxl_internal.h:2784: libxl__ctx_unlock: Assertion!r' failed`

I can confirm having such a line in my logfile.
(Can not say whether that was generated at the time of the described
error, but probably.)

After this happens, libvirtd is restarted automatically, but Qubes
Manager do not reconnect to the new instance. Restarting Qubes
Manager should help.

Should this be reported upstream or how do you want to handle such
problems? Does Qubes 3.0 use an unmodified libvirtd?
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJVSR+HAAoJENGIB/ssoMC2lwkQAKr2t2mXDDjw2EmSkJB/ccXH
IxD0dxAyFxhFrgKp4Yijabr5cbjytkyHY98rLkzZz8vLNp8TU5CKKCwQYchrR3OA
gN/34DZIw22SVNBqcURWMckxL66l+Ruau9/EZQDBYgY+4DRXB9oIsutRMjvtuCoI
75YPcrMXXWZwWudauAkEPzsKHFrTIhvQzbRzlrX3M4Jp8McrDlfXU+AqupqagZV2
+GK6jzpWMmXPISV4eaWXAbawaAU/Z/Q8fp+DFPHrtQL9VLNNIJCHOzL7ch3GtqeA
/QfmoqbiLu0SIDl6o2wr0axYcj2sNPMBVFqoTzVATjuyljCyC4I/S/9hdN2jvsGC
DZPR4pzskzr/z1KSioqaZXkN6AJaxwGCJk6Tn6kqXVC50GlaZAzNMC7RT2PxkzTc
ORhcJDcpgEVs9qCSR83NPWGknbRQ3rA7PGhbFhiUMTSgGADjjEQEiKN8J9lc3ekP
XO6GuJDGxx5Mur7OzNYAweffL6x6aD+LZlYc9+tueprUIexdhjvoLAoqOuBRNXKq
1jx410s4PZtAMaHFXtpLy+TaNAV2rZwHlFFStPPmEHDpCF9kTpGGo9++HPSh1ojj
qc89wN4THp1qK5DekK2d87TN1fKaf8J0fjX7Cyh5l++YHAff1BN2qQaOgr/y+oqB
pykbjjC2545cAfiW627X
=B3De
-----END PGP SIGNATURE-----

[marmarek]

On Tue, May 05, 2015 at 12:52:51PM -0700, qjoo wrote:

Marek Marczykowski-Górecki:

This actually looks like some libvirt crash. Check
/var/log/messages in dom0, I guess you'll find there something
like this: libvirtd: libxl_internal.h:2784: libxl__ctx_unlock: Assertion!r' failed`

I can confirm having such a line in my logfile.
(Can not say whether that was generated at the time of the described
error, but probably.)

After this happens, libvirtd is restarted automatically, but Qubes
Manager do not reconnect to the new instance. Restarting Qubes
Manager should help.

Should this be reported upstream or how do you want to handle such
problems? Does Qubes 3.0 use an unmodified libvirtd?

We have some patches on both libvirtd and libxl, but very unlikely there
is a cause (it doesn't touch anything lock related).
I've seen some rework of libxl context handling in newer libvirt version

• there is a chance it is already fixed there. Needs further
  investigation.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Tue, May 05, 2015 at 12:52:51PM -0700, qjoo wrote:

Marek Marczykowski-Górecki:

This actually looks like some libvirt crash. Check
/var/log/messages in dom0, I guess you'll find there something
like this: libvirtd: libxl_internal.h:2784: libxl__ctx_unlock: Assertion!r' failed`

I can confirm having such a line in my logfile.
(Can not say whether that was generated at the time of the described
error, but probably.)

After this happens, libvirtd is restarted automatically, but Qubes
Manager do not reconnect to the new instance. Restarting Qubes
Manager should help.

Should this be reported upstream or how do you want to handle such
problems? Does Qubes 3.0 use an unmodified libvirtd?

We have some patches on both libvirtd and libxl, but very unlikely there
is a cause (it doesn't touch anything lock related).
I've seen some rework of libxl context handling in newer libvirt version

• there is a chance it is already fixed there. Needs further
  investigation.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

This bug is about qubes-manager not reconnecting to libvirtd after crash/restart.
I'll leave it here, but if we solve underlying issue (#995), this will most likely be closed as 'wontfix', or moved to next release.

[marmarek]

This bug is about qubes-manager not reconnecting to libvirtd after crash/restart.
I'll leave it here, but if we solve underlying issue (#995), this will most likely be closed as 'wontfix', or moved to next release.

[mfc]

This actually looks like some libvirt crash. Check /var/log/messages in dom0, I guess you'll find there something like this: libvirtd: libxl_internal.h:2784: libxl__ctx_unlock: Assertion!r' failed`

I can confirm having such a line in my logfile.

I get this same crash (libvirtError: internal error: client socket is closed) upon creating a DispVM but do not have this unlock error in my logs.

The crash message in Qubes VM Manager is:

----
line: if ret == -1: raise libvirtError ('virDomainIsActive() failed', dom=self)
func: isActive
line no.: 1338
file: /usr/lib64/python2.7/site-packages/libvirt.py
----
line: if libvirt_domain.isActive():
func: get_power_state
line no.: 852
file: /usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py
----
line: state = vm.get_power_state()
func: update_table
line no.: 698
file: /usr/lib64/python2.7/site-packages/qubesmanager/main.py

If I try to close the Qubes VM Manager from the systray, the system draws the right-click box without content inside it, the system hangs, and ultimately the machine must be rebooted.

In addition (potentially related, potentially not): default whonix gateway proxyvm starts yellow on boot, must be restarted -- however it cannot be started on its own (throws an error), must be started by starting an AppVM that is dependent on the whonix gateway proxyvm. Only then does it successfully start.

Qubes 3.1rc1

[mfc]

This actually looks like some libvirt crash. Check /var/log/messages in dom0, I guess you'll find there something like this: libvirtd: libxl_internal.h:2784: libxl__ctx_unlock: Assertion!r' failed`

I can confirm having such a line in my logfile.

I get this same crash (libvirtError: internal error: client socket is closed) upon creating a DispVM but do not have this unlock error in my logs.

The crash message in Qubes VM Manager is:

----
line: if ret == -1: raise libvirtError ('virDomainIsActive() failed', dom=self)
func: isActive
line no.: 1338
file: /usr/lib64/python2.7/site-packages/libvirt.py
----
line: if libvirt_domain.isActive():
func: get_power_state
line no.: 852
file: /usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py
----
line: state = vm.get_power_state()
func: update_table
line no.: 698
file: /usr/lib64/python2.7/site-packages/qubesmanager/main.py

If I try to close the Qubes VM Manager from the systray, the system draws the right-click box without content inside it, the system hangs, and ultimately the machine must be rebooted.

In addition (potentially related, potentially not): default whonix gateway proxyvm starts yellow on boot, must be restarted -- however it cannot be started on its own (throws an error), must be started by starting an AppVM that is dependent on the whonix gateway proxyvm. Only then does it successfully start.

Qubes 3.1rc1

[marmarek]

On Mon, Jan 04, 2016 at 03:08:07AM -0800, Michael Carbone wrote:

If I try to close the Qubes VM Manager from the systray, the system draws the right-click box without content inside it, the system hangs, and ultimately the machine must be rebooted.

This is some Qubes Manager hang, while holding X mouse grab, so
effectively blocking the whole X server. But you can resolve this by
switching to text console, logging as user there and killing
qubes-manager process.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Mon, Jan 04, 2016 at 03:08:07AM -0800, Michael Carbone wrote:

If I try to close the Qubes VM Manager from the systray, the system draws the right-click box without content inside it, the system hangs, and ultimately the machine must be rebooted.

This is some Qubes Manager hang, while holding X mouse grab, so
effectively blocking the whole X server. But you can resolve this by
switching to text console, logging as user there and killing
qubes-manager process.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Mon, Jan 04, 2016 at 03:08:07AM -0800, Michael Carbone wrote:

In addition (potentially related, potentially not): default whonix gateway proxyvm starts yellow on boot, must be restarted -- however it cannot be started on its own (throws an error), must be started by starting an AppVM that is dependent on the whonix gateway proxyvm. Only then does it successfully start.

Can you elaborate about that error? Haven't seen anything like that.

Also on my system sys-whonix starts just fine on system boot (green).
Check sudo systemctl status qubes-netvm in dom0 for details.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Mon, Jan 04, 2016 at 03:08:07AM -0800, Michael Carbone wrote:

In addition (potentially related, potentially not): default whonix gateway proxyvm starts yellow on boot, must be restarted -- however it cannot be started on its own (throws an error), must be started by starting an AppVM that is dependent on the whonix gateway proxyvm. Only then does it successfully start.

Can you elaborate about that error? Haven't seen anything like that.

Also on my system sys-whonix starts just fine on system boot (green).
Check sudo systemctl status qubes-netvm in dom0 for details.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[mfc]

Can you elaborate about that error? Haven't seen anything like that.

For sys-whonix = whonix and whonix-dependent appvm: tor-personal-debian, it threw the following error:

Error while starting the 'tor-personal-debian' VM: Failed to kill old QubesDB instance (PID 2526). Terminate it manually and retry. If that isn't QubesDB process, remove the pidfile: /var/run/qubes/qubesdb.whonix.pid

The computer froze (I couldn't switch to text console) in this case.

sys-whonix actually works even though it is yellow so I am just not touching upon reboot, and not doing DispVMs right now.

[mfc]

Can you elaborate about that error? Haven't seen anything like that.

For sys-whonix = whonix and whonix-dependent appvm: tor-personal-debian, it threw the following error:

Error while starting the 'tor-personal-debian' VM: Failed to kill old QubesDB instance (PID 2526). Terminate it manually and retry. If that isn't QubesDB process, remove the pidfile: /var/run/qubes/qubesdb.whonix.pid

The computer froze (I couldn't switch to text console) in this case.

sys-whonix actually works even though it is yellow so I am just not touching upon reboot, and not doing DispVMs right now.