Qubes VM Manager in R3.0 is very slow and unresponsive compared to R2

[andrewdavidwong]

Background: I did a clean install of R3.0, then restored my AppVMs from a backup.

For example, selecting a VM in the list and clicking shutdown --> yes results in QVMM becoming unresponsive for a while. It takes about 3x longer to manually shut down a series of VMs using QVMM in R3.0 than it did in R2.

Is there any reason for this? Is it just me?

[marmarek]

The only thing we know is slower in R3.0 manager, is collecting VM stats (CPU, MEM). It is because libvirt API doesn't allow to fetch stats for all the VMs at once - needs to be done for each VM separately. We have some idea how to fix it in R4.0 (basically introducing qubesd daemon, which - among other things - would collect stats asynchronously).

But the above shouldn't affect VM operations - only updating table content.

[marmarek]

The only thing we know is slower in R3.0 manager, is collecting VM stats (CPU, MEM). It is because libvirt API doesn't allow to fetch stats for all the VMs at once - needs to be done for each VM separately. We have some idea how to fix it in R4.0 (basically introducing qubesd daemon, which - among other things - would collect stats asynchronously).

But the above shouldn't affect VM operations - only updating table content.

[andrewdavidwong]

https://groups.google.com/d/topic/qubes-devel/vNemSzo_Fek/discussion

SUBJECT: libvirt freeze after VM shutdown
FROM: hw42@ipsumj.de

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

when I shutdown a VM the qubes/qvm tools (qubes-manager, qvm-ls, ...)
become often (but not always) unresponsive for about 60 s.

After a bit debugging it seems the libvirt requests are blocking. And
indeed "virsh -c xen:/// list" blocks also. A strace of virsh shows
that it sends some request to the libvirt socket and then blocks while
waiting for an answer.

This problem is not new. If I remember correctly this always existed
in R3. I just haven't had time to debug.

The most reliable way to trigger to problem is to shutdown multiple
VMs at the same moment (for example via qvm-shutdown --all).

I observed that often the loop devices doesn't get cleaned up
correctly - but this might be a separate problem.

Does somebody observe the same problem? Has somebody already debugged
this?

Qubes-Issue #1288 might be related.

HW42
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJWOQI+AAoJEOSsySeKZGgWTUQQAKhX2O0gloC+W303H6gmBTJu
b6o1mpZybLmzqODHjcKX0szRoJ4fYaDW3BkDoBIPNc+pKl4aEV7YZ3lcxUr4pfcX
mvPiK3vRMtY83B2TdRDVxWopgRtW3KhzvlnmaQhb2MvkjeWJnFxeEXUpUC/lSPSo
K1JMAQFf9RogOgJmhCMX1k00DEBCdiIsU1zgyOOdxMBQdhPQRHgsuzvzQUF87gPz
LEjpN7301vLq5fOmKcV8L+NE08fwHLIozH+W7LIZPrqBS7sGRMeUMOE0sYnNGGBw
hOZDni0+WCrztUZKlfA9RVp4InsXJOGmysZSljSRk59gzqnSPGzzCM+cSxDzlSeL
qBpMMSA1hRy38wlZnJT97nnB6J/XlcDjRCvLI/OL4TFZPMWbA7Dk4/MVaaSggai7
cauMZ6qCm7kAAUA8Qq5f5laOOi3NESjNqaBAk1rTBnuwqnEjNNfrwRDfaNp+yoS8
jKEo+8WCVqMu2YjWoROkp6esIhpSHDFeC9y79UVFFcCuBkYMFdWGgOu5+u0Jsl9z
Y1Cqer+U/pLstzswbbKt8Qsi5qn1AKO3L81sV3heqg8pF1ATXDibbfvbU8IbuqVa
5G25oP2ahwyoAZ+4rdOQpZ6IZkKqOWDP0R8ej2HUTI41JCLxLbhsjzOMCeX1L7vM
lIIQhgygmmfqpaM98V7C
=xZZu
-----END PGP SIGNATURE-----

[andrewdavidwong]

https://groups.google.com/d/topic/qubes-devel/vNemSzo_Fek/discussion

SUBJECT: libvirt freeze after VM shutdown
FROM: hw42@ipsumj.de

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

when I shutdown a VM the qubes/qvm tools (qubes-manager, qvm-ls, ...)
become often (but not always) unresponsive for about 60 s.

After a bit debugging it seems the libvirt requests are blocking. And
indeed "virsh -c xen:/// list" blocks also. A strace of virsh shows
that it sends some request to the libvirt socket and then blocks while
waiting for an answer.

This problem is not new. If I remember correctly this always existed
in R3. I just haven't had time to debug.

The most reliable way to trigger to problem is to shutdown multiple
VMs at the same moment (for example via qvm-shutdown --all).

I observed that often the loop devices doesn't get cleaned up
correctly - but this might be a separate problem.

Does somebody observe the same problem? Has somebody already debugged
this?

Qubes-Issue #1288 might be related.

HW42
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJWOQI+AAoJEOSsySeKZGgWTUQQAKhX2O0gloC+W303H6gmBTJu
b6o1mpZybLmzqODHjcKX0szRoJ4fYaDW3BkDoBIPNc+pKl4aEV7YZ3lcxUr4pfcX
mvPiK3vRMtY83B2TdRDVxWopgRtW3KhzvlnmaQhb2MvkjeWJnFxeEXUpUC/lSPSo
K1JMAQFf9RogOgJmhCMX1k00DEBCdiIsU1zgyOOdxMBQdhPQRHgsuzvzQUF87gPz
LEjpN7301vLq5fOmKcV8L+NE08fwHLIozH+W7LIZPrqBS7sGRMeUMOE0sYnNGGBw
hOZDni0+WCrztUZKlfA9RVp4InsXJOGmysZSljSRk59gzqnSPGzzCM+cSxDzlSeL
qBpMMSA1hRy38wlZnJT97nnB6J/XlcDjRCvLI/OL4TFZPMWbA7Dk4/MVaaSggai7
cauMZ6qCm7kAAUA8Qq5f5laOOi3NESjNqaBAk1rTBnuwqnEjNNfrwRDfaNp+yoS8
jKEo+8WCVqMu2YjWoROkp6esIhpSHDFeC9y79UVFFcCuBkYMFdWGgOu5+u0Jsl9z
Y1Cqer+U/pLstzswbbKt8Qsi5qn1AKO3L81sV3heqg8pF1ATXDibbfvbU8IbuqVa
5G25oP2ahwyoAZ+4rdOQpZ6IZkKqOWDP0R8ej2HUTI41JCLxLbhsjzOMCeX1L7vM
lIIQhgygmmfqpaM98V7C
=xZZu
-----END PGP SIGNATURE-----

[adrelanos]

I can confirm the occasionally unresponsiveness.

[adrelanos]

I can confirm the occasionally unresponsiveness.

[marmarek]

Created #1383 for a libvirt freeze on VM shutdown. I think it is a subset of this issue, but not the only problem.

[marmarek]

Created #1383 for a libvirt freeze on VM shutdown. I think it is a subset of this issue, but not the only problem.

[andrewdavidwong]

I've noticed that it's not just Qubes Manager, but the entire system which slows down to a crawl whenever there is a CPU-intensive operation. For example, in R2, I would be able to generate a new DVM template, and the rest of the system would still remain responsive and usable. However, in R3.0, the whole system just locks up, and I can barely even move windows or switch virtual desktops. Even CPU-intensive processes inside AppVMs cause the whole host system to slow down much more noticeably than in R2. It's almost like a DoS attack. And this is all on a very powerful system with lots of RAM.

[andrewdavidwong]

I've noticed that it's not just Qubes Manager, but the entire system which slows down to a crawl whenever there is a CPU-intensive operation. For example, in R2, I would be able to generate a new DVM template, and the rest of the system would still remain responsive and usable. However, in R3.0, the whole system just locks up, and I can barely even move windows or switch virtual desktops. Even CPU-intensive processes inside AppVMs cause the whole host system to slow down much more noticeably than in R2. It's almost like a DoS attack. And this is all on a very powerful system with lots of RAM.

[andrewdavidwong]

Is this due to a change in the fundamental way processing power is allocated in R3.0? Personally, I would much rather give independent processes only limited processing power even if that means things take a little longer. I'd much rather retain "control" of the whole system and still be able to actually use it while I wait.

[andrewdavidwong]

Is this due to a change in the fundamental way processing power is allocated in R3.0? Personally, I would much rather give independent processes only limited processing power even if that means things take a little longer. I'd much rather retain "control" of the whole system and still be able to actually use it while I wait.

[marmarek]

Take a look at #1404. Do you have package referenced there installed?

[marmarek]

Take a look at #1404. Do you have package referenced there installed?

[andrewdavidwong]

Indeed, I have qubes-core-dom0-3.0.26-1.fc20.

[andrewdavidwong]

Indeed, I have qubes-core-dom0-3.0.26-1.fc20.

[qjoo]

This is a major annoyance. I'm hoping for an update on this issue since a long time.
Does upgrading to 3.1rc help? Are there any other known workarounds?

[qjoo]

This is a major annoyance. I'm hoping for an update on this issue since a long time.
Does upgrading to 3.1rc help? Are there any other known workarounds?

[andrewdavidwong]

R3.1 seems much faster in my experience so far. 👍

[andrewdavidwong]

R3.1 seems much faster in my experience so far. 👍

[andrewdavidwong]

Closing this since 3.0 is no longer supported, and this isn't an issue in 3.1 or 3.2 IME.

[andrewdavidwong]

Closing this since 3.0 is no longer supported, and this isn't an issue in 3.1 or 3.2 IME.