Long interval or forever since guest poweroff until Qubes announce VM shutdown

[logoerthiner1]

How to file a helpful issue

Qubes OS release

R4.1

Brief summary

In many various cases, when a VM guest has shutdown (in VM console log the systemd says that VM has shutdown), the dom0 will wait for a long time until announcing that the VM has shutdown. Obviously there are some other things that Qubes OS need to do.

Even on qvm-kill a VM, such time interval exists, where the VM cannot be shutdown any more nor cannot be started up.

Sometimes it seems that the VM is impossible to be admitted to be shutdown anymore and I have to reboot the whole system, for example when sys-net driver gets crazy and deadlocks the machine.

I do not understand where I can see relevant logs to check out what happens and what task is blocking the VM from being poweroff from the hypervisor side.

Steps to reproduce

Use the OS a little bit and shutdown various machines.

Expected behavior

For a short time after guest shutdowns, dom0 announces the VM shutdown and the VM can be started again.

Actual behavior

Sometimes for longer then 1 minutes after the guest shutdown, the guest is not fully shutdown and cannot be started up again.

[DemiMarie]

For a VM to be fully shut down, some cleanup needs to happen in dom0. In particular, the volumes used by the VM need to be either deleted or committed, depending on whether the volume is persistent or not. This can take some time, and at least some of it is necessary for e.g. cloning the VM to reflect the changes made before the VM shut down.

[logoerthiner1]

For a VM to be fully shut down, some cleanup needs to happen in dom0. In particular, the volumes used by the VM need to be either deleted or committed, depending on whether the volume is persistent or not. This can take some time, and at least some of it is necessary for e.g. cloning the VM to reflect the changes made before the VM shut down.

Good point. Now I wonder effective way of finding logs about the tasks Qubes OS need to finish between the time guest exits and the time Qubes OS announce that the VM exits. Sometimes it is lvm operations as you says, but I am afraid that sometimes it is not that simple. Any suggestions on the log hunting?

[DemiMarie]

For a VM to be fully shut down, some cleanup needs to happen in dom0. In particular, the volumes used by the VM need to be either deleted or committed, depending on whether the volume is persistent or not. This can take some time, and at least some of it is necessary for e.g. cloning the VM to reflect the changes made before the VM shut down.

Good point. Now I wonder effective way of finding logs about the tasks Qubes OS need to finish between the time guest exits and the time Qubes OS announce that the VM exits. Sometimes it is lvm operations as you says, but I am afraid that sometimes it is not that simple. Any suggestions on the log hunting?

journalctl -b -u libvirtd.service -u qubesd.service -f --no-tail?

[logoerthiner1]

For a VM to be fully shut down, some cleanup needs to happen in dom0. In particular, the volumes used by the VM need to be either deleted or committed, depending on whether the volume is persistent or not. This can take some time, and at least some of it is necessary for e.g. cloning the VM to reflect the changes made before the VM shut down.

Good point. Now I wonder effective way of finding logs about the tasks Qubes OS need to finish between the time guest exits and the time Qubes OS announce that the VM exits. Sometimes it is lvm operations as you says, but I am afraid that sometimes it is not that simple. Any suggestions on the log hunting?

journalctl -b -u libvirtd.service -u qubesd.service -f --no-tail?

Good tricks on journalctl. However the logs I saw there only contains that some xen bus is being removed.

I would rather want to know for example, the progress bar of each actions - for example the LVM snapshot removal. It would be better to provide a method of measuring how much work needs to be done in order to remove a specific snapshot, and the speed of the work being done.

I would be glad to hear suggestions about this.