'qvm-shutdown --all' jams up the system

[tasket]

Although I have had good luck with general system shutdown on R3.1rc2 and rc3, whenever I try to shutdown all vms for, say, maintenance the operation hangs. The only thing that happens is that all the vms are listed in the CLI, and they all go yellow in Manager... but then it will sit like that forever. Other qvm commands become unusable and so does Manager. Adding the --force option doesn't appear to help at all.

If I try to reboot or shutdown the whole system at this point, the system shutdown will also hang because of the qvm-shutdown jam.

I recall in earlier Qubes releases qvm-shutdown worked pretty well.

[marmarek]

Didi you have some DispVMs running at that time? There was a bug affecting qvm-shutdown on DispVM. Or simply install updates and check if the problem still happens.

[marmarek]

Didi you have some DispVMs running at that time? There was a bug affecting qvm-shutdown on DispVM. Or simply install updates and check if the problem still happens.

[tasket]

No dispvm any of those times. I'm also sure most of those sessions never ran a dispvm.
I'm on latest update, and the problem has existed for all of the 3.1 candidates. I know this because I also have a system shutdown script that includes qvm-shutdown to ensure the system will go down properly and without busy-waiting for 3 minutes; It never worked with 3.1rc*.

[tasket]

No dispvm any of those times. I'm also sure most of those sessions never ran a dispvm.
I'm on latest update, and the problem has existed for all of the 3.1 candidates. I know this because I also have a system shutdown script that includes qvm-shutdown to ensure the system will go down properly and without busy-waiting for 3 minutes; It never worked with 3.1rc*.

[andrewdavidwong]

Possibly related to #1320.

[andrewdavidwong]

Possibly related to #1320.

[andrewdavidwong]

I'm assuming this issue has been resolved based on the lack of recent activity. If not, please feel free to re-open it.

[andrewdavidwong]

I'm assuming this issue has been resolved based on the lack of recent activity. If not, please feel free to re-open it.

[tasket]

I'd like to re-open this. qvm-shutdown --all still hangs if there are more than 5 vms running, and @lorenzog says it happens to him #1581 (comment)

[tasket]

I'd like to re-open this. qvm-shutdown --all still hangs if there are more than 5 vms running, and @lorenzog says it happens to him #1581 (comment)

[andrewdavidwong]

Try excluding all ProxyVMs and NetVMs from the first pass of qvm-shutdown --all, then just qvm-shutdown --all should work.

[andrewdavidwong]

Try excluding all ProxyVMs and NetVMs from the first pass of qvm-shutdown --all, then just qvm-shutdown --all should work.

[lorenzog]

@andrewdavidwong that works but since most VMs depend either on the proxy or netVM, it just reduces the numbers of VMs to less than 5..

[lorenzog]

@andrewdavidwong that works but since most VMs depend either on the proxy or netVM, it just reduces the numbers of VMs to less than 5..

[andrewdavidwong]

I'm not sure what you mean. Are you saying that you have fewer than five non-proxy non-net VMs? Why is that a problem?

[andrewdavidwong]

I'm not sure what you mean. Are you saying that you have fewer than five non-proxy non-net VMs? Why is that a problem?

[lorenzog]

I'm saying that in my case the proxy VM can't shut down unless all VMs that use it are off. If I shut down the VMs depending on the proxy, then I'm left with less than five running VM (firewall, net, proxy, vault) and as such qvm-shutdown works flawlessly.

[lorenzog]

I'm saying that in my case the proxy VM can't shut down unless all VMs that use it are off. If I shut down the VMs depending on the proxy, then I'm left with less than five running VM (firewall, net, proxy, vault) and as such qvm-shutdown works flawlessly.

[tasket]

Perhaps we need a --kill option that takes effect after a set interval.

[tasket]

Perhaps we need a --kill option that takes effect after a set interval.

[andrewdavidwong]

Oh, I see.

[andrewdavidwong]

Oh, I see.

[andrewdavidwong]

(Reposting from #1581 (comment), in case it helps anyone here.)

FWIW, I've been using simple shutdown and reboot scripts to work around this issue (I think since I first started using Qubes). Here's an example:

#!/bin/bash
qvm-shutdown --wait --all \
--exclude=sys-net \
--exclude=sys-firewall \
--exclude=sys-usb \
--exclude=sys-whonix \
qvm-shutdown --wait --all
sleep 3
shutdown now

Works every time. The idea is to exclude all service VMs from the first pass of qvm-shutdown so that it doesn't jam up (#1826), so if you use VPN VMs, exclude those too.

[andrewdavidwong]

(Reposting from #1581 (comment), in case it helps anyone here.)

FWIW, I've been using simple shutdown and reboot scripts to work around this issue (I think since I first started using Qubes). Here's an example:

#!/bin/bash
qvm-shutdown --wait --all \
--exclude=sys-net \
--exclude=sys-firewall \
--exclude=sys-usb \
--exclude=sys-whonix \
qvm-shutdown --wait --all
sleep 3
shutdown now

Works every time. The idea is to exclude all service VMs from the first pass of qvm-shutdown so that it doesn't jam up (#1826), so if you use VPN VMs, exclude those too.

[andrewdavidwong]

In light of @lorenzog's comments above, I should mention that, in my preceding example, I left out some of the service VMs that I personally use. So, after the first pass, there are still more than 5 VMs running, and yet the system doesn't jam up.

[andrewdavidwong]

In light of @lorenzog's comments above, I should mention that, in my preceding example, I left out some of the service VMs that I personally use. So, after the first pass, there are still more than 5 VMs running, and yet the system doesn't jam up.

[tasket]

I think even without the above example, qvm-shutdown appears to be working much better now. The VPN VM may hang by itself for a while (weird Linux kernel issue, I think) but qvm-shutdown will eventually kill it. So qvm-shutdown itself isn't freezing for me anymore.

Also, one might even use this shorter command to quickly shut them all down:
qvm-kill VPN ; qvm-shutdown --all --wait

In KDE, you can even script this in System Settings / Startup and Shutdown so shutting down from the GUI is quick(er).

[tasket]

I think even without the above example, qvm-shutdown appears to be working much better now. The VPN VM may hang by itself for a while (weird Linux kernel issue, I think) but qvm-shutdown will eventually kill it. So qvm-shutdown itself isn't freezing for me anymore.

Also, one might even use this shorter command to quickly shut them all down:
qvm-kill VPN ; qvm-shutdown --all --wait

In KDE, you can even script this in System Settings / Startup and Shutdown so shutting down from the GUI is quick(er).

[entr0py]

The VPN VM may hang by itself for a while (weird Linux kernel issue, I think) but qvm-shutdown will eventually kill it.

Yes. This behavior started with the ~Dec 4 Qubes updates to R3.1. The openvpn process is not being killed properly and times out after 90 seconds. Manually killing the openvpn process and issuing shutdown command afterwards results in a normal speedy shutdown.

[entr0py]

The VPN VM may hang by itself for a while (weird Linux kernel issue, I think) but qvm-shutdown will eventually kill it.

Yes. This behavior started with the ~Dec 4 Qubes updates to R3.1. The openvpn process is not being killed properly and times out after 90 seconds. Manually killing the openvpn process and issuing shutdown command afterwards results in a normal speedy shutdown.

[tasket]

I think Linux+libvirt has an issue with shutdowns under certain circumstances. FWIW, I haven't experienced this issue in over 6 months, so it has improved to the point where I don't have to worry about the outcome if I just wait a minute or so... VM will shutdown anyway.

As for openvpn, this is probably a separate issue and probably has to do with Linux' handling of modified routing tables (guess).

[tasket]

I think Linux+libvirt has an issue with shutdowns under certain circumstances. FWIW, I haven't experienced this issue in over 6 months, so it has improved to the point where I don't have to worry about the outcome if I just wait a minute or so... VM will shutdown anyway.

As for openvpn, this is probably a separate issue and probably has to do with Linux' handling of modified routing tables (guess).