Cannot Start VMs

[stone212]

Qubes OS version (e.g., R3.2):

R3.2 64bit

Affected TemplateVMs (e.g., fedora-23, if applicable):

All templates affected

---

Expected behavior:

When I choose "Start/Resume VM" or open an application that requires a VM, the VM is expected to start.

Actual behavior:

The VM does not actually start. Instead, I get an error in the status area saying:

Error starting VM: <vm-name>
Cannot execute qrexec-daemon!

Steps to reproduce the behavior:

Perform any action that should start a VM. Including new VMs that you create after this error starts, old/existing VMs, template VMs, etc.

General notes:

EDIT: I have played with this more since writing, so let me add that I think the problem is that the sys-net VM will not start, and therefore the others won't start (giving the error message above). If I create a new NetVM and attach the firewall to it, then the other VMs start.

Except that sometimes if the moon is right and I reboot, the sys-net VM starts, and everything is good.

It is important to note that I followed the command line instructions here:
https://www.qubes-os.org/doc/anonymizing-your-mac-address/

Also, if I detach the network devices from the sys-net VM, then it starts and so does everything else. Without networking of course.

So I think the problem is that the setup in that link, when executed on a NetVM that has the network interfaces attached, combines to make that VM unbootable. And if other VMs depend on that VM, none of them are bootable.

I no longer think the issue has to do with creating and cancelling the creation of a VM as originally mentioned (below), except that when I rebooted the above problem magically started.

Original post follows:

Before I had this problem, I had been creating a new VM, and I hit "Cancel" halfway through the creation process because I realized that I was using the wrong template. But that process was buggy because when I tried to create the template again, although it was not in the list, I was not allowed to use the same name again. I got an error to the effect that the name is already in use.

At that point, I tried a reboot. The reboot took more than 5 minutes, so I manually turned the computer off by holding the power button.

Maybe it's significant that one domain that was open at the time of reboot currently has a log entry (qrexec.personal.log) saying:

domain dead
cannot connect to qrexec agent: No such process

However the other domains do not have this log entry.

---

Related issues:

Please see above. And thank you for reading.

[stone212]

I am closing this issue for the time being. Further playing indicates that I left out an important factor: I was sometimes booting with my wireless turned off. And I think that that was causing sys-net to fail to start, which was causing the others to fail to start.

Even if I turned wireless on after booting, sys-net wouldn't start. But if wireless is on before I boot, everything seems fine.

So I think the mac changer howto might need looking at, so that the VM can come up even if the wireless is off.

If I find that I was wrong about this (again) then I will re-open the issue. Thank you for reading and taking it seriously enough to read and assign to 3.2 updates.

[stone212]

I am closing this issue for the time being. Further playing indicates that I left out an important factor: I was sometimes booting with my wireless turned off. And I think that that was causing sys-net to fail to start, which was causing the others to fail to start.

Even if I turned wireless on after booting, sys-net wouldn't start. But if wireless is on before I boot, everything seems fine.

So I think the mac changer howto might need looking at, so that the VM can come up even if the wireless is off.

If I find that I was wrong about this (again) then I will re-open the issue. Thank you for reading and taking it seriously enough to read and assign to 3.2 updates.

[stone212]

It wasn't the wireless. I have tested a few times. Even with wireless enabled, sometimes sys-net does not start (with the MAC changer stuff from the link I posted earler.)

[stone212]

It wasn't the wireless. I have tested a few times. Even with wireless enabled, sometimes sys-net does not start (with the MAC changer stuff from the link I posted earler.)

[schnurentwickler]

Set the network-interface in sys-firewall to 'None' and try starting an AppVM. This information should shrink the errorzone.

Maybe you are using a problematic pci\usb device in netvm? This would be correlated to #1525.

[schnurentwickler]

Set the network-interface in sys-firewall to 'None' and try starting an AppVM. This information should shrink the errorzone.

Maybe you are using a problematic pci\usb device in netvm? This would be correlated to #1525.

[stone212]

Yes, that was good advice. Okay, two new items:

1. For any VM that would not start, I can make it start if I change its NetVM to "none"

2. The problem goes away if I completely shut down the computer and then start up cold. The problem only happens on soft reboot of the computer. So that means it is much less serious now. But still a bug.

Perhaps there is output from the shutdown process that you would like me to share? Or from the boot process?

[stone212]

Yes, that was good advice. Okay, two new items:

1. For any VM that would not start, I can make it start if I change its NetVM to "none"

2. The problem goes away if I completely shut down the computer and then start up cold. The problem only happens on soft reboot of the computer. So that means it is much less serious now. But still a bug.

Perhaps there is output from the shutdown process that you would like me to share? Or from the boot process?

[stone212]

Aaaannd... now I find that the problem sometimes happens on cold/hard reboot as well. So I have not yet found a pattern. I realize now that I did not give feedback to schnurentwickler's advice and I do not remember the result. I will try and report back next time I have the error.

[stone212]

Aaaannd... now I find that the problem sometimes happens on cold/hard reboot as well. So I have not yet found a pattern. I realize now that I did not give feedback to schnurentwickler's advice and I do not remember the result. I will try and report back next time I have the error.

[code9n]

I had the same problem - Recently downloaded Qubes 3.2, updated Fedora 23 and then sys-net gave the error message in question:

Error starting VM:
Cannot execute qrexec-daemon!

And all other net connected vms failed under it with the same message. The problem was still there after a cold boot.

But I'm using a desktop machine with an ethernet cable not wifi so I followed Stone212's ideas and removed the wireless network adapter from sys-net devices and then I could start everything up like normal.

Also it was still fine after another cold boot.

Hope it lasts...

[code9n]

I had the same problem - Recently downloaded Qubes 3.2, updated Fedora 23 and then sys-net gave the error message in question:

Error starting VM:
Cannot execute qrexec-daemon!

And all other net connected vms failed under it with the same message. The problem was still there after a cold boot.

But I'm using a desktop machine with an ethernet cable not wifi so I followed Stone212's ideas and removed the wireless network adapter from sys-net devices and then I could start everything up like normal.

Also it was still fine after another cold boot.

Hope it lasts...

[schnurentwickler]

Seems like you are using network devices with firmwares which does not support pci reset. (In example I cannot use my atheros card)
Look at: https://www.qubes-os.org/doc/wireless-troubleshooting/
And especially: https://www.qubes-os.org/doc/user-faq/#i-created-a-usbvm-and-assigned-usb-controllers-to-it-now-the-usbvm-wont-boot

[schnurentwickler]

Seems like you are using network devices with firmwares which does not support pci reset. (In example I cannot use my atheros card)
Look at: https://www.qubes-os.org/doc/wireless-troubleshooting/
And especially: https://www.qubes-os.org/doc/user-faq/#i-created-a-usbvm-and-assigned-usb-controllers-to-it-now-the-usbvm-wont-boot

[stone212]

@schnurentwickler You might be right but you did not explain that in any way, or how to troubleshoot. The first link you said talks about re-loading wireless drivers. But why would I have done this? The wireless was working perfectly well when I had the error.

The second link seems to be related to USBs, not VMs not starting? How is that connected to this issue? I did have that issue also, but the computer I was using did not have USB 3.0 so I don't think it is related to that link. And it's not related to this issue I don't think. Unless you have something to back up your statements?

[stone212]

@schnurentwickler You might be right but you did not explain that in any way, or how to troubleshoot. The first link you said talks about re-loading wireless drivers. But why would I have done this? The wireless was working perfectly well when I had the error.

The second link seems to be related to USBs, not VMs not starting? How is that connected to this issue? I did have that issue also, but the computer I was using did not have USB 3.0 so I don't think it is related to that link. And it's not related to this issue I don't think. Unless you have something to back up your statements?

[marmarek]

pci_strictreset may help with PCI devices not supporting reset (FLR or similar). This applies mostly to USB controllers, but sometimes other devices lack this feature too. Anyway, if device lack this feature, it would happen 100% reproducibly, not "sometimes". So, probably not this case.

I'd check /var/log/xen/console/guest-sys-net.log - maybe there is kernel-related startup error. If there are no new messages at all after failed startup, it may be about RAM - how much do you have? Try shutting down other VMs before starting sys-net.

[marmarek]

pci_strictreset may help with PCI devices not supporting reset (FLR or similar). This applies mostly to USB controllers, but sometimes other devices lack this feature too. Anyway, if device lack this feature, it would happen 100% reproducibly, not "sometimes". So, probably not this case.

I'd check /var/log/xen/console/guest-sys-net.log - maybe there is kernel-related startup error. If there are no new messages at all after failed startup, it may be about RAM - how much do you have? Try shutting down other VMs before starting sys-net.

[stone212]

@marmarek I won't be able to help you test. I gave up on Qubes a couple of months ago. I was getting too many errors and when I reported them I only got rude replies from the Qubes team, but no help troubleshooting. I couldn't use a system where I could only sometimes access my data, and never use my USB port.

Anyway, that message wouldn't help even though I know you were trying. I don't know what "reset" you mean, and I have no idea what "FLR" means. If you want the community to help you test (and that is the point of GitHub), you guys have to learn to communicate with people who are not developers, but still capable of troubleshooting. You also didn't mention which VM to check that log file in. The master (I forget what it's called) or the client?

The RAM was high, that wasn't the problem.

[stone212]

@marmarek I won't be able to help you test. I gave up on Qubes a couple of months ago. I was getting too many errors and when I reported them I only got rude replies from the Qubes team, but no help troubleshooting. I couldn't use a system where I could only sometimes access my data, and never use my USB port.

Anyway, that message wouldn't help even though I know you were trying. I don't know what "reset" you mean, and I have no idea what "FLR" means. If you want the community to help you test (and that is the point of GitHub), you guys have to learn to communicate with people who are not developers, but still capable of troubleshooting. You also didn't mention which VM to check that log file in. The master (I forget what it's called) or the client?

The RAM was high, that wasn't the problem.

[0spinboson]

"I don't know what "reset" you mean, and I have no idea what "FLR" means."

I appreciate that you still feel annoyed, but a., knowing what FLR stands for isn't really all that important (though you may have encountered it as part of an error message at the time), and b., it's extremely easy to 'google' for. So I don't really see how what you say is a sensible response to marmarek.
As for the log, it indeed is found in dom0 (the hypervisor) -- and even if it wasn't, since there are only two places to look, I don't really see the harm in not specifying it.

[0spinboson]

"I don't know what "reset" you mean, and I have no idea what "FLR" means."

I appreciate that you still feel annoyed, but a., knowing what FLR stands for isn't really all that important (though you may have encountered it as part of an error message at the time), and b., it's extremely easy to 'google' for. So I don't really see how what you say is a sensible response to marmarek.
As for the log, it indeed is found in dom0 (the hypervisor) -- and even if it wasn't, since there are only two places to look, I don't really see the harm in not specifying it.

[stone212]

Ha! I see that @andrewdavidwong has closed this, even though the issue wasn't resolved and that @0spinboson shared his feelings, even though it was unrelated to the ticket. At least @marmarek was trying, ineffective as it was. Can anyone be surprised that Qubes has the reputation it does? Wake me when you're ready to collaborate with the rest of the community.

[stone212]

Ha! I see that @andrewdavidwong has closed this, even though the issue wasn't resolved and that @0spinboson shared his feelings, even though it was unrelated to the ticket. At least @marmarek was trying, ineffective as it was. Can anyone be surprised that Qubes has the reputation it does? Wake me when you're ready to collaborate with the rest of the community.

[andrewdavidwong]

Ha! I see that @andrewdavidwong has closed this, even though the issue wasn't resolved and that @0spinboson shared his feelings, even though it was unrelated to the ticket. At least @marmarek was trying, ineffective as it was. Can anyone be surprised that Qubes has the reputation it does? Wake me when you're ready to collaborate with the rest of the community.

Should this issue not have been closed? As far as I can tell, no one is experiencing the issue any longer, and no one can reproduce it. Of course, if anyone starts to experience it again or can reproduce it, we'll reopen the issue, but it does us no good to keep the issue open if it's not actionable and not affecting any users.

I know that you were still experiencing the issue when you last used Qubes and that you've decided to stop using Qubes, and I respect that decision. However, since you were the only person still experiencing the issue (as far as I know), there's no one left who is affected by it or who can reproduce it. (Again, if that's not the case, I'm more than happy to reopen.)

In general, closing an issue shouldn't be interpreted as a permanent decision, an irreversible action, or even a serious pronouncement in itself. On the contrary, it's just a single click to reopen issues. Think of it more like having a discussion in which someone says, "Ok, so, the matter is settled then" (closing the issue). If you think the matter isn't yet settled and provide a good reason for thinking it's not, that person is likely to respond, "Oh, I suppose it's not actually settled yet then!" (reopening the issue).

[andrewdavidwong]

Ha! I see that @andrewdavidwong has closed this, even though the issue wasn't resolved and that @0spinboson shared his feelings, even though it was unrelated to the ticket. At least @marmarek was trying, ineffective as it was. Can anyone be surprised that Qubes has the reputation it does? Wake me when you're ready to collaborate with the rest of the community.

Should this issue not have been closed? As far as I can tell, no one is experiencing the issue any longer, and no one can reproduce it. Of course, if anyone starts to experience it again or can reproduce it, we'll reopen the issue, but it does us no good to keep the issue open if it's not actionable and not affecting any users.

I know that you were still experiencing the issue when you last used Qubes and that you've decided to stop using Qubes, and I respect that decision. However, since you were the only person still experiencing the issue (as far as I know), there's no one left who is affected by it or who can reproduce it. (Again, if that's not the case, I'm more than happy to reopen.)

In general, closing an issue shouldn't be interpreted as a permanent decision, an irreversible action, or even a serious pronouncement in itself. On the contrary, it's just a single click to reopen issues. Think of it more like having a discussion in which someone says, "Ok, so, the matter is settled then" (closing the issue). If you think the matter isn't yet settled and provide a good reason for thinking it's not, that person is likely to respond, "Oh, I suppose it's not actually settled yet then!" (reopening the issue).

[stone212]

@andrewdavidwong If that is how you think an issue tracker should be run, then I am not surprised that Qubes has such a poor reputation for fixing its issues. Honestly, I have never heard something so counter-productive (regarding bug tracking) before. No one said "ok, so, the issue is settled then" except you, when you closed the ticket before the bug was fixed.

[stone212]

@andrewdavidwong If that is how you think an issue tracker should be run, then I am not surprised that Qubes has such a poor reputation for fixing its issues. Honestly, I have never heard something so counter-productive (regarding bug tracking) before. No one said "ok, so, the issue is settled then" except you, when you closed the ticket before the bug was fixed.

[andrewdavidwong]

@andrewdavidwong If that is how you think an issue tracker should be run, then I am not surprised that Qubes has such a poor reputation for fixing its issues. Honestly, I have never heard something so counter-productive (regarding bug tracking) before. No one said "ok, so, the issue is settled then" except you, when you closed the ticket before the bug was fixed.

I think there's been a misunderstanding. In this context, saying, "Ok, so, the matter is settled then" isn't supposed to be something that anyone else says or a decree from me or anyone on the Qubes team. Rather, it's just the best-effort conclusion being drawn after hearing everyone's input. If it's the wrong conclusion, then it should (and will easily) be revised.

In this case, it seemed like a reasonable conclusion based on your statement that you wouldn't be able to help test this issue anymore and that you were no longer using Qubes. That, along with the fact that no one else reported being affected by it (except @code9n, who seemed to also report no longer being affected) strongly suggested that this issue had run into a dead end (and should therefore be closed). The act of closing the issue was not a final pronouncement of this. Rather, it was more likely than not the most useful state for the issue to be in, given the available evidence. Think of the act of closing the issue as a way of saying, "It seems to us that this issue should be closed. If anyone thinks otherwise, please signal that to us somehow, and we'll reopen it." Rather than explicitly saying this every time we close an issue, it's generally more efficient to just have a general understanding that it's fine to say, "Actually, I think this issue should be open. Here's why...." Nonetheless, I can certainly see the value in communicating the reason for closing and our willingness to re-open every time we close an issue. I usually say something to this effect when closing issues where that may be unclear, but perhaps I'll start doing it more consistently to avoid similar misunderstandings.

[andrewdavidwong]

@andrewdavidwong If that is how you think an issue tracker should be run, then I am not surprised that Qubes has such a poor reputation for fixing its issues. Honestly, I have never heard something so counter-productive (regarding bug tracking) before. No one said "ok, so, the issue is settled then" except you, when you closed the ticket before the bug was fixed.

I think there's been a misunderstanding. In this context, saying, "Ok, so, the matter is settled then" isn't supposed to be something that anyone else says or a decree from me or anyone on the Qubes team. Rather, it's just the best-effort conclusion being drawn after hearing everyone's input. If it's the wrong conclusion, then it should (and will easily) be revised.

In this case, it seemed like a reasonable conclusion based on your statement that you wouldn't be able to help test this issue anymore and that you were no longer using Qubes. That, along with the fact that no one else reported being affected by it (except @code9n, who seemed to also report no longer being affected) strongly suggested that this issue had run into a dead end (and should therefore be closed). The act of closing the issue was not a final pronouncement of this. Rather, it was more likely than not the most useful state for the issue to be in, given the available evidence. Think of the act of closing the issue as a way of saying, "It seems to us that this issue should be closed. If anyone thinks otherwise, please signal that to us somehow, and we'll reopen it." Rather than explicitly saying this every time we close an issue, it's generally more efficient to just have a general understanding that it's fine to say, "Actually, I think this issue should be open. Here's why...." Nonetheless, I can certainly see the value in communicating the reason for closing and our willingness to re-open every time we close an issue. I usually say something to this effect when closing issues where that may be unclear, but perhaps I'll start doing it more consistently to avoid similar misunderstandings.

[andrewdavidwong]

...when you closed the ticket before the bug was fixed.

There may be another underlying issue here, which we discussed in the comments on #3099. Our policy can't be that bug reports must remain open unless they're fixed by a patch. At a minimum, we have to be able to close bug reports that can't be reproduced. We also need to be able to close reports for bugs that affect only very specific hardware configurations that our team doesn't have access to, for example. More generally, though, the number of bug reports (and the rate at which new ones are added) vastly outstrips our developer-hours. Given the size of our team, we'll never be able to fix all the bugs, at least not anytime soon. We have to prioritize. Closing issues that our team won't be able to address (e.g., because there aren't any affected users who can help us test them, and it would be prohibitively time-consuming to try to go through all the permutations necessary to reproduce them ourselves) is one way to prioritize the more important issues. While we could leave them all open, the practical effect would be the same. We'd just have a lot of open issues that we never have time to address. (Well, we already do, so I suppose we'd just have even more of them.)

At any rate, I think there can be multiple reasonable approaches to a problem like this one (and to issue management in general), and if @rootkovska or @marmarek would like us to take a different approach, I'm happy to oblige.

[andrewdavidwong]

...when you closed the ticket before the bug was fixed.

There may be another underlying issue here, which we discussed in the comments on #3099. Our policy can't be that bug reports must remain open unless they're fixed by a patch. At a minimum, we have to be able to close bug reports that can't be reproduced. We also need to be able to close reports for bugs that affect only very specific hardware configurations that our team doesn't have access to, for example. More generally, though, the number of bug reports (and the rate at which new ones are added) vastly outstrips our developer-hours. Given the size of our team, we'll never be able to fix all the bugs, at least not anytime soon. We have to prioritize. Closing issues that our team won't be able to address (e.g., because there aren't any affected users who can help us test them, and it would be prohibitively time-consuming to try to go through all the permutations necessary to reproduce them ourselves) is one way to prioritize the more important issues. While we could leave them all open, the practical effect would be the same. We'd just have a lot of open issues that we never have time to address. (Well, we already do, so I suppose we'd just have even more of them.)

At any rate, I think there can be multiple reasonable approaches to a problem like this one (and to issue management in general), and if @rootkovska or @marmarek would like us to take a different approach, I'm happy to oblige.

[stone212]

@andrewdavidwong

think there can be multiple reasonable approaches to a problem like this one

Yes. None of which you have outlined. I really can't believe you and @0spinboson have turned this thread into a conversation, but I am just as guilty for responding so I'll continue. Issues/tickets are not a threat. They are the best means of improving the project. And the people who report them are resources to aid in that improvement. You have alienated one (me) and closed tickets that other people besides myself have said they have issues with. You should have used us as allies, not treated us like a threat, and you will continue to have trouble with man-hours until you see that. As it is, the project will move much more quickly as soon as the big roadblock - ie: you - is removed.

[stone212]

@andrewdavidwong

think there can be multiple reasonable approaches to a problem like this one

Yes. None of which you have outlined. I really can't believe you and @0spinboson have turned this thread into a conversation, but I am just as guilty for responding so I'll continue. Issues/tickets are not a threat. They are the best means of improving the project. And the people who report them are resources to aid in that improvement. You have alienated one (me) and closed tickets that other people besides myself have said they have issues with. You should have used us as allies, not treated us like a threat, and you will continue to have trouble with man-hours until you see that. As it is, the project will move much more quickly as soon as the big roadblock - ie: you - is removed.

[shyney7]

I'm having the same problem. I can't starte any VM. As soon as I start for example Firefox on Work VM a pop-up is showing that work VM will start but nothing happens!

[shyney7]

I'm having the same problem. I can't starte any VM. As soon as I start for example Firefox on Work VM a pop-up is showing that work VM will start but nothing happens!

[marmarek]

@shyney7 is sys-net and sys-firewall running when you try to start application in work VM? If not, try to start sys-net first.
Generally I guess the problem may be with sys-net. Other VMs are affected, because when you start a VM, it first starts VMs required for network access as set in VM settings: work requires sys-firewall, sys-firewall requires sys-net. Changing netvm to 'none' in VM setting would also allow to confirm this hypothesis.

[marmarek]

@shyney7 is sys-net and sys-firewall running when you try to start application in work VM? If not, try to start sys-net first.
Generally I guess the problem may be with sys-net. Other VMs are affected, because when you start a VM, it first starts VMs required for network access as set in VM settings: work requires sys-firewall, sys-firewall requires sys-net. Changing netvm to 'none' in VM setting would also allow to confirm this hypothesis.

[shyney7]

@marmarek only dom0 is running from the beginning. I can't start any VM. If I try to start for example sys-net there is an error saying: "Error starting VM: Interner Fehler: PCI-Gerät 0000:03:00.1 kann nicht zurückgesetzt werden: Interner Fehler: Aktive 0000:03:00.0 Geräte auf Bus mit 0000:03:00.1, Bus wird nicht zurückgesetzt"

Translation:
"Error starting VM: internal error: PCI-Device 0000:03:00.1 can't be reset: internal error: active 0000:03:00.0 devices on Bus with 0000:03:00.1, Bus will not be reset."

[shyney7]

@marmarek only dom0 is running from the beginning. I can't start any VM. If I try to start for example sys-net there is an error saying: "Error starting VM: Interner Fehler: PCI-Gerät 0000:03:00.1 kann nicht zurückgesetzt werden: Interner Fehler: Aktive 0000:03:00.0 Geräte auf Bus mit 0000:03:00.1, Bus wird nicht zurückgesetzt"

Translation:
"Error starting VM: internal error: PCI-Device 0000:03:00.1 can't be reset: internal error: active 0000:03:00.0 devices on Bus with 0000:03:00.1, Bus will not be reset."

[marmarek]

This looks to be a duplicate of #1393
In short: there are two devices sharing resources (in this case: two functions of the same device). You can either assign both of them to the same VM, or disable one of them. In #1393 it is about network adapter and card reader.

[marmarek]

This looks to be a duplicate of #1393
In short: there are two devices sharing resources (in this case: two functions of the same device). You can either assign both of them to the same VM, or disable one of them. In #1393 it is about network adapter and card reader.

[andrewdavidwong]

Closing as a duplicate of #1393. If you believe this is not really a duplicate, please leave a comment briefly explaining why. We'll be happy to take another look and, if appropriate, reopen this issue. Thank you.

[andrewdavidwong]

Closing as a duplicate of #1393. If you believe this is not really a duplicate, please leave a comment briefly explaining why. We'll be happy to take another look and, if appropriate, reopen this issue. Thank you.

[Seabass5701]

Just wanted to let everyone know that I had this same error

"Error starting VM: sys-net
Cannot execute qrexec-daemon!"

Once I removed the PCI Express Card Reader in the 'Devices' tab from 'Selected' to 'Available' in sys-net Qube settings, the problem solved itself automatically and completely, and sys-net started again normally.

[Seabass5701]

Just wanted to let everyone know that I had this same error

"Error starting VM: sys-net
Cannot execute qrexec-daemon!"

Once I removed the PCI Express Card Reader in the 'Devices' tab from 'Selected' to 'Available' in sys-net Qube settings, the problem solved itself automatically and completely, and sys-net started again normally.