Cannot start VMs with initial memory too low

[qavenull]

Qubes OS version:

R4.0

Affected TemplateVMs:

N/A

---

Steps to reproduce the behavior:

Reduce initial memory of a VM to a low value (for a fedora-26 template, 200 MB was low enough).

Expected behavior:

Either the VM starts, or a popup telling me I need to add more memory is displayed.

Actual behavior:

The VM does not start, I get a popup saying 'Cannot connect to qrexec agent: No such process' (and the same goes into the VM logs).

General notes:

I thought that I would not be able to reclaim any memory that's given at the VM startup, so I tried to lower it... Since then I noticed that it wasn't so bad to have more memory than needed at VM startup.

---

Related issues:

[taradiddles]

@andrewdavidwong - is it really a bug ? It's somewhat expected that a VM without enough memory would crash. Granted, displaying the cause of the failure to connect to the qrexsec agent would really be helpful but is it feasible technically ? Some out-of-memory errors can be seen in xen's hypervisor.log (eg. p2m_pod_demand_populate: Dom... out of PoD memory) but I doubt kernel panics, the lack of installed qrexec agent or other errors in VMs can be seen from dom0 (save for parsing the VM's console).

FWIW the log files listed in Qubes Manager aren't the ones I usually look at when debugging: /var/log/qubes/vm-vmname.log, /var/log/xen/console/hypervisor.log and /var/log/xen/console/guest-vmname-dm.log are missing...

Maybe a list of relevant log files should be displayed in the 'can't connect to qrexec' error message so that users would know where to look.

@qavenull : out of curiosity, do you see something in the logs mentioned above that is relevant to your VM when it fails to start ?
BTW the only reason (IMO) to decrease the minimum memory is when memory balancing isn't enabled (eg. VMs with PCI passthrough like sys-net). Otherwise the balancing algoritm works pretty well (you'll notice that a lot of memory is allocated after the VM boots but that it will decrease when the whole system is under memory "pressure" ; for instance my sys-firewall VM drops from ~800M to ~200M when I have a lot of VMs started).

[taradiddles]

@andrewdavidwong - is it really a bug ? It's somewhat expected that a VM without enough memory would crash. Granted, displaying the cause of the failure to connect to the qrexsec agent would really be helpful but is it feasible technically ? Some out-of-memory errors can be seen in xen's hypervisor.log (eg. p2m_pod_demand_populate: Dom... out of PoD memory) but I doubt kernel panics, the lack of installed qrexec agent or other errors in VMs can be seen from dom0 (save for parsing the VM's console).

FWIW the log files listed in Qubes Manager aren't the ones I usually look at when debugging: /var/log/qubes/vm-vmname.log, /var/log/xen/console/hypervisor.log and /var/log/xen/console/guest-vmname-dm.log are missing...

Maybe a list of relevant log files should be displayed in the 'can't connect to qrexec' error message so that users would know where to look.

@qavenull : out of curiosity, do you see something in the logs mentioned above that is relevant to your VM when it fails to start ?
BTW the only reason (IMO) to decrease the minimum memory is when memory balancing isn't enabled (eg. VMs with PCI passthrough like sys-net). Otherwise the balancing algoritm works pretty well (you'll notice that a lot of memory is allocated after the VM boots but that it will decrease when the whole system is under memory "pressure" ; for instance my sys-firewall VM drops from ~800M to ~200M when I have a lot of VMs started).

[qavenull]

I have nothing in /var/log/qubes/vm-vmname.log or /var/log/xen/console/hypervisor.log. However, /var/log/xen/console/guest-vmname-dm.log is more interesting,

log_oom_killer.txt

Two attempts are recorded in the file. OOM killer is mentioned, and the last messages don't seem to be seen in normal VM startups.

I only checked the /var/log/qubes directory, now that I know I should look into /var/log/xen debugging such problems will be easier! (Maybe displaying the list of files in the 'can't connect to qrexec' is indeed enough to help people dealing with the issues ?)

[qavenull]

I have nothing in /var/log/qubes/vm-vmname.log or /var/log/xen/console/hypervisor.log. However, /var/log/xen/console/guest-vmname-dm.log is more interesting,

log_oom_killer.txt

Two attempts are recorded in the file. OOM killer is mentioned, and the last messages don't seem to be seen in normal VM startups.

I only checked the /var/log/qubes directory, now that I know I should look into /var/log/xen debugging such problems will be easier! (Maybe displaying the list of files in the 'can't connect to qrexec' is indeed enough to help people dealing with the issues ?)

[marmarek]

However, /var/log/xen/console/guest-vmname-dm.log is more interesting,

This file really looks like /var/log/xen/console/guest-vmname.log (no -dm).

[marmarek]

However, /var/log/xen/console/guest-vmname-dm.log is more interesting,

This file really looks like /var/log/xen/console/guest-vmname.log (no -dm).

[qavenull]

Oops, you're right, there's no -dm file.

[qavenull]

Oops, you're right, there's no -dm file.

[andrewdavidwong]

is it really a bug ? It's somewhat expected that a VM without enough memory would crash. Granted, displaying the cause of the failure to connect to the qrexsec agent would really be helpful but is it feasible technically ? Some out-of-memory errors can be seen in xen's hypervisor.log (eg. p2m_pod_demand_populate: Dom... out of PoD memory) but I doubt kernel panics, the lack of installed qrexec agent or other errors in VMs can be seen from dom0 (save for parsing the VM's console).

The absence of a sufficiently informative error message could be a UX bug. However, for the reasons you point out, it may be that this is the best we can do. I leave it to the experts to determine whether this is the case.

[andrewdavidwong]

is it really a bug ? It's somewhat expected that a VM without enough memory would crash. Granted, displaying the cause of the failure to connect to the qrexsec agent would really be helpful but is it feasible technically ? Some out-of-memory errors can be seen in xen's hypervisor.log (eg. p2m_pod_demand_populate: Dom... out of PoD memory) but I doubt kernel panics, the lack of installed qrexec agent or other errors in VMs can be seen from dom0 (save for parsing the VM's console).

The absence of a sufficiently informative error message could be a UX bug. However, for the reasons you point out, it may be that this is the best we can do. I leave it to the experts to determine whether this is the case.

[marmarta]

Unfortunately, it is the best we can do. The popup will now not vanish and have path to the log file ( commit QubesOS/qubes-desktop-linux-manager@0e52962 ).

[marmarta]

Unfortunately, it is the best we can do. The popup will now not vanish and have path to the log file ( commit QubesOS/qubes-desktop-linux-manager@0e52962 ).