Make it easier to configure VM swap space

[Eric678]

Qubes OS version:

4.0

Affected component(s) or functionality:

core + Qubes Manager

---

Steps to reproduce the behavior:

Larger VMs need more than the standard 1G of swap otherwise kernel issues "out of memory" eventually.
All VM swap partitions are misaligned, starting directly at the end of the dos partition table in xvdc (sect 1).
All VM swaps do not have discard set to return unused pages to the thin pool, so if swap is turning over, even slowly, the kernel will colour all the pages of swap and keep all the disk space allocated.

Expected or desired behavior:

Given the ease of use of Qubes Manager for controlling resources of a VM I expect the swap space to be able to be set in Qubes Manager/Settings/Advanced along with private disk space. I suggest a line under private disk space that, in its simplest implementation, is red stared and only settable when the qube is not running. A more ambitious implementation would have the down tick greyed out when running but allow increasing on the fly with an additional swap partition created by the "Apply" button. Of course the property needs to be a exposed by qvm-prefs as well.

Since 4.1 is targeting usability - I suggest this is an appropriate enhancement, especially for those memory constrained.

Actual behavior:

As a new user it was difficult to find how to extend VM swap and it just seemed to belong on the Qubes Manager advanced settings page.
Misalignment slows down swap.
Not having discard enabled fills up your disk unnecessarily.

General notes:

My hack to get around these problems in the current 4.0 setup was to add this script to all my templates and call it from the top of rc.local with the appropriate required swap size (not exactly user friendly):

#!/bin/bash
# Set swap size during VM start (call from rc.local)
if [ $# != 1 ]; then
	echo "usage: $0 <size: 8MiB to 8GiB>">&2; exit 1
fi
case "$1" in
    *MiB) size="${1%MiB}" ;;
    *GiB) size=$((1024 * ${1%GiB})) ;;
    *) echo "$0: Please use MiB or GiB suffix for swap size">&2; exit 1 ;;
esac
set -e
if [ $size -lt 1024 ]; then
    /sbin/parted -s /dev/xvdc mkpart primary linux-swap 1028MiB $((1028 + $size))MiB
    /sbin/mkswap /dev/xvdc2
    /sbin/swapon -d /dev/xvdc2		# enable discard for thin prov
    /sbin/swapoff /dev/xvdc1
    /sbin/parted -s /dev/xvdc rm 1
else
    /sbin/swapoff /dev/xvdc1		# OK for larger VMs - plenty of ram
    /sbin/parted -s /dev/xvdc rm 1
    /sbin/parted -s /dev/xvdc mkpart primary linux-swap 4MiB $((4 + $size))MiB
    /sbin/mkswap /dev/xvdc1
    /sbin/swapon -d /dev/xvdc1
fi

Ended up doing all my sys+appVMs. Allocated swap disk space tracks swap used, lagging 60-100MB when declining. I also tested using all 10G of xvdc and it behaved as expected using:

/sbin/swapoff -a
/sbin/mkswap -f /dev/xvdc   # Zap partition table, aligned
/sbin/swapon -d /dev/xvdc   # Turn on discard pages for thin prov

Note systemd does try to remount xvdc1 a few times and I thought that was a lead into where to do this change properly... no luck there, also fstab is not updated.

---

I have consulted the following relevant documentation:

I could not find any answer to the question of extending VM swap, only others asking. I posted this topic to the mailing list: https://groups.google.com/forum/#!topic/qubes-users/AVimvVal47g and awokd did provide a reference to his IAQ that I did not originally find, even with a Duck search on the site. Unfortunately the answer was not that useful. @unman gave me the lead into a solution to the issue, thank you.
This has been running here since not long after that post.

I am aware of the following related, non-duplicate issues:

[Eric678]

Would one of the devs please advise why partition xvdc3 was added to a standard instal (apart from breaking this script)?
Adding /sbin/parted -s /dev/xvdc rm 3 before set -e seems not to cause any issues that I can see - just to keep this facility running on 4.0.
Thanks.

[marmarek]

The /dev/xvdc3 is there to ease using that space for something else - you can simply use that partition as is. Some use it for swap, others for bigger /tmp or /var/tmp, there are different opinions.
Also, it is called specifically xvdc3 not xvdc2 to make it not mix it up with a partition that is used in some cases (when root volume is really read only). So - if xvdc3 is there - use it, if not, it means there is no unused space on xvdc (and especially - do not remove any existing partition from there).

[Osndok]

FWIW: I really appreciate your effort in filing this ticket. Having just experienced the OOM killer tragically murder a valuable process (even with my 64gb of RAM), I was on the hunt to increase domN swap, and (assuming there are no contraindications) I think your three-line suggestion will work well for any vm that I get an OOM death!

[Osndok]

Just commenting to link #6659 as relevant/related.

[brendanhoar]

@marmarek

Couple questions on the way forward:

1. is there a proposed partition layout for using the volatile volume as copy-on-write backing for both appvs (swap and readonly root) and disposable VMs (swap and readonly root/personal)?

2. Will the total volatile volume size be determined dynamically during pre-VM-startup prep, e.g. based on default swap space, the existing root and (sometimes for disposables) existing personal volume sizes?

If so, perhaps user configured swap size in qvm-preferences/gui could be part of that approach, maybe with future support for additional user-customizable volatile partitions using GPT?

B

[marmarek]

1. is there a proposed partition layout for using the volatile volume as copy-on-write backing for both appvs (swap and readonly root) and disposable VMs (swap and readonly root/personal)?

Not yet. I've tried to use dm-thin to share CoW space between root and private (to sidestep having new partition layout there), but wasn't happy about it (startup performance among other issues).

2. Will the total volatile volume size be determined dynamically during pre-VM-startup prep, e.g. based on default swap space, the existing root and (sometimes for disposables) existing personal volume sizes?

I guess it should be. Currently it defaults to 10GB (or rather: same as default root volume size). But it can be resized with qvm-volume.

If so, perhaps user configured swap size in qvm-preferences/gui could be part of that approach, maybe with future support for additional user-customizable volatile partitions using GPT?

I'd like to keep this part as simple as possible (another reason why I didn't liked dm-thin on volatile volume). This includes more or less static partition table - it is created in initramfs on every VM startup, which means:

• it is critical for VM startup time
• any failure there fails the whole VM startup; introducing variable parts - including user-configurable parts - increase the risk of failure

That said, I think the layout can include a partition for user to use (xvdc3 is in practice such partition, since it is unused with r/w root volume) using remaining available space. And then, have a startup service later in the boot process that makes use of it, depending on user preference (be it extra swap, tmp, or anything else).