implement live boot by porting grub-live to Qubes - amnesia / non-persistent boot / anti-forensics

[adrelanos]

https://github.com/Kicksecure/grub-live [0] is a tiny, simple package with a few dependencies (available from packages.debian.org) that can be installed on Debian [1] hosts which adds a live boot menu entry, which means, the operating system gets booted non-persistently. All changes go to RAM instead of disk.

It might not be perfectly amnesic as per [1], [2] but it might be a huge step towards amnesic Qubes?

Not tested for anti-forensics yet but should be doable to test. [4]

Could grub-live be ported to Qubes? Am I overlooking something?

//cc @Algernon-01 (original author of grub-live)

• [0] https://forums.whonix.org/t/whonix-live-mode/3894/121
• [1] And likely Debian based and other grub based distributions.
• [2] Wipe RAM on shutdown #1562
• [3] Wipe Video RAM on Shutdown #1563
• [4] https://www.whonix.org/wiki/Dev/Technical_Introduction#Forensics

[marmarek]

I haven't looked at this in details, but I guess it's similar thing was done for Qubes live (a device mapper or overlayfs with a writable layer in ram). And the main problem there is the amount of changes you need to keep in ram. When normal live system would generate for example 300MB of delta, in Qubes you have this multiplied by a number of VMs (dom0 + sys-net,sys-firewall,sys-whonix and possibly sys-usb). This added to standard Qubes memory requirements, result in something not feasible to run on broadly available machines. As far I remember development of Qubes 3.2 live, on a hardware with 8GB you run out of memory in minutes of usage.
A large part of making Qubes live usable, is making amount of changes as small as possible. For example disable most of logging in all the VMs, updates checks, rebuilding caches etc.

[Algernon-01]

@marmarek

I tried the official USB live image from the Qubes website in a VM an it indeed just crashes after a while just by starting and stopping a few VMs. I think this is related to how live systems are normally implemented in Fedora. They make use of device mapper which writes changed blocks to RAM but the problem is it never gives them back even when they are not used anymore. In contrast overlayfs works on the file level and does not have this issue. A disadvantage is, however, that files which are opened for writing or just by changing permissions will first all be copied to RAM. I think this would be problematic for large files i.e. HVMs which could then easily occupy all of the available memory (unless they are configured as a live system themselves)
The default size for the device mapper overlay in RAM back then was 512MB but I think it has been increased recently. Also the images by @unman set it to a higher value.
There is also the rd.live.overlay.thin=1 option for dracut which could help, but when I tested it a while back I could not see any differences.
Since a while dracut also supports overlayfs as alternative to device mapper.
It depends also on what kind of live system you want, iso or normal filesystem image.
The fedora live systems have quite a different layout than Debian ones, but with the Debian live tools you can also create live systems which are not just an iso but something like a normal OS installation.
There is a package in Debian called rootovl which can be used on Qubes to create a live root filesystem with overlayfs. I actually tried it with 3.2 already a while back and it generally works. I did not do much testing though beyond "it boots and runs some VMs". I can try with 4.0 again and report back with some instructions when it works.
I don't know how it will behave over a day though with new VMs being created/destroyed.

I think logging itself would not create too much overhead. But if each VM downloads its own repo files then this could of course easily occupy some hundred MB. Could you maybe give a number what amount of data a Qubes Fedora or Debian VM downloads automatically?

[marmarek]

overlayfs with disk images doesn't sound good. It would be copied to RAM on very small modification. This is especially a problem for TemplateVMs.
For TemplateBasedVMs, it should be easy to configure them to write just to "volatile" and "private" volumes, which would hold only actual modifications/data, not a copy of unchanged data. But still that amount of data times number of VMs require significant amount of RAM. For example volatile image of my mail VM with 8 days uptime takes about 700MB, or volatile image of a DispVM with just Firefox and 3 days uptime takes 800MB.

Anyway, optimizing our templates to write less data would be beneficial not only to Live USB system, but for general use too.

[Algernon-01]

I tried it with Qubes 4.0 now and actually just the mounted rootfs is kind of a live system i.e. dom0.
Changes in a normal VM are persistent. Probably because they are stored as LVs and not as part of the root filesystem.
What does volatile mean, can writes go to RAM? A quick search on that or on some combination of overlayfs + lvm did not give anything useful.

[marmarek]

volatile image is one of disk images connected to every VM, see here for details.

As for Qubes 4.0, you can try installing it with btrfs storage layout - it will use files to store VMs data instead of LVM.

[Algernon-01]

I tested an installation of 4.0 with just /boot and encrypted / with ext4. In this case also all VMs are stored in the main partition.
dom0 is amnesic and I guess the VMs are too, but the problem is due to overlayfs and the general VM setup (creating snapshots, opening VM files as rw) I run out of RAM. So not a single VM can start.
I know that the setup of the block devices is part of the core structure and unlikely to change.
A debian or fedora HVM with just a single VM disk would be the easiest to setup as a live system.
I guess it should also work to mount an overlayfs on top of each partition which would need to be writable i.e. volatile and private.img. But those still need to be (re)created in the first place on the host which is a live system. So probably does not make much sense.

In general every file operation which opens large files for writing would need to be disabled for such a live system to be feasible. Or alternatively maybe the size of the images which need to be writable needs to be reduced significantly. Do you see any chance for those options?
Another possible requirement would be some kind of mechanism to change how the VM boots i.e. when the host boots live, the kernel command line of the VM would change so that this also boots live.

[marmarek]

You can find here my (very) old attempts at making it work, by selectively handling some files with overlayfs, some with dm-snapshot and some storing in tmpfs directly. Even with the problem of copying whole disk images to RAM on first open reduced to minimum, it was barely usable as described in #4982 (comment)