Coping with OS-level snapshot rotation

[tasket]

@tlaurion Your video presentation clarified the OS snapshot rotation vs. receive conflict for me, so I decided to open an issue.

A good example of the problem is when Qubes VM is running during a restore to the canonical volume name for that VM (or those rare times when Qubes system halted and left the VM snapshots in inconsistent state). The subsequent snapshot rotation by the OS can take the restored version out of service.

Possibilities:

1. Restore to a volume named with a suffix like "-restored" that the OS recognizes as taking precedence over "-back" volumes during rotation.
2. Let user specify some guard conditions, such as presence of "-back"; Then --force would be required to proceed with receive.
3. Test for live running VM. Could be combined with number 2.

Some of these measures could become active on condition that Qubes admin environment is detected.

For the suffix idea, this is very simple/solid implementation if the OS/service meets us half-way.

PS - If there is some kind of "best practice" or convention for dealing with this situation, I'm unaware of it but am happy to learn.

[tlaurion]

root-revert

An revert LVM snapshot of root LVM can be created in RW mode from prior of wyng restoration (receive) which can then be asked to be merged back into root LVM.

Here taking dom0 (root) as an example, but a similar call for all volumes (live or not) could be done so that lvm deals with the merge only when the destination LVM is inactive (on qube shutdown, or for root, at system shutdown) to abstract the problem with a same solution for all OSes:

/usr/sbin/lvremove --noudevsync --force -An qubes_dom0/root-revert || true
/usr/sbin/lvcreate --noudevsync --ignoremonitoring -ay -prw -kn -s qubes_dom0/root -n root-revert

Wyng can then receive its volume in that Read/Write snapshot:
sudo wyng --meta-dir=/var/lib/wyng-backups-vm receive root-snapshot --save-to=/dev/qubes_dom0/root-revert --sparse-write

• here we specify a different receive volume LVM (--save-to root-revert) then the actual root.

And then, we can ask that snapshot to be merged on next volume inactivation
sudo lvconvert --merge /dev/qubes_dom0/root-revert

[tlaurion]

@tasket any thought on the above?

[tlaurion]

@tasket hope you are well.

I finally managed to use 03alpha with a custom restore script, taking advantage of #88 (comment) idea for all LVMs, even the ones being active, so that latest state can be restored within Qubes after having validated that the archive dir info and digests are valid prior of arch-init from dom0 against locally mounted filesystem (for higher sparse-write restore: which restores that state of templates, dom0 root-autosnap and deployed qubes in around 5 minutes!!!!)

The script finishes the restoration by taking a new qubes-wyng-util meta-prep, instructing the user to restore qubes configs after restore for the unassociated qubes (since dom0 state is clean as per OEM original deployment and won't contain, nor start, any user created qubes/Templates unless those configurations are restored upon reboot.

I decided to go the way of bzip'ing /dev/sda1 to the backup qube's LVM, and having the restoration script have that bzip2 also taken into consideration under creation of detached signed digest. So the restoration script just failsafe prior of attempting to restore anything.

The ideal here to me, since /boot and qubes should match per backup session, would be to have qubes-xml-backup renamed to qubes-backup and contain /boot per snapshot saved by qubes-wyng-util meta-prep and stored as part of any backup session, so that /boot + qubes.xml matches when restored?

[tasket]

@tlaurion I'm OK, hope you are too! Using --merge seems fine, as long as the user is in a workflow (such as directed by your script) that doesn't leave any expectation they can preserve recent changes in currently-running VMs.

But for this reason I'm reluctant to use --merge in Wyng itself. It could make more sense to have receive get exclusive locks on the to-be-restored volumes and/or use a special *.recv snapshot and then swap it for the original as the last step in restoring. This way, there are clear-cut failure modes and nothing 'magical' (e.g. lvmthin doing delayed merge) happens on the next reboot. I think this would affect your script in only minor ways, and others looking to create a high level restore process around Wyng could simply (for example) ask the user "Shutdown all affected VMs y/n?" before doing the receive.

so that /boot + qubes.xml matches when restored

Seems like an elegant and support-able way to enable this would be to have the meta-prep routine accept an additional custom path to be included, or maybe (since the script is Qubes-specific after all) just an option to include /boot plus whatever sfdisk reports for the partition table.

[tlaurion]

That thread https://forum.qubes-os.org/t/dom0-backup-snapshot/11937

[tasket]

Since this was last updated, both the v0.3 and v0.4 branches were changed to use fcntl.lock() on the received local volume. Although this does not completely arbitrate conflicts between Wyng using the volume and a competing snapshot manager (qubesd), it does prevent direct, concurrent conflicts.

A sequential conflict (such as qubesd rotating snapshots after wyng receive completes) can be resolved by either using an environment-specific wrapper for wyng (such as wyng-util-qubes) to make sure restoring vms aren't running, and/or have the OS snapshot manager check the volume status (such as modification time) before doing a rotation.

[tlaurion]

@tasket

Seems like an elegant and support-able way to enable this would be to have the meta-prep routine accept an additional custom path to be included, or maybe (since the script is Qubes-specific after all) just an option to include /boot plus whatever sfdisk reports for the partition table.

I think you should go ahead with it.
Would definitely go for it. I would use that meta-prep call and reuse it on restore for dom0 as well and call it a day.