Prune all pruneable sessions of all backups

[tlaurion]

This is a feature request. Hacked my way around it with this to recover 200GB of unneeded backups (last backup session not being pruneable)

sudo ~/sparsebak/sparsebak.py list |sort|tail -n +7 | while read appvm; do echo $appvm:; sudo ~/sparsebak/sparsebak.py list $appvm|tail -n -4|head -n +1|awk -F " " '{print $(NF-1)}'| while read last_deleteable_backup; do echo $last_deleteable_backup|sudo ~/sparsebak/sparsebak.py -u prune $appvm --all-before --session $last_deleteable_backup; done; done

[tasket]

@tlaurion In the beta1 (wyng) branch I changed the date-time code so that if you specified a date-time later than the latest session, it wouldn't skip the volume but instead include the specified sessions (minus the latest). So this should now work:

sudo wyng --all-before --session=<current-date-time> prune <volname>

# or

sudo wyng --session=20191101-120000,20200117-231500 prune <volname>

BTW, I'm now up to beta2 which has had more testing than beta1. I highly recommend switching over to it since it has more features and safeguards than sparsebak. And it would really help me get a true release underway to have some feedback from the latest branch... You should only need to rename both '/var/lib/sparsebak' and the dest path to 'wyng.backup' to make the switch. (You can also make a no-copy backup of your archive in reasonable time with a command like cp -rl sparsebak sparsebak-backup1). Let me know if you have any questions about it.

[tasket]

Ref commit 36cb98d

[tasket]

@tlaurion To make it easier, here is a working example that uses the current date-time whenever its run:

sudo wyng --all-before --session=$(date +%Y%m%d-%H%M%S) prune <volname>

[tlaurion]

@tasket :

I transitioned away of sparsebak by doing as described in precedent post in dom0:
sudo mv /var/lib/sparsebak /var/lib/wyng.backup

Then took a less important backup and did, prior to looping in all preexisting backup:
sudo ~/sparsebak/wyng --all-before --session=$(date +%Y%m%d-%H%M%S) prune vm-build-x230-coreboot-48-private
Which results in never ending (ctrl-c'ed because no disk activity):

Pruning Volume : vm-build-x230-coreboot-48-private
  Preserving latest session.
  Removing 1: S_20191103-090906

Then:
Completing prior operation in progress: merge vm-build-x230-coreboot-48-private

[tlaurion]

Just realized I havent't moved sparsebak backup directory on destination disk to wyng.backup, just on dom0. Wiped wyng.backup on destination disk (backup AppVM) and retried.

Now I get:

Completing prior operation in progress: merge vm-build-x230-coreboot-48-private
Traceback (most recent call last):
  File "/home/user/sparsebak/wyng", line 2234, in <module>
    aset.in_process[3], clear_sources=bool(aset.in_process[2]))
  File "/home/user/sparsebak/wyng", line 1772, in merge_sessions
    +" && python3 "+tmpdir+"/rpc/dest_helper.py merge" +(" --resume" if resume else "")
  File "/home/user/sparsebak/wyng", line 751, in do_exec
    raise subprocess.CalledProcessError(err.returncode, err.args)
subprocess.CalledProcessError: Command '['/usr/bin/qvm-run', '-p', 'backup', '/bin/sh /tmp/wyng/rpc/tmp2j4w_ydh']' returned non-zero exit status 1

[tlaurion]

Hmmm. Even trying to delete that backup tries to continue resuming past operation on that volume. Any insight?

[tasket]

Yeah, that's why I suggested doing a cp -rl snapshot first, in case the rename to wyng.backup didn't go right. Actually if you have a backup of the /var/lib/sparsebak dir you could restore a copy of that too.

What's going on is that the prune op crashed at a point I don't cover yet in recovery sanity tests. And its not yet sophisticated enough to see that it has already re-tried the op.

[tasket]

@tlaurion If you look at the volume's dir on dest, and there is a "merge" subdir with contents in it, then its part-way into the move/remove operations. The data won't come back out of that merge dir until the op is complete. There may also be a "merge.log" in /var/lib/wyng.backup/default/volname.

If you're really stuck with this, I can recreate the problem by renaming my dest archive and then trying to start a prune op.

[tasket]

I tried to recreate the problem with:

cd /dest
mv wyng.backup sparsebak
wyng  --all-before --session=$(date +%Y%m%d-%H%M%S) prune
rm -r wyng.backup
mv sparsebak wyng.backup
wyng  --all-before --session=$(date +%Y%m%d-%H%M%S) prune

However, the in_process flag did not take and there was no attempt to resume the prior op.

[tlaurion]

What needs to be done to discard merge attempt in case it tries to resume operation? Delete the whole backup, or there is something that can be attempted?

[tasket]

@tlaurion If you're importing from the archive that had the prune interruption (and the in_process status), you may still have a bit of detective work to do before the import. Let me know if on the dest volume there is a 'wyng.backup/default/in_process' file and/or a 'wyng.backup/default/VOLNAME/merge*' dir.

I think beta3 can recover the situation now, but I only did a few tests. Here is the decision matrix:

In the case there is no 'merge' or 'merge-init' dir, the merge procedure will be aborted and the in_process status cleared. If there is a 'merge-init' dir, it will try to return the session data files to their original locations before aborting and clearing in_process.

In the case there is a 'merge' dir, it will have to try to complete the merge procedure. (If this fails for some reason, it can be rescued with normal shell commands like cp, rm, etc. I can provide details.)

[tasket]

@tlaurion There is a bug in merge that I'm dealing with now, so please hold off pruning with beta3 until I post a fix.

[tasket]

@tlaurion OK, that bug is now fixed in beta3.

[tlaurion]

@tlaurion If you're importing from the archive that had the prune interruption (and the in_process status), you may still have a bit of detective work to do before the import. Let me know if on the dest volume there is a 'wyng.backup/default/in_process' file and/or a 'wyng.backup/default/VOLNAME/merge*' dir.

[user@backup QubesOS]$ cd wyng.backup/default/

[user@backup default]$ find ./ -name merge*
[user@backup default]$ find ./ -name in_process*
[user@backup default]$ 

I think beta3 can recover the situation now, but I only did a few tests. Here is the decision matrix:

In the case there is no 'merge' or 'merge-init' dir, the merge procedure will be aborted and the in_process status cleared. If there is a 'merge-init' dir, it will try to return the session data files to their original locations before aborting and clearing in_process.

In the case there is a 'merge' dir, it will have to try to complete the merge procedure. (If this fails for some reason, it can be rescued with normal shell commands like cp, rm, etc. I can provide details.)

[tlaurion]

Since the problematic volume was vm-build-x230-coreboot-48-private, I also deleted that non-completed pruned session.

Same result when trying to arch-init with --from in #50