DietPi-Backup | Add Option to Automatically Restore on First Boot

[ravenclaw900]

Status: Ready

Reference: #4066

Commit list/description:

• DietPi-Software | Add way to automatically restore from backup on first boot
• DietPi-Software | Make mawk more specific
• DietPi-Software | Various fixes
• DietPi-Software | Boot into normal DietPi after restoring backup
• DietPi-Software | Don't change install state and sync filesystem
• DietPi-Software | Switch from sed to G_CONFIG_INJECT
• DietPi-Backup | Fix search function

[MichaIng]

Awesome! I think we can add this to v7.0 already.

[Joulinar]

will this work already if I do fresh installation and use this branch as target?

[MichaIng]

Yes, since the update is performed first 🙂.

[MichaIng]

Let's also unmount the auto-mounted drives and remove the mount points:

umount /mnt/dietpi-backup[0-9]*
rmdir /mnt/dietpi-backup[0-9]*

[MichaIng]

Testing procedure on an existing system:

dietpi-backup 1
G_DEV_BRANCH restore-boot
G_CONFIG_INJECT 'AUTO_SETUP_BACKUP_RESTORE=' 'AUTO_SETUP_BACKUP_RESTORE=1' /boot/dietpi.txt # Respectively "=2"
systemctl enable dietpi-firstboot # To simulate a full fresh install
reboot

[MichaIng]

Dammit that we log to /var/log/dietpi-backup.log (RAMlog) so it cannot be reviewed. Probably we should move it to /var/tmp/dietpi/logs/dietpi-backup.log 🤔.

[Joulinar]

probably a good idea to check what happen. but I can try to catch it. 🤔

[Joulinar]

ok I login during initial setup trying to catch what happen. Right after restore finished, some commands are missing

root@DietPi:/var/log# ls
-bash: /usr/bin/ls: No such file or directory
root@DietPi:/var/log#

attached the log file. Really strange. It goes to delete entire modules and firmware 😭

dietpi-backup.zip

[ravenclaw900]

That is very strange. Did you try it with a different backup/normal DietPi-Backup?

[MichaIng]

I got it, it's two issues. First is the merge of /bin /lib etc. with /usr/bin /usr/lib etc.

Background on this:

• https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/
• Debian and many other distributions aim to merge /lib and /bin with /usr/lib and /usr/bin since this separation has historical reasons that do no apply anymore and is not always compatible with other UNIX-like OS'es.
• To achieve best compatibility, /lib and /bin (and alike) are created as symlinks to /usr/lib and /usr/bin (and alike).

I just verified that the new RPi images contain this merged directories as well, which is quite new, AFAIK. For new Debian images is was the case for years already, but for RPi I see it the first time (doesn't mean something 😄). So this is what I guess happens:

• You try to restore a backup from an older image, with non-merged directories.
• DietPi-Backup restores hence the original actual files to /bin and /lib (removing the symlinks they were before). You see all those >f+++++++++ entries in the log, stating that new files are created.
• Then, inside /usr/bin and /usr/lib, those files, which were only placed there due to the symlink, are accordingly removed.
• hash -d should however fix the case that for that reason the hashed /usr/bin commands are not found. Finally a reboot ultimately solves it.

The second more problematic thing: One can only restore a backup to a new image where UUIDs and PARTUUIDs do still match, as otherwise fstab and boot configs contain invalid entries. To allow more, we'd need to either manually change fstab and boot configs (including grub update etc), or, probably easier, change the UUIDs/PARTUUIDs of the current root drive partitions to match the ones from the restored backup: https://www.tecmint.com/change-uuid-of-partition-in-linux/
tune2fs works on ext4 only. For the vfat boot partition:

• https://superuser.com/a/1248329/1183051
• https://manpages.debian.org/mlabel

But now there can be F2FS, GPT partition tables etc where things become more complicated 🤔. It looks like it can only be safely used when the original image/archive that was once flashed is kept... But not practically when it's very old.

It becomes quite complicated but we could loop through the drives partitions UUIDs and PARTUUIDs, check whether one of the two is present in the backups fstab (if it matches, assume that the other one does as well), otherwise check the partition table type and file system type to know whether we can adjust it, and if not print that to the user to deal with it manually. Auto-adjusting fstab and boot config is probably less save since we cannot know where else those IDs are used.

Ah another practical issue: Where do we get the old IDs from? fstab contains UUIDs but usually not PARTUUIDs. For the latter we again need to check and scrape boot configs. Arrg sadly this will never work failsafe...

[Joulinar]

uhh probably you are on the right track @MichaIng

basically I have a master backup on my USB stick that I use to restore/reset the same RPi again and again. This backup will be updated time by time. Usually after larger apt package updates, to avoid long apt installations in front of a software install. Means, the initial backup was create some month ago and refreshed last time on the 12th of Feb. A restore on the current live system is working without issues. However it's gonna fail if I restore on a fresh image.

Just for reference, backup status log attached.

dietpi-backup_stats.log

[MichaIng]

I think this is something that needs to be added to dietpi-backup restore code directly, as a pre-check before restoring the backup, since users might also manually restore the backup on a new image. But I'm not sure how to effectively deal with it:

• We can scrape the IDs from the backups fstab/cmdline.txt/boot.ini/armbianEnv.txt, but it is not failsafe, since only either UUID or PARTUUID is used while on the new image it might be different.
• ... rethought: I think it's safer to adjust fstab and boot configs instead of adjusting the UUID and PARTUUID where i don't even know if it's possible with all relevant file systems and how the (flashed) bootloader behaves, U-Boot in particular.
• So we check the current root and boot partitions UUID and PARTUUID, then search and replace them in the above list of files + update-grub + update-initramfs -u / update-tirfs after the backup has been restored. A warning + G_WHIP_YESNO is required in any case when partitions/IDs have been changed.

[MichaIng]

Okay guys, I'll port the few unrelated little fixes and enhancements to the dev branch now but won't merge this feature into the first beta. We need to handle drive ID issue first. Should be possible to have it until next week for 2nd or 3rd beta.

[MichaIng]

Coincidence: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=563746
An update of dosfstools that allows to update the FAT volumes UUIDs via fatlabel has just been merged into Bullseye branch 😄. But not relevant anymore as of (my) current plan to mitigate the issue.

[MichaIng]

Ready for testing.

[Joulinar]

Ok will run some test on weekends

[MichaIng]

Tested with all boot configs, and works fine now. I'll merge it to dev branch, so it can be tested together with other recent changes.