Persistent partition between boots

[shasheene]

Users have regularly called for Rescuezilla to adopt a persistent partition to maintain files between boots.

This could be used to hold network configuration (static IP, wifi passwords etc) to make using Rescuezilla more efficient, particular for use in IT roles.

This is an interesting idea that is worth pursuing. Please post a reaction on this post if you would like this feature, as right now, it's very low priority.

There are security aspects to a persistent partition that need to be deeply considered. Alternative approaches include looking for another storage medium (eg, CD or second USB stick), and checking if contains a file (let's call it rescuezilla.cfg) in the root directory.

[shasheene]

A user named "coth" in a now-unavailable comment on the AlternativeTo website wanted persistent partitions. I suggested to keep all discussion in this GitHub issue to keep the AlternativeTo comment section tidy.

Coth writes:

Then, may be, if possible, some edit mode to boot directly from USB without RAM to preinstall apps.

My response:

Your suggestion is difficult, but possible. Just like other Linux live environments, Rescuezilla's root filesystem is a SquashFS compressed archive, which is not trivial to modify if Rescuezilla is booted. As of writing, the easiest way to "bake in" new applications into a Rescuezilla ISO image is by adding the package to the build script and building a completely new custom ISO image. This is not easy enough for the many home users who have customization requirements but do not have the skills to build an ISO from scratch (and be satisfied that their custom ISO image was made correctly).

The persistent partition goal is achievable with a significant change to Rescuezilla's standard live environment architecture. Instead of an ISO9660 image containing a SquashFS root filesystem, a Rescuezilla image could be very similar to a regular uncompressed Linux hard drive. That is, using an EXT4 root partition. To create an immutable live environment, the typical boot would then have the fsprotect enabled via a kernel parameter in the default boot menu item. This would make the root partition mounted as read-only with a copy-on-write RAM overlay. A second boot menu item without the fsprotect kernel option then boots the system in read/write mode without this overlay.

[regs01]

Well, i said if possible. As there are many reasons for that - ability to save settings, ability to install and update apps right on place. But of course there are far more important things - like stable backup and restore in any conditions and optional EFI boot.

Still, can you point to how "bake in" apps?

[shasheene]

Still, can you point to how "bake in" apps?

The build instructions provides a basic overview on how to build Rescuezilla ISO images from source. Adding a new package requires first git clone'ing the rescuezilla repository, then git checkout the most recent tag (1.0.5 right now), then modifying the package list in the file chroot.steps.part.1.sh (taking care to preserve the use of trailing backslashes to escape the newlines). Then adding your changes to a new git commit, and finally using make to build a brand new ISO.

The resulting ISO image will be the same as the official release, but with slightly newer packages (due to a more recent apt-get update). The will also contain a version string like 1.0.5-1-g7db66e, indicating that it is based on a single commit beyond the v1.0.5, and the build associated is with the git commit SHA 7db66e. This version string is currently written into the same directory where a backup is made, and will be processed by future versions of Rescuezilla so it knows "this backup was created by a v1.0.5 compatible version of Rescuezilla". Other than that, it should be identical to the official release.

For thoroughness I will mention that a custom Rescuezilla ISO image can be analysed using the script extract_and_checksum_iso.sh in the root of the git repository, which helps compare it to the official Rescuezilla v1.0.5 with the help of the meld diff'ing tool.

Anyway, the process is relatively involved so I think only IT businesses using Rescuezilla heavily will go to the effort to customize the ISO images. Also to anyone reading this, if something like a new package is useful to the general Rescuezilla userbase then please push your commits to your Rescuezilla fork and then create a Pull Request, so it can be merged in and help everybody.

[shasheene]

(Please ignore the reference to issue 108 immediately above this comment. I mistyped a comment in that issue)

[shasheene]

To re-iterate, the currently officially recommended tool to write Rescuezilla images onto USB sticks is balenaEtcher, which simply directly writes the Rescuezilla ISO image file directly onto a USB stick. Rescuezilla boot media made in this way don't have persistent partitions because the filesystems used internally are read-only and the lack of a partition table means new partitions cannot be added.

In this GitHub task I was suggesting to modify the Rescuezilla build scripts to enable persistence, but a user named Julia has brought to my attention that Rufus is able to create persistent USB sticks from arbitrary Debian/Ubuntu ISO images. From the Rufus ChangeLog.txt:

o Version 3.7 (2019.09.09)
Finalize persistent partition support for Debian and Ubuntu [EXPERIMENTAL]:

• Debian with persistence should work out of the box with any recent "Debian Live" ISO.
• Ubuntu with persistence should also work IF using a post 2019.08.01 ISO. Note that,
  because of bug #1489855, trying to use persistence with Ubuntu ISOs that were released
  before August 2019 will only result in rescue mode during boot - You have been warned!
• Other distros may work with persistence as long as they use a Debian-like or Ubuntu-like
  method, and, in the case of Ubuntu-like, if they use casper with the #1489855 bugfix.
• Persistence and ext formatting support should still be considered EXPERIMENTAL at this stage.

This Rufus feature effectively implements this issue but without modifying the build scripts: Rufus creates a FAT32 root filesystem on the USB stick and extracts the Rescuezilla files, and a storage partition with an ext3 filesystem, then installs GRUB to make the stick bootable.

However, as of writing, the Rufus persistent partition is not usable by the Rescuezilla v2.0 frontend because the ext3 filesystem is being automatically mounted as the /var/log folder (task #130).

It's also worth noting that Rufus is only available on Windows systems, and for Windows/macOS user reading this, I should probably highlight the method described above is creating an ext3 filesystem, which is not recommended for non-Linux users because it won't be easily accessible on Windows/macOS systems without using a third-party filesystem implementation (which is risky).

[oam7575]

Based on the 5+ years of comments on this bug : https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1489855
It would seem that USB based persistence might be easier said than done at this point in time for an Ubuntu based distro.

The TLDR is that every other release the bug is fixed and then broken again; then even when its fixed they break something else causing a similar but different regression.

[shasheene]

Yeah that's exactly the Launchpad bug referenced in the Rufus v3.7 changelog quoted above.

It's very disappointing to hear that the Ubuntu developers keep re-introducing the bug, because converting Ubuntu-based ISO images like Rescuezilla to persistence USB sticks will be a massive feature!

[oam7575]

What if, as part of the build process the scripts create an empty file using fallocate switch to create an empty file.
This would create a file of size that takes up practically no space when empty.
Then format the loop file with mkfs.vfat
Then as part of the boot process the file can be mounted via start up scripts

Not sure how practical this would be; but if it works the resulting change would allow rufus and other programs to write the image directly to USB with no changes on their side as the USB should be Read / Write as FAT32 or NTFS , and the file system would be seeing a normal file that just happens to get larger from time to time.

[shasheene]

I've thought about the idea. It might be possible. From the build script:

• The ISO9660 specification has had a number of extensions over the years. They are important to better understand xorrisofs command-line arguments:
  • El Torito is the CD-ROM boot record specification, which was released in 1995 (named after the restaurant it was supposedly defined in)
  • An ISO 9660 filesystem can contain an MBR (Master Boot Record), ESP (EFI System Partition) and GPT (GUID Partition Table) without hampering the (up-to 63) El Torito boot records. The MBR/GPT is stored in the first 32KiB of the filesystem. For developers familiar to hard drive layouts, it may seem like a strange concept to have the MBR partition table stored within filesystem, rather than before the filesystem definition.

The MBR/GPT partition tables are within the ISO9660 filesystem. Rescuezilla's ESP (EFI System Partition) is a 6MB FAT16 filesystem that is simply a file that resides within the ISO9660 filesystem but gets exposed using the partition table as created by the xorrisofs ISO9660 filesystem contruction command.

Due to the design of ISO9660, it's not possible to arbitrarily modify the files in ISO9660 filesystems in-place: the entire ISO9660 filesystem would need to be unpacked then repacked eg, using xorriso. The question is whether it's possible to place the filesystem image you describe immediately after the ISO9660 filesystem, then using the use the ISO9660's MBR/GPT partition tables to point to this. I'm not sure this is possible, but it could be.

By the way, I may be wrong on this but as far as I understand fallocate sparse allocation is a operating-system kernel feature: a shortcut to make large allocations quickly. When the ISO image + file is copied to a different machine (say a file-server) for distribution the allocation would be silently "expanded". This would suggest that any ISO image + concatenated multi-gigabyte filesystem image of mostly-zeroes would need to be compressed before distribution. Not a show stopper or anything, but I thought I'd mention it.

As somewhat alluded to earlier this GitHub task, it's definitely possible (and actually pretty easy) to completely remove the ISO9660 and SquashFS filesystems from the build and instead construct an image in exactly the way a normal desktop Linux environment is setup: MBR/GPT + read/write partitions which would achieve a completely persistent environment when written to read/write media such as a USB stick. The only gotcha is you would generally want to "grow" the filesystems on first boot so that your entire eg, 128GB USB stick is utilized rather than only whatever sizes were selected by the build script.

[GralfR]

I wonder where to draw the line. I use a live-system to make sure, each time the system boots it has the same state, no matter what I (or any collegue) did last time. Especially concerning backup/restore it's important to have a stable system!
I use rescuezilla like acronis or other imaging software for Windows-PCs to save and restore a stable state of a desktop/laptop. But how often is that done? At maximum once a year (documents are backuped by copy more often).

I do not really need a persistent partition. You can connect to the internet after boot and download/install what you need. Okay, I agee, it would be nice to have the last config available, but in most cases I just want a working environment. No blinky glitter stuff, just basics is the best.

My suggestion:

1. Add the possibility for anybody to easily (by GUI) install the live-system booted from ISO to the HDD. After installation one might change settings and add any software he/she likes to this new system, which might be a virtual machine.
   This installation also needs a tool to create a new bootable ISO from that installation by GUI/one-click. This way anybody, who wants to adjust rescuezilla live can create his own live-ISO, even if he is a windows-guy like me.

2. Add a tool to the live-ISO that stores all settings by one click into a file (human-editable, maybe XML?) where the user gives path to (e.g. a USB-drive). So the user can adjust the live-system after boot (change network settings, display resolution, standby settings etc.) and store that easily to a file.
   If the system is booted the next time, the defaults are restored, so anybody gets a clean, working system. But a tool on the desktop offers to open/import previously saved settings from external drive (or even an online-cloud?).

That's how I'd like to see it, but I didn't really miss a persistence yet. There are a few tiny things I'd like to change (e.g. disable standby/energy-savings #173), but I can get along so far.

[shasheene]

Yep, I expect default behavior for 95% of Rescuezilla uses will remain non-persistence.

I agree it would be ideal to achieve your first point. It would be easier if Rufus was available on Linux, but the author has no plans to do so. It could be achieved using an alternative method though.

For your second point, a user was suggesting I look into Puppy Linux copy-on-write approach which would basically achieve what you're saying. It's definitely a reasonable approach that works well for Puppy Linux project, but such an approach is not as well suited to Rescuezilla: at this stage the path to long-term maintainability is to keep things as standard as possible by staying with SquashFS and ISO9660, so I'm strongly leaning towards the Rufus approach for persistence.

[pfrouleau]

Ventoy may be another solution to achieve this, see https://www.ventoy.net/en/plugin_persistence.html.

[shasheene]

That does look really good. I haven't tried Ventoy yet, but I know it's very popular. I will try it and see how it goes.

It sounds like the persistent partition is just a file on disk overlay-mounted as read/write. This is fine for updating a few configuration files, but I don't think I would trust it with an important multiterabyte backup.

Also, the workflow for Windows users is more complex than the current balenaEtcher instructions, but it might not be too bad.

[GralfR]

I wonder if storing to the fist partition is such a good idea. My concerns:

• A live-system should not change the installed system without user-knowledge
• Having the config file on the first partition requires a working partition and known filesystem. If this is not the case (e.g. while restoring the system) I do not have my configs available.
• Can a partition be backup/restored if rescuezilla has read/write-access to a file on that partition?

The ventoy might be okay, if that file is on a USB-drive, so it can be moved to any system I run rescuezilla on.

[shasheene]

On the dark theme GitHub task linked above, a user writes:

Actually, [dark theme] is a definitely a need. But, persistence stuff is much more need.

Unfortunately the persistent partition between boots (task #8) continues to be quite a complicated task with a lot of design considerations, so I am still not intending to complete it for a long time.

Instead I think it makes more sense to focus my finite development time on making well-tested standalone Rescuezilla packages for non-DEB distributions: RPM (RedHat/CentOS), Pacman (Arch Linux) (task #264), and improving the existing standalone deb package.

Then I will work to make Rescuezilla available in the official package managers of all Linux distributions. This will allow advanced users to create their own custom persistent Linux environments using whatever tools they feel like using.

A persistent partition would be very nice, but it's still a niche feature: the user-impact I will have from creating an easy-to-use graphical frontend to TestDisk undelete (task #12) will definitely be far larger than anything else I can spend my time on.

[shasheene]

After previously saying "this is a low priority", based on user-feedback I am bumping this issue back to high priority. As discussed on the Sourceforge forum I will write documentation for this issue (based on Rufus) and implement task #265 for the upcoming Rescuezilla v2.3 release:

But yes, using "mount USB into RAM" is a good temporary workaround for Rescuezilla v2.2 in a persistent environment. But here's what I'll do to improve things for Rescuezilla v2.3 (scheduled for release next month). I will implement task #265 (Ability to backup and restore partitions on same physical drive task). This will improve Rescuezilla for all non-live environments (fixing the "device or resource is busy" bug etc). I will also write official Rescuezilla documentation on how to create a Rescuezilla persistent environment created by Rufus.

[rachetdown]

So pleased to see you have begun to accept the need for persistence. Using Linux live boots on a Dell XPS13 2in1 Ultra HD screen is literally a pain in the neck. All text and graphics are tiny, I need to set that display change once and it be remembered. This is a feature set, finding it's way to more and more laptops, so the Linux world needs to plan on accepting it's requirements and provide solutions. I can use Rufus on a spare windows machine, fine solution for me. I also Really Like what I see you are about to do implementing task #265, for me this would be OUTSTANDING like the persistence. When v2.3 is released and I see those changes are in place I will join your support plan. You have created a fine tool, keep making good decisions like this and it will be a Great tool. It is a pleasure to use. Thank you for your expertise and skill providing a novice user, me, with a preferred GUI reliable and simple solution.

[candrapersada]

This could be used to hold network configuration?

for shared over a network

[shasheene]

In future yes. I will add a "remember network credentials" checkbox to the Rescuezilla application. Hopefully one day even a network profile system.

But initially, the persistent partition will only save system network configuration without any change to the Rescuezilla frontend.

[ehawman-rosenberg]

A workaround for the moment would be to use Ventoy's Boot Conf Replace plugin https://www.ventoy.net/en/plugin_bootconf_replace.html

By editing your own grub.cfg, you can set anything GRUB can set. I use it for networking stuff, mainly, although it seems many people want resolution changes as well which you can do. For Clonezilla I use ocs_prerun to mount my SMB share. Is there a Rescuezilla equivalent?