ER: Add Casper persistence support

[pbatard]

According to the following, the way to enable persistence is either to:

• create a partition with label casper-rw and formatted with an ext2 file system (or later)
• create an image, at the root level of the device, called /casper-rw and formatted with an ext2 file system (or later)

As there exist ext2 formatting tools for Windows (mke2fs, genext2fs...), it looks like what would be required to add persistence in Rufus, is add the ability to format either an image or a partition to ext2. This may however require more effort than it looks, as if we want to reuse any code, it needs to be GPLv3 compatible, which genext2fs isn't for instance. So we may have to write our own ext2 formatting code from scratch... 😢

[InfoLibre]

If you or somebody else add persistence one day, do not propose to write manually the size or to choose a checkbox like in Unetbooting or in MultibootUSB but use a bar like in YUMI or in Linux Live USB Creator. It's more convenient.
PS : I will add a Casper documentation in http://www.numopen.fr/documentation, it's a great software.

[pbatard]

I'll probably have both a slider and an input field, to keep everyone happy.
Something like this:

Persistence size:
[==========^=======] [ 1.5GB]

[InfoLibre]

+1

[pbatard]

Just a note that the /lib/ext2fs/ part of e2fsprogs is licensed under LGPLv2.0 (whereas the rest of the source is GPLv2 only), which means that, due to Clause 3 of the LGPLv2.0 license (see below), we can relicense this part of the source to be GPLv3+, which in turn makes it usable in Rufus.

You may opt to apply the terms of the ordinary GNU General Public License instead of this License to a given copy of the Library. To do this, you must alter all the notices that refer to this License, so that they refer to the ordinary GNU General Public License, version 2, instead of to this License. (If a newer version than version 2 of the ordinary GNU General Public License has appeared, then you can specify that version instead if you wish.) Do not make any other change in these notices.

However, mke2fs itself appears to be GPLv2 only, so we will have to write our own version...

[micker]

hello any new about this feature ?
thnks for your work !

[pbatard]

https://github.com/pbatard/rufus/wiki/FAQ#Why_isnt_Rufus_development_faster

[micker]

hello its just a question i didn't want to offense you ... i already know about free time dev no problem

[pbatard]

No worries, I didn't mean to be rude either. The only thing I can say is I'm still planning to look into it, but I have no idea when that will happen.

[micker]

Ok no problem

[ltc-data]

+1 for persistence support.

As a side note to licensing of third party components, genext2fs is gplv2 and it does not need to be GPLV3 see https://www.gnu.org/licenses/gpl-faq.html#ManyDifferentLicenses which is what you already do for other portions in the about menu.

In the eventuality that you make changes to the source code for that, since it is 2007 date of last release (sourceforge) you can simply upgrade the licence to GPLv3.

[gregs5]

rufus is fantastic. But +1 for persistence support, if that's a possibility. mkusb (now "dus?") seems to be the only thing that works with >=Ubuntu16.04 to build persistent USB installs - most of the other USB creators don't support persistent/casper-rw options with 16.04 onward. Which is fine, but dus only runs on Linux, so one either needs to already have a Linux build running to make a new persistent Ubuntu USB build, or one needs build a non-persistent Ubuntu build, boot it, modify it, install mkusb/dus, and THEN image a second USB device. If one could do this all with rufus in one shot, that would be super useful!

(And it took me burning a lot of time with lili, yumi, unetbootin, etc. to figure this out)

[Useihwd]

I need this feature, please achieve it!

[NSbuilder]

Today there are a lot of other tools to create bootable drives ( many of them are open source ) that can do this job for you. For example: Universal USB Installer and YUMI / YUMI - UEFI for multiboot.

Persistence (if available) – Ubuntu, Xubuntu, and Lubuntu Casper Persistence feature works with FAT32 or NTFS formatted drives. Larger than 4GB casper-rw is possible only when the USB drive is formatted with the NTFS filesystem.

It would be nice to see this feature but, In the meantime you can use one of the many tools available.

Also, you can do a full install of Linux to a USB drive ( that is fast enough ) and that will function like a complete Linux system ( Basically you must create a virtual machine where there is no HDD, only the USB is attached and install Linux on the USB by choosing it ).

[mcuee]

If you have created the Linux USB disk using rufus, you can try to use the following tips to add casper-rw (it creates an empty image file and then call dd to create the final casper-rw file.
https://www.lifewire.com/create-uefi-bootable-ubuntu-usb-drive-2202085
https://www.pendrivelinux.com/casper-rw-creator-make-a-persistent-file-from-windows/

[mcuee]

It also has the tip to create a >4GB casper partition.
https://www.pendrivelinux.com/create-a-larger-than-4gb-casper-partition/

[steve6375]

tip: ext2 is very prone to corruption. If the persistence file gets corrupted then linux will not boot. I recommend ext3 which is much more resilient (or ext4 but it has extra journalling which may cause more wear on flash drives).

[pbatard]

Thanks for the tip. Unfortunately, I'm going to be constrained with whatever GPLv3-compatible ext# driver I manage to port to Windows, so it's not guaranteed that I will have much of a choice in that matter. But I'll try to use ext3 over ext2 if I can...

[steve6375]

OK - but in my experience, having ext2 persistence on a removable USB drive will always end in tears and Rufus will probably get the blame instead of the thing between the keyboard and the seat!

[pbatard]

Alright, with the next commit I add, I'm going to consider that this request is officially closed, as Rufus 3.6 will allow the creation of a persistent partition (which will be ext3 by default, Steve 😉).

I will however point out of few things:

• If working with an ISOHybrid, persistence will only be applied when creating the image in ISO mode, not in DD mode. As a matter of fact, when persistence has been selected, you will not be prompted to choose between ISO and DD, as Rufus will consider that, if you enabled persistence, then it means you want to use ISO mode.
• I am not currently planning to edit config files (Syslinux, GRUB), to add persistence to the kernel parameters. First of all, that's because it's a bit tricky to do (for instance Ubuntu has multiple kernel entries, and we're only want to do it for the 'Live' one) and second, it's because currently Ubuntu has a MAJOR BUG that will result in init failing with the messages (initramfs) mount: mounting /cow on /root failed: Invalid argument/overlay mount failed (yes, we tried moving the casper-rw partition before the main partition, and, no, I'm not going to add a 3rd EFI partition when creating Ubuntu drives just to work around a BUG that should have been fixed years ago!). So, I'd rather have Ubuntu fix this very disappointing issue before I start to look into adding persistence to conf files, and end up with a bunch of Ubuntu users complaining that their media cannot complete boot...
• ext3 formatting performance is rather slow (compared to doing it from Linux) even though I made sure to use the exact same formatting library as the one that's used on Linux (lib/ext2fs from e2fsprogs's) and left the sources pretty much unmodified. The only thing that changes of course is the I/O channel that provides low level access, but I didn't change much of it either, so it's really disappointing to go out of your way to use the official tools, and see them have such sluggish performance as soon as you move out of their preferred platform... 😢 Oh, and of course, adding journalling (for ext3) also adds extra delay for the formatting...

Anyway, you'll have the ability to create persistent partitions in Rufus starting with Rufus 3.6. Happy now?

[JonnyTech]

@pdatard afaik ext3 is kinda being deprecated: https://www.phoronix.com/scan.php?page=news_item&px=linux-kernel-dropping-ext3

More info re ext3/4 in another project: xcp-ng/xcp#122

In order to preempt issues concerning ext3 usage, would it be preferable to implement ext2 or ext4 instead?

(Great work btw!)

[pbatard]

In order to preempt issues concerning ext3 usage, would it be preferable to implement ext2 or ext4 instead?

You do realize that I went with ext3 over ext2 (which was my initial plan) because @steve6375 said that:

having ext2 persistence on a removable USB drive will always end in tears

And while I could go all the the way to add ext4 my concerns are that:

• Some people might use older distros that don't support ext4 features (though I suspect they might be smart enough to use the fs as ext3)
• It will add additional complexity and increase the app size
• I have yet to see any major benefit to using ext4 over ext3 for something that is not going to be used as a 'permanent' (read installed) Linux distros. From what I gather, persistence is mostly used by people who want to toy with Linux for a while, and who won't be running high transfer rate computations or anything where having ext4 might give them an edge.

At any rate, I did add a registry setting UseExtFsVersion where you can set the version of ext you want to use for the peristent partition (though, right now, it is limited to choosing between 2 and 3), so if you really want ext2, you can have ext2...

Of course, if I start to receive tons of requests to add ext4 I'll see what I can do, but, Jeez, you can't wait a minute after you spent weeks adding feature X before someone asks you to add feature X+1... 😄

[JonnyTech]

Hehe, only trying to help. Anyway, I mentioned ext2 because I use that on all my USB sticks without problem. Ext4 has been part of the Linux kernel for over 10 years so I guess that all current distros should cope equally well with ext3/4. But as we all know, any file system can be corrupted when not handled correctly by users, Windows ones too.

[pbatard]

Agreed. And that's precisely why I would agree with @steve6375's suggestion that using ext3 over ext2 should avoid some issues (especially on USB media that might be yanked out without warning), as, while not a panacea, a journal can at least offer some help in recovering or reverting operations that were started and never completed...

[tonijarjour]

Can I reformat the partition to F32?

[pbatard]

Of course. Rufus would be pretty useless if it couldn't reformat an ext partition (or any other file system partition for that matter).

Oh, and it doesn't hurt to try. You should be able to find the answer to your question by simply playing with 3.6.

[RuijieYu]

I was trying to put Ubuntu 1904 on my USB with persistence, and when booting encountered the bug that you had mentioned (/cow onto /root). If possible, I would suggest that before starting, the program should prompt a warning message when it "detects Ubuntu" (even parsing the filename could help) while persistence is nonzero. In this case people are less likely to encounter the bug.

[pbatard]

This has been requested before, and is not as simple as it seems, because we'd need to detect the version of Ubuntu (because 19.10 is fine) and we can't simply use something like "all versions before 19.10 should get the warning" because there are talks of porting the fix to 18.04 LTS. Besides, 19.10 will be released in less than a week now, so the 19.04 issue should quickly become less of a nuisance.

But more importantly, users of Ubuntu images already receive a notice that, if they encounter an issue during boot, they should try re-creating the drive in DD mode (in which case they won't get the cow error since persistence will be disabled). Which means that, if a user pays attention to Rufus notices like the one you suggest I add, they should be able to sort themselves out. And if they don't, then it won't make much sense to add a 3rd notice after pressing START (because right now a Ubuntu user already received 2 prompts, and one also has to consider how many prompts an application can subject its users too before they become really annoyed).

Finally, I have to say that I am personally quite annoyed that I had to step in to get what I consider to be a glaring bug fixed, that has been part of Ubuntu for years. As far as I am concerned, this is a pure Ubuntu issue, and I am really dismayed that Ubuntu users haven't bothered to press on getting this major issue fixed a long time before, since the recommended method to add persistence has always been to create a second partition, and, regardless of whether you are using Rufus or not, as soon as you do that, you get into the /cow mounting error. So, if anything, this actually tells me that Ubuntu users don't care as much about persistence as they seem to pretend, because, if they did, then a major bug that pretty much rendered any attempt at adding persistence on Ubuntu fruitless wouldn't have been left open for 4 years...

Plus, as far as I am concerned, I have done more than enough to get this issue sorted, so I will now leave it up to Ubuntu and its users to do the rest. For one thing, if you are really annoyed by this, then please let the Ubuntu maintainers know that they should retrofit the fix for this in the next 18.04 LTS release. Otherwise, I can guarantee that, just like the original bug, whereas if enough users had bothered letting the relevant people (the Ubuntu maintainers) know that they were affected a lot let users would find themselves impaired by the consequences of this longstanding issue, history will just repeat itself, with maintainers not being aware that having broken persistence with 18.04 is negatively affecting a lot of users, and leaving it broken. So please remember, it's not just developers that get things done with Open Source Software: YOU too get a chance to get things fixed, as long as you do choose to report the issues you are facing to the proper set of people.

[steve6375]

https://rmprepusb.blogspot.com/2017/10/how-to-add-ubuntulinuxmint-to-e2b-with.html
Ubuntu bug detailed in this blog 2 years ago!

[pbatard]

Yes, I am aware of the workarounds, none of which I found satisfying for Rufus.

And again, I am pretty annoyed that I had to be the one that tried to convince the Ubuntu maintainers that this was a major problem that needed to be fixed. This was a major bug, that needed to be pointed out as such from the get go.

It really doesn't seem to me like the Ubuntu maintainers realized the severity of the issue, which, for something that many people like you reported as a major issue on various websites, is very disappointing. Again, IMO, we wouldn't be in the mess we are right now, with users annoyed that they can't simply add a persistent partition after their main FAT32 partition on the drive they boot from, if more people had told the Ubuntu folks that they had a major issue on their hand. 4 years to leave such a bug open is way too long...

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue if you think you have a related problem or query.