Consider changing the BTRFS subvolume layout

[KhalilSantana]

Hello,

I'm very happy to see that v2.3.X includes BTRFS subvolume support as a beta feature! This was a major blocker for me and now it's (mostly) gone.

However, I would like to comment on the current subvolume layout used by Archinstall's guided mode, and why (and how) I think it could be improved.

The Current Layout

For reference this is the current layout used by guided mode as of 2021-12-1 using archinstall v2.3.0:

# btrfs subvol list -t /
ID	gen	top level	path	
--	---	---------	----	
256	21	5		home
258	21	5		var/log
259	17	5		var/cache/pacman/pkg
260	10	5		.snapshots
264	14	5		var/lib/portables
265	15	5		var/lib/machines

How BTRFS handles subvolumes, and snapshots

(this section exists mostly to add context, skip it if you are familiar with BTRFS)

In BTRFS, each subvolume has an ID and a path, relative to the toplevel (a.k.a subvolid=5), these subvolumes can be mounted anywhere, and can be used for many things just like folders, however, the main differentiator between a regular folder and a subvol is that the later has special powers: atomic snapshots, etc.

Snapshots are just a clone of that filesystem root (subvol) at that time, and it shares extents with the snapped subvol, this is one of reasons snaps are so useful: they are fast and small.

However, there are some important quirks/behaviors that BTRFS currently does:

• Rule 0: You cannot replace the toplevel (subvolid=5, subvol=/) with another subvol.
• Rule 1: btrfs subvol snap won't recurse into nested subvolumes. So you cannot create atomic snaps of nested subvolumes and their parents at once.
• Rule 2: As btrfs subvol snap doesn't recurse, the resulting subvolume (the snapshot) won't contain ANY of it's nested subvols.

With this in mind, I hope the next section becomes easy to understand.

What's wrong with the current layout?

The current layout basically prohibits using snapshots as a rollback feature, why?

• Per Rule 0, you cannot rollback (without lots of jiggling mv's and rm's) the root filesystem as it's been mounted at the toplevel subvol.
• Per Rule 1 and 2, even if you manually rollback using methods, like using the (often confusing) btrfs sub set-default, your homedir will be empty! You'd need to manually snap RW the old home subvol into the right place.

In short the main issues with the current subvol layout are:

• The root filesystem using the toplevel subvolume directly.
• The home dir0 being a nested subvolume.

What would be a better layout?

• Reserve the toplevel subvolume for managing other subvolumes
• Utilize a subvolume as the root filesystem
• Utilize a non-nested subvolume for the home dir.

This is often called the "flat-layout", often used with the ubuntu-style naming scheme (@ as /, @home as /home), and it's documented on the wiki

What other distros are using?

• Ubuntu's installer uses the flat layout, with @ and @home, mounted at / and /home, respectively
• Manjaro's installer uses the flat layout, identical to the Ubuntu one.
• Fedora's Installer uses the flat layout, with a different naming scheme (root and home AFAIK)

My layout suggestion

Follow the flat layout, with the Ubuntu-style naming scheme, but keep the extra subvols (/var/log & /var/cache/pacman/pkg), those extras not nested, of course.

Basically, this:

# btrfs subvolume list -t /
[sudo] senha para khalil: 
ID      gen     top level       path
--      ---     ---------       ----
256     1408526 5               @
257     1408526 5               @home
258     1408526 5               @pkg
259     1408526 5               @log
260     1391176 5               @snapshots

In tree view for clarity's sake:

# exa -TL1 /mnt/toplevel # this is the mount point of subvolid=5 on my system, for maintance purposes)
/mnt/toplevel
├── @
├── @home
├── @pkg
├── @log
└── @snapshots

Why the Ubuntu naming scheme? Why not Fedora's? Or something else entirely?

Mostly because of Timeshift, as it only supports that standard. And while it might seem like a silly reason, I feel like it's reasonable: Timeshift is the one of the few snapshot utilities that has an incredibly easy to use restore button, which makes undoing changes incredibly easy, and also un-breaking a system after a bad upgrade.

---

Thank you for your time!

[Torxed]

Thank you for this very detailed feedback! Really awesome to see everyone suggesting good things to improve the btrfs subvolume support. We'll track it in this issue and in #718 where the topic came up during an issue of re-using btrfs subvolumes that was anything other than what we supply.

[wllacer]

@KhalilSantana Great explanation for what i also do empirically,

With a slightly different layout (i don't care -still- for snapshooting but more for usage flexibility. At the end of issue #718 you will find an unoffical patch that allows archinstall to work with flat layouts IF they are defined and mounted previously. I would be glad i could have some feedback about it. I could make a PR of it, but i'm not that sure of its quality (works for me, but this is what i can guarantee).
I want to expand it to full support of flat schemas. Any suggestion and ideas would be welcome

[Torxed]

I would be glad i could have some feedback about it.

It's coming in a bit : ) Gathering my thoughts to not just blurp out unfinished thoughts :)

[rexhent]

This is definitely important. Btrfs is a bit of a pain to remember how to use when doing the arch install "the old way", and one of my main reasons to use EndeavourOS right now.

[DrymarchonShaun]

Just to put it out there as well, why not go with opensuse's layout, so that snapper can be fully utilized? Timeshift works, but the last time I tested it, it left like it lacked alot that Snapper has; rolling back with snapper is as easy as booting a snapshot of your working system, and running snapper rollback when you are using the right subvolume layout.

From https://en.opensuse.org/SDB:BTRFS

/home
If /home does not reside on a separate partition, it is excluded to avoid data loss on rollbacks.

/opt
Third-party products usually get installed to /opt. It is excluded to avoid uninstalling these applications on rollbacks.

/root
The root users home directory should also be preserved during a rollback

/srv
Contains data for Web and FTP servers. It is excluded to avoid data loss on rollbacks.

/tmp
All directories containing temporary files and caches are excluded from snapshots.

/usr/local
This directory is used when manually installing software. It is excluded to avoid uninstalling these installations on rollbacks.

/var
This directory contains many variable files, including logs, temporary caches, third party products in /var/opt, and is the default location for many virtual machine images and databases. Therefore this subvolume is created to exclude all of this variable data from snapshots and is created with Copy-On-Write disabled.

What this ends up looking like (with the top-level subvolume mounted at /btrfs)

/Btrfs
└── /@
      ├── /home                       /home
      ├── /opt                        /op
      ├── /root                       /root
      ├── /srv                        /srv
      ├── /tmp                        /tmp
      ├── /usr_local                  /usr/local
      ├── /var                        /var      (cow disabled)
      └── /.snapshots                 /.snapshots
              └── /1/snapshot       /

Putting the root subvolume in .snapshots makes using snapshots much cleaner, as the original root subvolume is essentially a snapshot, so what happens after you boot into a read-only snapshot and run snapper rollback is 1. A read only snapshot of the current (broken) system is created, 2. a read write copy of the currently booted snapshot is created and set as the default btrfs partition, so that when you boot back up, it gets mounted as the new root subvolume.

It is more complicated to wrap your head around to begin with, but once you understand it, it makes way more sense.

If it helps make it understandable at all, I just finished fixing a (janky) script that installs arch with this layout (albeit with a lot of separation of /var) https://github.com/ShaunTheQuietGamer/Arch-Setup-Script/blob/main/install.sh
It might help explain the process to get this functioning, but I understand that there's probably a better way to do all the stuff its doing, and some of is important to get the system functional (like modifying fstab, and adding some stuff to the grub config,) a lot of it is not.

EDIT: I didn't fully understand how this whole thing worked, and still probably don't, but isn't it theoretically possible to add both layouts as an option for users? Instead of it asking Yes or No if they want a default subvolume layout, it could ask if they

1. Don't want subvolumes
2. Want the Ubuntu style flat layout
3. Want the opensuse layout with snapper support
4. Manual subvolume setup (like the disk partitioning section)

(Something else to look at is possibly adding an extra Y/N if the Opensuse layout is selected to also install and set up snapper, grub-btrfs, and snap-pac with snapper rollback functioning.)

[Torxed]

Before we close this, and you'll have to correct me if I'm wrong because there's almost too much context here so I might have gotten a bit lost in all the reading. But the desired outcome is:

/Btrfs
└── /@
            ├── /home                       /home
            ├── /opt                        /opt
            ├── /root                       /root
            ├── /srv                        /srv
            ├── /tmp                        /tmp
            ├── /usr_local                  /usr/local
            ├── /var                        /var      (cow disabled)
            └── /.snapshots                 /.snapshots
                      └── /1/snapshot       /

Which we currently can't support because we don't support a mountpoint (/ in this case) inside another subvolume, is this a correct assumption @wllacer (pinging you because you've done more work on the subvolumes than me at this point).

If we can live without mountpoints inside subvlumes, this issue is fulfilled. Otherwise I'll keep this open but I'll move it for v2.4.0 as I'd like to get v2.3.1 out ASAP for patch reasons.

[wllacer]

With the code just merged you can create almost any setup you can imagine (most of them dead wrong). Subvolume hierachy and File System hierachies can (and should be) totally distinct. The only things to take care are:

• A mount point must be either the partition or a subvolume, but not necesarily a first level one. (imagine a subvolume structure /snapshots, /snapshots/root /snapshots/yesterday .... you can mount /snapshots/root as /, f.i.
• It is dumb to replicate file system hierachy into subvolume one (unless there are specific btrfs reasons (nodatacow, compression, snapshots policy, ...). To create / and /home is dumb to make subv. @ and @/home (parent / child), but a good idea to make @ and @home (siblings, see there is no slash here).
  As a matter of fact, it might not even work. Using the still defaults at archinstall (fiercely copying FS hierachy at the subvolume level), results in only / being mounted

[KhalilSantana]

Commenting a little bit on @ShaunTheQuietGamer's layout:

/srv
Contains data for Web and FTP servers. It is excluded to avoid data loss on rollbacks.

I also do this on my system, but omitted from my original proposal for KISS principles, but it's a pretty sound suggestion in my opinion.

/opt
Third-party products usually get installed to /opt. It is excluded to avoid uninstalling these applications on rollbacks.
/usr/local
This directory is used when manually installing software. It is excluded to avoid uninstalling these installations on rollbacks.

While I can understand the point you are making, I also think this is somewhat of a foot gun: applications seldom exist within a vacuum, they have dependencies that may be scattered elsewhere on the filesystem, thus rolling back the rootfs without also rolling back these may not lead to a working (read: consistent) set of packages.

And yes, one could configure snapper to also snapshot this in sync with the rootfs subvol, then figure out a way to rollback from that (either manually, or via snapper itself somehow??), but that feels like overcomplicating things.

/tmp
All directories containing temporary files and caches are excluded from snapshots.

I'm not sure what openSUSE does with /tmp, but on ArchLinux, /tmp is a ramdisk (tempfs) by default, so I don't see the point on this being a subvol.

/var
This directory contains many variable files, including logs, temporary caches, third party products in /var/opt, and is the default location for many virtual machine images and databases. Therefore this subvolume is created to exclude all of this variable data from snapshots and is created with Copy-On-Write disabled.

I don't fully understand how Archlinux package manager (libalpm) works, but I believe it uses /var/lib/pacman/local to keep track on what's installed or not, thus having /var out of sync (ie: after a restore) from the rootfs would lead from pacman -Q disagreeing on what's on the rest of the filesystem (say, /usr/bin/firefoxbeing installed or not).

If my understanding is correct, splitting the entire /var into a subvol should be avoided, as it would break pacman each and every time you rollback.

--

Regarding CoW vs no-CoW I also disagree, nodatacow is basically an escape hatch, that disables or otherwise breaks many core features of BTRFS, including, but not limited to:

• Disables checksums
• Disables compression
• Scrub can't find corruption on nodatacow stuff as there are no checksums to verify
• Even trivial cases like a -mraid1 -draid1 cannot detect which disk has the newest or correct version of a nodatacow file, so there's no (good) way to "sync" an array with nodatacow files.
• and probably more

Those are very big downsides which should be decided on a per-application basis, either by the user or by the application itself, not a blanket disable on the entire /var directory.

[DrymarchonShaun]

(To Khalil) Your points make total sense, I see no issue with modifying the layout to work better with arch, every distro is different and is going to require different things. I guess I should specify/have specified that it was more that I think it's worth looking at opensuse's layout to see if anything they are doing would makes sense to add for the (or one of the) defaults for arch. I think the only thing that is absolutely required (subvolume wise) for snapper is for the original root be mounted in /@/.snapshots/1/snapshot, having .snapshots mounted at /@/.snapshots may also be required. Speaking as the one that's not having to do any of the work, (you or others may have a different opinion as the ones actually implemented this,) I would think is important to give choices. Rather than have the only default option be timeshift, give the choice between the two (or a totally manual layout, obviously.) Part of the problem with choosing one over the other is that, for snapper at least, you have to modify some configs during the installation to make it function correctly. (Setting an option to true in grub, modifying fstab to not force /@/.snapshots/1/snapshot to be mounted, and instead tell it to mount the default subvolume, etc.)

[DrymarchonShaun]

I really suck at explaining things, and snapper doesn't have nearly enough documentation on the required layout...
I'm going to try to make it a bit more understandable, and try to change it to match the points Kalil made.

Subvolumes	Filesystem mount point
/mnt/@/	N/A
/mnt/@/home	/home
/mnt/@/root	/root
/mnt/@/srv	/srv
/mnt/@/var_log	/var/log
/mnt/@/var_pkg	/var/cache/pacman/pkg
/mnt/@/.snapshots	/.snapshots
/mnt/@/.snapshots/1/snapshot	/

just for the sake of clarity from here down, / is the filesystem root, \ is the partition root that can be mounted somewhere within the filesystem root but doesn't have to be mounted. I've been using / for both and I think that's the main reason why there is some confusion
I'm not entirely sure why opensuse does \@/* instead of \@*, I think the idea is to distinguish the file system root / and the files in it from the partition root \@/ and the subvolumes in it. If I say \@/.snapshots I'm talking about the .snapshots subvolume, if I say /.snapshots I'm talking about the filesystem location, its to stop the confusion that is currently happening. Regardless, its what is required for snapper.

Actually, I think I might have an analogy. Think of each of the subvolumes as physical disks on a normal system within /dev/*. They are all in /dev so you know they are physical drives, just like the subvolumes are in \@/* so you know they are subvolumes, because technically you could create a text file in the partition root right next to '@', (although I imagine its bad practice,) and if it was all just in the partition root it could get confusing.

Anyways, the filesystem root is mounted as @/.snapshots/1/snapshots (1 is just a folder in the @/.snapshsots subvolume,) because it is essentially the first snapshot of the system. When you cd into @/.snapshots/1/snapshots its your current filesystem. The next snapshot that is created is @/.snapshots/2/snapshot/ and if you cd into it, its a copy, a snapshot, of a root filesystem.

[Torxed]

Subvolumes Filesystem mount point
/mnt/@/ N/A
/mnt/@/home /home
/mnt/@/root /root
/mnt/@/srv /srv
/mnt/@/var_log /var/log
/mnt/@/var_pkg /var/cache/pacman/pkg
/mnt/@/.snapshots /.snapshots
/mnt/@/.snapshots/1/snapshot /

This looks pretty similar to what we got with the exception of the root subvolume not being mounted and a separate @/root being created. And I think the current code can handle this one with a breeze. @wllacer will have to correct me if my assumptions here are wrong.