Mount option space_cache makes snapshots unbootable

[berton7]

OS: Arch linux

Currently all my mounts in /etc/fstab, most importantly my "/@" root subvolume, have the space_cache mount option;
because grub-btrfs pretty much just copies the fstab options, grub tries to mount the snapshots with space_cache as well, leading to the following boot error message:

mount: /new_root: wrong fs type, bad option, bad superblock on /dev/sdb9, missing codepage or helper program, or other error. You are now being dropped into an emergency shell.
sh: can't access tty: Job control turned off
[rootfs] #

Removing the space_cache option by editing the grub menu allows to boot correctly.

I believe some regex filters should to the trick for just ignoring this mount option.

Some references:

• /etc/fstab options filtering (in 41_snapshots-btrfs):

## Detect rootflags
detect_rootflags()
{
    local fstabflags=$(grep -oE '^\s*[^#][[:graph:]]+\s+/\s+btrfs\s+[[:graph:]]+' "${gbgmp}/${snap_dir_name}/etc/fstab" \
                        | sed -E 's/^.*[[:space:]]([[:graph:]]+)$/\1/;s/,?subvol(id)?=[^,$]+//g;s/^,//')
    rootflags="rootflags=${fstabflags:+$fstabflags,}${GRUB_BTRFS_ROOTFLAGS:+$GRUB_BTRFS_ROOTFLAGS,}"
}

• my current root mount options (/etc/fstab):

# /dev/sdb9 LABEL=root
UUID=23ae229b-334a-43dd-b08c-596748a781c4       /               btrfs           rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=256,subvol=/@       0 0

• generated mount options in grub for root (/boot/grub/grub.cfg):

echo    'Loading Linux linux-zen ...'
linux   /vmlinuz-linux-zen root=UUID=23ae229b-334a-43dd-b08c-596748a781c4 rw rootflags=subvol=@  loglevel=3 quiet nvidia-drm.modeset=1

• generated mount options in grub for a snapshot (/boot/grub/grub-btrfs.cfg):

echo 'Loading Snapshot: 2021-12-06 20:07:37 @/.snapshots/1/snapshot'
echo 'Loading Kernel: vmlinuz-linux-zen ...'
linux "/vmlinuz-linux-zen" root=UUID=23ae229b-334a-43dd-b08c-596748a781c4  loglevel=3 quiet nvidia-drm.modeset=1 rootflags=rw,noatime,compress=zstd:3,ssd,space_cache,subvol="@/.snapshots/1/snapshot"

PS: another good thing to do would be to check if the snapshot is read-only, and if it is change the "rw" option to "ro".

[Antynea]

I can't reproduce your error.

You indicate that /dev/sdb9 is your root partition and its type is btrfs.
space_cache option is valid for btrfs.
But you have to remove it to boot properly 😕

Are you using a multi-device Btrfs filesystem ? raid array or whatever ...

Do you use a modified initramfs ? No initramfs ?
Are you using mkinitcpio or dracut to generate your initramfs ? Did you change any options in the mkinitcpio.conf file ?

mount: /new_root: wrong fs type, bad option, bad superblock on /dev/sdb9, missing codepage or helper program, or other error. You are now being dropped into an emergency shell.

Generally, this means that there is a filesystem detection problem on your device.
When you are in emergency shell, can you use these commands ?

blkid -s TYPE -s UUID # to find out if /dev/sdb9 with UUID=23ae229b-334a-43dd-b08c-596748a781c4 is btrfs.
mount -t btrfs -o rw,noatime,compress=zstd:3,ssd,space_cache,subvol="@/.snapshots/1/snapshot" /dev/sdb9 /new_root # if nothing appear after that, it seems that everything is ok.
exit # continue the normal boot sequence.

I believe some regex filters should to the trick for just ignoring this mount option.

Possibly, if we can determine that the problem lies here.

PS: another good thing to do would be to check if the snapshot is read-only, and if it is change the "rw" option to "ro".

Maybe

[berton7]

So I did some more testing today; I modified the fstab to use space_cache=v2, regenerated the grub config and I could boot into the snapshots with no problems; using the old space_cache brings me back to the emergency root shell.

I also remembered I have an Arch vm setup almost exactly as my pc, just using space_cache as boot option, and there I could boot from snapshots with no problems; so at this point my guess is that I mixed the space_cache versions (v1, the default, and v2) when installing the system...

To answer your questions:

Are you using a multi-device Btrfs filesystem ? raid array or whatever ...

Nope, just plain btrfs.

Do you use a modified initramfs ? No initramfs ? Are you using mkinitcpio or dracut to generate your initramfs ? Did you change any options in the mkinitcpio.conf file ?

I'm using mkinitcpio, I added the BTRFS module to the MODULES array; I should probably note I'm using the linux-zen kernel too

Finally, trying to mount root from the emergency shell (meaning: I booted a snapshot using the space_cache option, got into emergency shell, mounted root manually with space_cache=v2) WORKS and gets me into a usable system.

[Antynea]

To answer your questions:

Are you using a multi-device Btrfs filesystem ? raid array or whatever ...

Nope, just plain btrfs.

Do you use a modified initramfs ? No initramfs ? Are you using mkinitcpio or dracut to generate your initramfs ? Did you change any options in the mkinitcpio.conf file ?

I'm using mkinitcpio, I added the BTRFS module to the MODULES array; I should probably note I'm using the linux-zen kernel too

Seems consistent to me.

Finally, trying to mount root from the emergency shell (meaning: I booted a snapshot using the space_cache option, got into emergency shell, mounted root manually with space_cache=v2) WORKS and gets me into a usable system.

This confirms a misconfiguration of your fstab in the snapshot you wish to boot from.

So I did some more testing today; I modified the fstab to use space_cache=v2, regenerated the grub config and I could boot into the snapshots with no problems; using the old space_cache brings me back to the emergency root shell.
I also remembered I have an Arch vm setup almost exactly as my pc, just using space_cache as boot option, and there I could boot from snapshots with no problems; so at this point my guess is that I mixed the space_cache versions (v1, the default, and v2) when installing the system...

You exceeded my expectations.
I did not mention the version of space_cache which could be incorrect to avoid any ambiguities in my first answer,
see here

space_cache
space_cache=version
nospace_cache
(nospace_cache since: 3.2, space_cache=v1 and space_cache=v2 since 4.5, default: space_cache=v1)

Options to control the free space cache. The free space cache greatly improves performance when reading block group free space into memory. However, managing the space cache consumes some resources, including a small amount of disk space.

There are two implementations of the free space cache. The original one, referred to as v1, is the safe default. The v1 space cache can be disabled at mount time with nospace_cache without clearing.

On very large filesystems (many terabytes) and certain workloads, the performance of the v1 space cache may degrade drastically. The v2 implementation, which adds a new B-tree called the free space tree, addresses this issue. Once enabled, the v2 space cache will always be used and cannot be disabled unless it is cleared. Use clear_cache,space_cache=v1 or clear_cache,nospace_cache to do so. If v2 is enabled, kernels without v2 support will only be able to mount the filesystem in read-only mode.

The btrfs-check(8) and mkfs.btrfs(8) commands have full v2 free space cache support since v4.19.

If a version is not explicitly specified, the default implementation will be chosen, which is v1.

NOTE: Once enabled, the v2 space cache will always be used and cannot be disabled unless it is cleared.

From your original post:

Removing the space_cache option by editing the grub menu allows to boot correctly.

So this is the result of a user configuration error.

Should we give the user the ability to remove settings that they feel are incorrect when 100% of the time it is their fault?
My answer remains the same:
NO
It is not dependent on this project. refer to #129 and discussion #133

The fstab file can be generated automatically or edited by the user, however it is up to the user that this file be consistent with his system.
Is using it wrong? Maybe ...
But it leads to this kind of discussion that allows the user to be informed of his misconfiguration ...

I believe some regex filters should to the trick for just ignoring this mount option.

I would say no, you don't correct the cause, you correct the effect.
So if the upstream configuration is incorrect, the resulting effect is not to blame.

Thank you for doing all these tests.
I am open to any suggestions.

[Antynea]

However a problem with btrfs persists.
the space_cache option should determine by itself if v2 or v1 is used...

[berton7]

I 100% agree with your thoughts. In the end just misunderstood the effects of the space_cache option, thinking that having it enabled when booting on a read-only filesystem was not possible 🥴

However a problem with btrfs persists.
the space_cache option should determine by itself if v2 or v1 is used...

Maybe? Rembember that I could boot just fine from the root subvolume, where in grub there was no space_cache specified; only when it was added (in the snapshots) as an option it wouldn't boot/mount; so I guess at some point btrfs does recognize that there is space cache active (if you don't manually specify it), but if you say that you have the v1 (default) it just "trusts" you

[Antynea]

I 100% agree with your thoughts. In the end just misunderstood the effects of the space_cache option, thinking that having it enabled when booting on a read-only filesystem was not possible 🥴

Thanks, space_cache option is used on the fs, regardless of the subvolume.

Note: most mount options apply to the whole filesystem and only options in the first mounted subvolume will take effect. This is due to lack of implementation and may change in the future. This means that (for example) you can’t set per-subvolume nodatacow, nodatasum, or compress using mount options. This should eventually be fixed, but it has proved to be difficult to implement correctly within the Linux VFS framework.

Their documentation lacks a precise explanation.

If a version is not explicitly specified, the default implementation will be chosen, which is v1.

So I understand that if we use space_cache then =v1
But if you don't specify anything, what happens if a cache version is still present ?

Removing the space_cache option by editing the grub menu allows to boot correctly.

Ok, is the cache used ? Is the right version detected ?
if not, cache are not used ?
maybe dmesg informs us if space_cache is used, I don't know where to find the information.
When your are boot without option, did you check if findmnt returns a space_cache parameter ?

[geckolinux]

This is a known issue with kernel 5.15, Calamares adds this advisory:

https://github.com/calamares/calamares/blob/calamares/src/modules/fstab/fstab.conf

# With kernels 5.15 and newer be cautious of adding the option space_cache
# to the btrfs mount options.  The default in 5.15 changed to space_cache=v2.
# If space_cache or space_cache=v1 are specified, it may fail to remount.

[Schievel1]

I think this is stale. Is this still a problem with newer kernels?