systemd-gpt-auto-generator fails silently with /usr on btrfs raid

[SjonHortensius]

systemd version the issue has been seen with

250.2

Used distribution

Arch Linux

Linux kernel version used (uname -a)

5.15.13-arch1-1

CPU architecture issue was seen on

x86_64

Expected behaviour you didn't see

running systemd-gpt-auto-generator should work, or generate a useful error

Unexpected behaviour you saw

it fails silently, increasing SYSTEMD_LOG_LEVEL=debug shows:

Found container virtualization none.
Disabling root partition auto-detection, root= is defined.
Failed to open device: No such device

strace shows this:

openat(AT_FDCWD, "/usr", O_RDONLY|O_NOFOLLOW|O_CLOEXEC) = 4
newfstatat(4, "", {st_mode=S_IFDIR|0755, st_size=70, ...}, AT_EMPTY_PATH) = 0
fstatfs(4, {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096, f_blocks=xxx, f_bfree=xxx, f_bavail=xxx, f_files=0, f_ffree=0, f_fsid={val=[0xxxx, 0xxx]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
ioctl(4, BTRFS_IOC_FS_INFO, {max_id=2, num_devices=2, fsid=xxx, nodesize=16384, sectorsize=4096, clone_alignment=4096}) = 0
close(4)                = 0
openat(AT_FDCWD, "/", O_RDONLY|O_CLOEXEC|O_PATH|O_DIRECTORY) = 4
openat(4, "sys", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 5
newfstatat(5, "", {st_mode=S_IFDIR|0555, st_size=0, ...}, AT_EMPTY_PATH) = 0
close(4)                = 0
openat(5, "dev", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 4
newfstatat(4, "", {st_mode=S_IFDIR|0755, st_size=0, ...}, AT_EMPTY_PATH) = 0
close(5)                = 0
openat(4, "block", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 5
newfstatat(5, "", {st_mode=S_IFDIR|0755, st_size=0, ...}, AT_EMPTY_PATH) = 0
close(4)                = 0
openat(5, "0:0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = -1 ENOENT (No such file or directory)
close(5)                = 0

It seems what happens is:

gpt-auto-generator::run > gpt-auto-generator::add_mounts > get_block_device_harder("/usr") > btrfs_get_block_device_fd which returns an uninitialized struct for multi-device partitions that isn't properly handled by its callers

[poettering]

So, what did you expect what happen?

if things aren't unique, i.e. if there is no single root fs backing device, then systemd-gpt-auto-generator cannot work.

This is documented:

Similarly, this generator will only look for the other partitions on the same physical disk as the root partition. In this case, boot loader support is not required. These partitions will not be searched for on systems where the root file system is distributed on multiple disks, for example via btrfs RAID.

That's from systemd-gpt-auto-generator(8).

[SjonHortensius]

fair question. My primary problem is systemctl daemon-reexec which logs the following in my journal:

systemd[3260647]: /usr/lib/systemd/system-generators/systemd-gpt-auto-generator failed with exit status 1.

I think this should output a proper error

[poettering]

It does output a debug log message, in particular with #22506 merged. I am not convinced it should generate anything noisy though, as the auto-discovery is not done on explicit user request, but a "graceful add-on", i.e. it will do its thing if things are set up that way, but if you use RAID, then they aren't, and we shouldn't complain about that, because the fact that people use RAID is not something to complain about, it's just how they configured it, and thus they apparently aren't interested in auto-detection logic.

I mean, it is documented, and it is visible if you turn on debug logging, which should be all we need.

[SjonHortensius]

I agree with that. Thanks for the cosmetic tweak!

[mvduin]

I am not convinced it should generate anything noisy though

Agreed, I think this issue has just been phrased a bit poorly: while systemd-gpt-auto-generator itself failed silently (or with a bogus message at loglevel debug), the exit code 1 resulted in a very noisy red error in journal every time systemd reloads its config. And this is most definitely a bug, a regression introduced by v250 (specifically b00651c) and fixed by #22506 which restores the behaviour that b00651c broke.