pid1 fails to start if /tmp doesn't exist

[AdrianVovk]

systemd version the issue has been seen with

v253.1-0-g6c327d74aa0d350482e82a247d7018559699798d

Used distribution

carbonOS (WIP A/B updates branch)

Linux kernel version used

v6.2.10-0-gcb9384f7deb195360e7cb2a1aee2a31792c4106c

CPU architectures issue was seen on

x86_64

Component

systemd

Expected behaviour you didn't see

1. systemd should be able to boot with root=tmpfs passed in. w/ v252 this did work, but after upgrading to v253 it stopped. This is because systemd itself started depending on the existence of /tmp during early boot, contrary to the documentation: Safely using /tmp

Both /tmp/ and /var/tmp/ are not necessarily available during early boot, or — if they are available early — are not writable. This means software that is intended to run during early boot (i.e. before basic.target — or more specifically local-fs.target — is up) should not attempt to make use of either

2. The documentation about generators should be updated to indicate that generators can now rely on having a private /tmp directory (and are sandboxed in general). As far as I can tell, the current documentation does not mention the changes introduced in v253

Unexpected behaviour you saw

When attempting to set up the sandbox for generators, systemd will try to overmount /tmp (so that generators can have a private /tmp directory, as documented in this changelog). However, in this root=tmpfs situation, /tmp does not exist: it's supposed to be created by tmp.mount or (if the unit is disabled) by tmpfiles. Neither of these have executed this early, so the overmounting fails

Steps to reproduce the problem

Boot a system with root=tmpfs (in my case, w/ usrhash=...). When transitioning out of the initramfs, pid1 will crash

Additional program output to the terminal or log subsystem illustrating the issue

systemd[1]: Successfully forked off '(sd-gens)' as PID 455.
(sd-gens)[455]: Failed to overmount /tmp/: No such file or directory
systemd[1]: (sd-gens) failed with exit status 1.
systemd[1]: Failed to fork off sandboxing environment for executing generators: Protocol error
<snip>
[!!!!!!] Failed to start up manager.

[AdrianVovk]

For the time being, I'm going to try a workaround for my distro: I'll add /tmp to base-filesystem so that it gets created during the transition out of the initramfs. However, I think that this is probably a bad solution to be upstreamed: in the event that tmp.mount is disabled/masked, systemd-tmpfiles would create /tmp as a btrfs subvolume. With my change to base-filesystem, tmp will always exist as a directory before tmpfiles gets to run, and so tmpfiles will never be able to make a subvolume

[poettering]

hmm, yeah, this is broken in the generator logic.

@keszybz you added this recentl. generators should really long without /tmp being around

[bluca]

The generator logic is fine without the tmpfs around, but it needs the mount point. I guess we can try to create it, and fail gracefully if we can't.