unmerged-usr doesn't apply to all distros

[ncfavier]

Component

systemctl, systemd

Is your feature request related to a problem? Please describe

See NixOS/nixpkgs#184826 for context.

systemctl status now reports Tainted: unmerged-usr when /bin is not a symlink to /usr/bin, but this heuristic fails on NixOS because everything is merged into /nix/store but we still have /usr/bin/env and /bin/sh links for compatibility.

Describe the solution you'd like

I'm not sure what exactly the best solution would be, but maybe simply checking os-release and avoiding the unmerged-usr check if we're on NixOS?

Describe alternatives you've considered

See NixOS/nixpkgs#184826

The systemd version you checked that didn't have the feature you are asking for

251

[bluca]

The taint is there to give a clear indication: we support something, and what is detected at runtime is not it, so be warned as things might break. It is perfectly fine if Nixos has different filesystem layouts and wants to support them, but then such support should be provided downstream.

We shouldn't add distro-specific heuristics upstream because we intend to drop support for anything other than merged-usr upstream eventually, so it's not just about this single taint, but other future changes too.

[poettering]

So if /bin/, /usr/ and /usr/bin/ do exist for compat on nixos, why doesnt /bin/ symlink to /usr/bin/ for compat, too? Compat or not, there is no reason to have those two dirs separate...

[K900]

Mostly because NixOS allows a very limited set of things in the build sandbox, and currently it's specifically /bin/sh and /usr/bin/env. If we do make those symlinks, we'll have two ways to refer to things in the sandbox which isn't really a big deal but also seems somewhat unnecessary?

[bluca]

I mean we could use /usr/lib for the heuristic instead of /usr/bin? But the general point remains, a custom layout is something that the distro providing it needs to support

[K900]

For NixOS there is no /usr/lib. Another thing we could do is just add a separate nixos taint flag downstream maybe?

[bluca]

Well the purpose of the taint flag is to catch the attention of the user toward something that isn't quite right, so it doesn't sound like it would be useful?

[K900]

The user or the systemd maintainers? I was under the impression it was largely the latter.

[poettering]

Distros merged /bin/ + /sbin/ into /usr/ precisely to minimize unnecessary compat issues since all binaries are now available under either path. If you really insist that the dirs should be different then you are just creating problems for yourself for no reason, even though the other distros made the difference just go away. Sounds weird though that even though you say "compat" is the reason for keeping things separate you just create incompatibilites because you insist on the 1990s paths for these things...

Also why even? Either way you need 4 inodes (two symlinks for the binaries, plus one dir, plus either a second dir or a third symlink), so what are you trying to save here?

I am pretty sure this is something for nixos to fix. If you create incompat for no reason then you have to deal with the incompat yourself i guess...

Or in other words: the current nixos setup delivers incompatibility -- not compatibility.

[bluca]

The user or the systemd maintainers? I was under the impression it was largely the latter.

The runtime taints in systemctl are mainly for users, and secondarily for distribution maintainers (but build-time ones are mainly used for that purpose)

[winterqt]

Per the original PR that added this taint, systemd is preparing to "remove support" for unmerged-user. What exactly does that mean in this context? For example, does it mean that things will start breaking, or simply that you will stop providing support for unmerged-usr setups (without breaking things in a technical manner)?

[poettering]

it means for example that we'll install binaries into subdirs of $prefix (i.e. /usr/) simply, instead of distinguishing $rootprefix (i.e. /) from it. i.e binaries would jut end up in /usr/bin/ instead of /bin/.

[poettering]

Anyway, I am pretty sure this is something to fix in NixOS. And if they don't, and want to deviate from other distros and insist that /bin/ is a dir, and not a symlink, then they should carry the burden. Given that the taints are mostly just cosmetics, and do not change behaviour I think we shouldn't change anything here.

In a way the taints are serving their purpose here: they make incompatible NixOS behaviour visible, so that people reported an issue. Against the wrong component, but well, at least it's tracked somewhere now.

So NixOS has two options:

1. fix their stuff, and make /bin/ a symlink, like everyone else
2. accept the ugly taint, and just live with it.

But here, on the systemd side, I think we should just close this...

[arianvp]

(systemd maintainer for NixOS here). I'm reading up on the discussion still to fully grasp what the issue being reported is. Will discuss it with @flokli asap but I think I agree with Lennart's assessment that we just have to cary this burden in our fork; if any.

If we come up with some change that is upstream-able and makes our lives easier whilst not burdening upstream we'll open a new RFE. So feel free to close.

[flokli]

NixOS has very different approach towards /bin and /sbin - they don't really exist (except for /bin/sh and /usr/bin/env, because these are used heavily in shebangs around the world).

The concept of merged or unmerged usr doesn't really apply here at all, it's neither of the two.

Users don't execute things from /bin or /sbin, there's nothing else in there.

All paths to binaries are hardcoded into the applications executing them at compile time, or $PATH is set from the outside.

In the case of systemd, things are quite simple. systemd can find its own binaries (like systemd-tmpfiles being specified in the unit files without absolute path) due to us patching DEFAULT_PATH_NORMAL to ${placeholder "out"}/bin/, which will become something like /nix/store/yzycn28101y9v6disd23r86g1q3n4dsj-systemd-251.3/bin, and all other paths to binaries are provided absolutely.

In some other cases, from some generators, systemd assumes some binaries to be available (mkfs.*, fsck, …). We fix this by setting $PATH before invoking systemd, and carry a downstream patch to propagate $PATH down to these generators - even though this could probably be provided by our own environment generator.

In all cases, systemd doesn't look things up from /usr/bin or /bin, so showing the taint isn't really helpful.

NixOS is a bit exotic here indeed, we've been shipping NixOS-specific patches downstream, and think we'll keep doing so. I however also agree with @ncfavier that the taint in systemctl status is misleading.

Is there a reason to not drop unmerged-usr support from systemd upstream entirely for good, simplify your logic around that to only look in one place ($prefix), instead of showing taints that users don't really know how to act upon?

If support for unmerged-usr can't be dropped any time soon, could you show the distro name in systemctl status too? I could then foresee NixOS patching out showing that taint.

[bluca]

Is there a reason to not drop unmerged-usr support from systemd upstream entirely for good, simplify your logic around that to only look in one place ($prefix), instead of showing taints that users don't really know how to act upon?

We need to wait for several distros to complete the switchover, so it cannot happen yet and for a while.

If support for unmerged-usr can't be dropped any time soon, could you show the distro name in systemctl status too? I could then foresee NixOS patching out showing that taint.

I'm not sure that would be useful, as that's not the purpose of the taint. I'd suggest to patch it downstream

[poettering]

NixOS has very different approach towards /bin and /sbin - they don't really exist (except for /bin/sh and /usr/bin/env, because these are used heavily in shebangs around the world).

See but that's the thing. because virtually all big, relevant Linux distros merged /usr/ these days you'll sooner or later find scripts that will put something else there, i.e. /usr/bin/sh or /bin/env. Simply because these things are often auto-generated and which sh nowadays reports /usr/bin/sh on many of those distros.

So if you are doing this for compat, then be compatible. Otherwise it's not worth the effort, and you can dump the whole thing?

[flokli]

See but that's the thing. because virtually all big, relevant Linux distros merged /usr/ these days you'll sooner or later find scripts that will put something else there, i.e. /usr/bin/sh or /bin/env. Simply because these things are often auto-generated and which sh nowadays reports /usr/bin/sh on many of those distros.

/usr/bin/env $application isn't a linux distro specific. It's a pattern used in more unixes to find that binary, while honoring $PATH.

For example, there's no /bin/bash in OpenBSD, but it'll be in /usr/local/bin/bash. Using #!/usr/bin/env bash will succeed though. So people providing bash scripts that should be somewhat portable use #!/usr/bin/env bash as shebang.

Python also seems to encourage using /usr/bin/env python3, so python from virtualenvs are picked up properly.

See but that's the thing. because virtually all big, relevant Linux distros merged /usr/ these days you'll sooner or later find scripts that will put something else there, i.e. /usr/bin/sh or /bin/env.

I don't think people will suddenly start writing /bin/env bash instead, "just because it's reachable now too". The main argument for using /usr/bin/env and is portability of the shebang across systems. Similarly, there's some agreement on /bin/sh pointing to some interpreter, even though it might not be bash, so that's seen in the wild, too.

NixOS supports these two patterns for compat, by providing symlinks into the nix store there, but we don't want to extend this to any permutations that might work on other distros.

Simply because these things are often auto-generated and which sh nowadays reports /usr/bin/sh on many of those distros.

That's not an issue for NixOS. The which sh commands are executed during build, by the build system, in a sandbox. /bin and /usr/bin are not in $PATH, but whatever shell we want it to see, so something like /nix/store/2r9n7fz1rxq088j6mi5s7izxdria6d5f-bash-5.1-p16/bin/sh will be returned by which sh.

Anyways, I agree this is probably out of scope for systemd. As long as systemd has some unmerged-usr support left, we can probably ship a patch downstream removing the taint warning (as it doesn't apply for NixOS).

Your issue templates ask for the distro being used, so NixOS users should still be clearly recognizable, and most of the time, people seem to be asking in https://github.com/nixos/nixpkgs first anyways ;-)

[poettering]

I don't think people will suddenly start writing /bin/env bash instead, "just because it's reachable now too". The main argument for using /usr/bin/env and is portability of the shebang across systems. Similarly, there's some agreement on /bin/sh pointing to some interpreter, even though it might not be bash, so that's seen in the wild, too.

it's not so much about people actively putting /usr/bin/sh in the shebang line. It's more about packages which do autoconf style @SHELL@ templating, where it will now end up pointing to that.

That's not an issue for NixOS. The which sh commands are executed during build, by the build system, in a sandbox. /bin and /usr/bin are not in $PATH, but whatever shell we want it to see, so something like /nix/store/2r9n7fz1rxq088j6mi5s7izxdria6d5f-bash-5.1-p16/bin/sh will be returned by which sh.

Sure, if the shell scripts originate in nix, that is. Which they might or might not.

[poettering]

What I really don't get though is why it's simportant for nix if you have to create the inodes anyway for /bin/sh and /usr/bin/env and thus also /usr/ and /usr/bin, why you don't simply do what other distros do, and make /usr/bin/ a symlink to /bin/.

I mean, it's irrelevant to me where you actually put your stuff and what your model is, but if you accept that you have to have those four inodes for compat, why not arrange them like evryone else? why arrange them in a 1990s way? Why do you think if you have to have them is /usr/bin being a dir so much more in line with nixos than it being a symlink?

[flokli]

What I really don't get though is why it's simportant for nix if you have to create the inodes anyway for /bin/sh and /usr/bin/env and thus also /usr/ and /usr/bin, why you don't simply do what other distros do, and make /usr/bin/ a symlink to /bin/.

NixOS/nixpkgs#185040 (comment)

[keszybz]

What I really don't get though is why it's simportant for nix if you have to create the inodes anyway for /bin/sh and /usr/bin/env and thus also /usr/ and /usr/bin, why you don't simply do what other distros do, and make /usr/bin/ a symlink to /bin/.

Yeah.

and would rather prefer shipping a downstream systemd patch in nixpkgs that stops showing the confusing taint in the systemctl status command

OK. I think it's complicating things for no good reason, but that is a valid downstream choice.

Let's close this here.

[poettering]

What I really don't get though is why it's simportant for nix if you have to create the inodes anyway for /bin/sh and /usr/bin/env and thus also /usr/ and /usr/bin, why you don't simply do what other distros do, and make /usr/bin/ a symlink to /bin/.

NixOS/nixpkgs#185040 (comment)

You are not adding "impure" paths. You are having 4 "impure" inodes right now. If you change one of them from dir to symlink you still have exactly 4 "impure" inodes, except you actually do what everybody else does, i.e. one of them is now a symlink, not a dir.

Anyway, I think we can end this discussion, apparently one symlink instead of one dir is too much for nix, but i guess nix has to live with the outcome of that, we don't have to care.

[ncfavier]

Making /bin a symlink does add two (non-canonical) paths to the filesystem: /bin/env and /usr/bin/sh. Those paths are called impure because their contents depend on the current state of the system, rather than being purely determined by the path itself. In other words, the function that reads the contents of a file given its path is not pure on these paths. "Impure inodes" aren't a thing because we don't access files by inodes.

Not saying I necessarily agree with the decision, just... don't misrepresent it, please.

[K900]

I think it's worth elaborating a bit more, as Lennart and others likely aren't familiar with how NixOS works internally: in NixOS, packages live in /nix/store/<unique hash>-<name>/, which is a directory managed by the Nix package manager, and well-behaved packages are expected to refer to things by the full store path, e.g. /nix/store/yaddayadda-bash-5.1/bin/sh instead of just /bin/sh, so that Nix can track dependencies by simply checking the package for references to other store paths.

The symlinks at /bin/sh and /usr/bin/env are only there for compatibility with existing outside-of-Nix scripts, and are set up during bootup based on the current "default" version in the system configuration, which is something we want to do as little as possible, as it makes things difficult to track and potentially leaks ambient system state into packages.

I'm not sure whether having these specific symlinks there or not actually matters in any real way, but that's an internal NixOS discussion that we shouldn't be dragging people here into.

Hopefully this helps explain why we're doing what we're doing.

[poettering]

Please, can someone explain me seven more times how Nix works, I can never have enough of that. Thanks!

Please, let's end this here, I understand there's more interest in a eschatological concept of purity than into actually addressing issues, which is fine, but it also means we don't really have to care upstream here anymore, so let's not continue the discussion here. Thank you for understanding.