Non-overridable paths in systemd.pc cause troubles for projects in certain use cases

[jtojnar]

systemd version the issue has been seen with

247

Used distribution

Reproduced on NixOS but it affects any distribution and configuration.

Unfortunately, it turned out the pkg-config file change from #17009 did actually break packages built using Nix, since many projects do depend on the prefix being overridable.

The support for overridable prefix really has two use cases:

• Installing to non-FHS package-specific prefixes for distros with immutable roots like NixOS. Those cannot use /usr. Of course it is then their responsibility to ensure systemd can find the service files.
• Installing to local path for development purposes. This is a compromise between running program from a build directory and running a program fully installed to system prefix. You will not want to write to your system prefix in that case.

We are reverting it locally for now, but for supporting either use case, you will need to support overridable prefix so that a project can be successfully installed as per https://www.bassi.io/articles/2018/03/15/pkg-config-and-paths/. Hardcoding the prefix means projects will no longer have reason (or a sensible way) to install the data relative to prexix (which will be /usr anyway on modern FHS distros) and it will force users wanting to use one of the aforementioned use cases to patch each project’s build system.

Other alternatives are asking projects to add a configure/meson flags for setting various systemd flags, or overriding the paths completely, through PKG_CONFIG_SYSTEMD_* environment variables. But both mean more work for projects wanting to install systemd files.

Downstream issue: NixOS/nixpkgs#106351

[bluca]

• Installing to local path for development purposes. This is a compromise between running program from a build directory and running a program fully installed to system prefix. You will not want to write to your system prefix in that case.

That's what using DESTDIR when installing is for, no?

Anyway, as it was mentioned in the PR, the problem was that usage of the prefix was inconsistent. The pc file is only one part of it. If the lack of configurability breaks downstreams then it should be fine to bring it back, but it has to be consistent - ie, across sd_path&friends as well.

[poettering]

We never supported anything but installation in /usr, and that has always been documented in README.

Have you read the documentation?

I really don't understand why "immutable" OSes, shouldn't use /usr? That appears entirely orthogonal?

I am not sure what NixOS does, but it's certainly not using systemd like we designed it. Which is entirely OK, downstreams are completely OK to patch the hell out of our software if they like, but if you override $prefix you still will be outside of any use we recommend.

If you want to use different paths, we recommend symlinks or bind mounts. In general we consider paths API, i.e. apps should be able to use them as stable references to stuff. This starts from the kernel (which will invoke the systemd binary on boot), but continues wherever apps drop stuff into our dirs. If they were configurable then things would be a mess, as apps would drop stuff at the wrong places and systemd would search for them at the wrong places.

I am not convinced we should change anything here.

[jtojnar]

• Installing to local path for development purposes. This is a compromise between running program from a build directory and running a program fully installed to system prefix. You will not want to write to your system prefix in that case.

That's what using DESTDIR when installing is for, no?

No, DESTDIR is for when you want the build system to install to a different directory than the one where it will be installed in the end (e.g. by extracting from a package by a package manager). If you used DESTDIR=local/path the project would still hardcode the original $prefix instead of local/path, defeating the purpose of this use case.

The desired result can be achieved using DESTDIR=tempdir and --prefix=local/path, and then moving the contents of tempdir/local/path to local/path manually but that is pretty clumsy and bothersome, especially if pkg-config has --define-variable flag precisely for this use case.

Anyway, as it was mentioned in the PR, the problem was that usage of the prefix was inconsistent. The pc file is only one part of it. If the lack of configurability breaks downstreams then it should be fine to bring it back, but it has to be consistent - ie, across sd_path&friends as well.

I think limiting the change of src/core/systemd.pc.in in #17009 to the following (not removing the ${prefix} interpolation in variables) would achieve that without breaking these use cases:

---src/core/systemd.pc.in
+++src/core/systemd.pc.in
-prefix=@prefix@
+prefix=/usr

[bluca]

Anyway, as it was mentioned in the PR, the problem was that usage of the prefix was inconsistent. The pc file is only one part of it. If the lack of configurability breaks downstreams then it should be fine to bring it back, but it has to be consistent - ie, across sd_path&friends as well.

I think limiting the change of src/core/systemd.pc.in in #17009 to the following (not removing the ${prefix} interpolation in variables) would achieve that without breaking these use cases:

---src/core/systemd.pc.in
+++src/core/systemd.pc.in
-prefix=@prefix@
+prefix=/usr

That's a good point, pkg-config has precise semantics around the prefix variable (eg: --define-prefix), and we shouldn't break it. Feel free to send a PR for that.

[jtojnar]

We never supported anything but installation in /usr, and that has always been documented in README.

That is okay but this is not actually about where systemd is installed. The pkg-config file also directs where other projects install their files and, for the aforementioned use cases, having the prefix overridable is necessary.

I really don't understand why "immutable" OSes, shouldn't use /usr? That appears entirely orthogonal?

Sorry, that was a shortcut. I meant Nix-style immutable package managers (e.g. Guix), even though other constraints require not using /usr, not the immutability itself.

I am not sure what NixOS does, but it's certainly not using systemd like we designed it. Which is entirely OK, downstreams are completely OK to patch the hell out of our software if they like, but if you override $prefix you still will be outside of any use we recommend.

I would agree with you except that this change affects other projects than just systemd and breaks other stuff than NixOS. The pkg-config file overridability has been advocated by GNOME/Freedesktop people long before us: https://www.bassi.io/articles/2018/03/15/pkg-config-and-paths/

If you want to use different paths, we recommend symlinks or bind mounts. In general we consider paths API, i.e. apps should be able to use them as stable references to stuff. This starts from the kernel (which will invoke the systemd binary on boot), but continues wherever apps drop stuff into our dirs. If they were configurable then things would be a mess, as apps would drop stuff at the wrong places and systemd would search for them at the wrong places.

Yes, symlinks are in fact what we use for things that need to be globally discoverable. We create a package with a directory tree linking the service files of all packages that are to be globally installed on the system and then link that directory tree to our FHS-like root¹ and make systemd look in there instead of /usr².

---

Footnotes

1. We use /run/current-system/sw instead of /usr to prevent software hardcoded to use /usr from looking for libraries installed globally instead of the explicitly declared dependencies, which would reduce reproducibility. ↩

2. Actually, this is not the case for systemd – it looks into /etc so just abuse³ that by linking the services there instead.⁴ But we do use that strategy for other stuff like ↩

3. Thanks to the way we build our unit dirs, we do not need to distinguish units created by the administrator from those installed by them on the file system, making this abuse tolerable. ↩

4. We consider /etc lesser risk for reproducibility so we kept that from FHS. Though, of course, it is also just a bunch of symlinks. ↩

[boucman]

$preix also needs to be overridable for cross tools

the RPATH that meson sets is based on $prefix, and can only be /usr (as @poettering mentionned)

When compiling cross-tools (i.e a version of systemctl that is run on the host to configure the target) the RPATH will point to a wrong version of libsystemd, since it will be the one for the host, not the one for the cross-tool.

Under buildroot, we end up rewriting the RPATH after the fact to make cross tools work correctly.

And yes, we need cross tools. We had real bugs due to different versions of systemctl on the host and on the target. In particular on debian systems which are spli-usr whereas buildroot systems are not....

[bluca]

Yeah the cross compilation case is also a very good point