Reduce dependencies of libsystemd

[Lastique]

Component

systemd

Is your feature request related to a problem? Please describe

The recent sshd/xz backdoor fiasco (CVE-2024-3094) has shown that the extra dependencies introduced by libsystemd may be the source of vulnerabilities in core and sensitive components on the system. libsystemd is being linked into all systemd services, as well as any third party services that intend to interact with systemd through the C API. For example, any service that wishes to notify systemd of its startup and termination would call sd_notify family of functions. Any process that wishes to write logs to systemd-journal would call sd_journal_print family of functions. And so on.

Describe the solution you'd like

This issue is asking to reduce the dependencies of libsystemd to the bare minimum, which is libc. Currently, on Kubuntu 22.04 (libsystemd0 version 249.11-0ubuntu3.12) ldd libsystemd.so.0 shows:

        linux-vdso.so.1 (0x00007fff97fbd000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f9519a77000)
        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f95199a8000)
        liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f9519988000)
        libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f951997d000)
        libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f951983f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9519600000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f9519b93000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f95195da000)

I believe, most of these dependencies are not necessary to implement core libsystemd functions, like those mentioned above.

This issue may mean splitting libsystemd to multiple libraries implementing different APIs, one of which, say, libsystemd-core, would only depend on libc, and other, more specialized libraries, would add other dependencies. Also, if some of the dependencies are only needed by certain systemd services, move the dependencies to those services.

The ultimate effect of this should be reduced attack surface and improved system security.

Describe alternatives you've considered

For some APIs, there are documented underlying protocols (e.g. there is native journal protocol). Third party processes could be implementing those protocols to interact with systemd. However, this approach is counter-productive and error-prone, as every process would need to implement the same protocol, and may do it incorrectly. This would also make future evolution of these protocols more difficult as there would be many implementations of it that would need to be upgraded. Furthermore, this does not solve the problem for systemd itself, as the dependencies remain loaded into every systemd process, whether they are needed or not.

The systemd version you checked that didn't have the feature you are asking for

249

[DaanDeMeyer]

In the next release all the compression libraries and gcrypt will be dlopen() style dependencies which means that the libraries will only be loaded if the relevant functionality in libsystemd is actually used by the service.

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/4VRKIAQAXGXU7C7A2ERO7SN3WTC5ML33/ gives an overview of why splitting libsystemd is not trivial and involves other tradeoffs.

[Lastique]

I think, dlopen-style usage would make the problem more difficult to diagnose, as it isn't obvious which APIs use which third-party libraries. It might be an improvement over the status quo in the immediate future, but I would still prefer the proper solution, where the dependencies are linked in and libsystemd is split. I understand this may mean a lot of refactoring work, but this is what needs to be done, IMO.

[Conan-Kudo]

I do think we need to start strongly considering splitting libsystemd back up into component libraries. Yes, this will be rather painful because there have been no architecture boundaries for the library code for 10 years now, but this incident has shown that the architecture of integration matters a lot.

The dlopen() path doesn't actually help us because it isn't changing the architecture, it just obscures what systemd uses to both users and maintainers.

[ncopa]

I would love to see the big mono-repo split up. There are many good things in systemd that I'd love to use in Alpine Linux but can not, due to only GNU libc is supported. A few things that could be nice to have separated out are:

• systemd-boot (gummiboot)
• sd_notify
• udev

It would be very nice if those could be built independently by init systems that have musl libc support.

[Conan-Kudo]

I don't think that's realistic or desirable by the contributors. And none of us in this issue are asking for that. But having a modular architecture does not necessarily require a polyrepo setup. And Alpine can still take the source code and build only the components they want to use.

As for Musl libc support, I think that's largely dependent on beefing up musl to support the missing primitives. Nobody here is against it as long as the stuff needed by systemd is present on musl-based systems.

[YHNdnzj]

@ncopa That's a different topic, and I don't think it's desirable. Maintaining components in one repo has a major benefit that allows us to reuse basic/ and shared/ util libs. Each component can still be built standalone, one repo or not.

Regarding musl, people from postmarketOS are working on it AFAIK. But that's pretty irrelevant too.

[jbwyatt4]

Some ignorant questions:

Why does libsystemd need 3 different compression libraries?

Can they be reduced?

(As I was typing this the above MR for dynamically loading compression libraries was merged.)

[bluca]

We are not going to split again the library, because that just creates usability, maintainability and integration problems for no gains. The dependencies will all be optional via dlopen, and I'll also document that using sd_journal APIs will result in optional dependencies to be loaded.

[ncopa]

I don't think that's realistic or desirable by the contributors. And none of us in this issue are asking for that. But having a modular architecture does not necessarily require a polyrepo setup.

Fair enough. My thinking here was poly repo setup would likely help enforcing a modular architecture if that was a goal. I just wanted to mention that it would also be beneficial for non-supported platforms.

And Alpine can still take the source code and build only the components they want to use.

Sure, I'm just saying it would be nice to have it more separated.

Nobody here is against it

Being against and supporting something are two different things.

as long as the stuff needed by systemd is present on musl-based systems.

This means "not supported".

Thanks for listening.

[bluca]

This is the dependency tree with a full-feature build and the pending PRs merged:

build/libsystemd.so.0 (interpreter => None)
    libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
    ld-linux-x86-64.so.2 => /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

There's a TODO item to replace libcap with ioctls, maybe this is a good time to look into that too? Any volunteers?

[bluca]

After lunch I'll also work on adding an MIT-0 example to the manpage on how to reimplement sd_notify in C, so that it is copy/paste ready. The notify protocol is stable and trivial for the manager <-> service case, and if one wants to prioritize avoiding dependencies we should facilitate that.

[YHNdnzj]

The dlopen() path doesn't actually help us because it isn't changing the architecture, it just obscures what systemd uses to both users and maintainers.

Well, nobody is against adding dlopen-ed deps to ELF metadata (#31131 (comment)). But there's no standard on this and I doubt we'll ever get that soon.

[kampka]

For some APIs, there are documented underlying protocols (e.g. there is native journal protocol). Third party processes could be implementing those protocols to interact with systemd. However, this approach is counter-productive and error-prone, as every process would need to implement the same protocol, and may do it incorrectly. This would also make future evolution of these protocols more difficult as there would be many implementations of it that would need to be upgraded.

I'm sorry if I'm sidelining the discussion a bit, but if I understand @poettering here correctly, these protocols are documented with the explicit intent ("on purpose") to enable third party projects to re-implement them as they see fit. As such, any evolution of these protocols would surely have to take that into account already. In his post he is talking about sd_notify, which is of course a much simpler protocol that the journal native protocol, but I understood the point to apply any Systemd protocol documented in such a way.
If my understanding is wrong, if would be nice if you could clarify the Systemd position with regards to these documentations, so I and probably others don't program myself into a long-term corner. Thanks :)

[bluca]

stable interfaces and protocols are already documented at: https://systemd.io/PORTABILITY_AND_STABILITY/

[Lastique]

I'm sorry if I'm sidelining the discussion a bit, but if I understand @poettering here correctly, these protocols are documented with the explicit intent ("on purpose") to enable third party projects to re-implement them as they see fit.

The ability to reimplement the protocols is good and very welcome, but there is benefit in having a reference implementation that is easy to use in third party services without much downsides, like unnecessary dependencies. This relieves those services from having to implement, test and maintain their own implementations, however simple it may be. My understanding is that libsystemd was supposed to be that reference implementation.

[bluca]

After lunch I'll also work on adding an MIT-0 example to the manpage on how to reimplement sd_notify in C, so that it is copy/paste ready. The notify protocol is stable and trivial for the manager <-> service case, and if one wants to prioritize avoiding dependencies we should facilitate that.

--> #32030

[bhaible]

After lunch I'll also work on adding an MIT-0 example to the manpage on how to reimplement sd_notify in C, so that it is copy/paste ready. The notify protocol is stable and trivial for the manager <-> service case, and if one wants to prioritize avoiding dependencies we should facilitate that.

Even better than example code in a documentation would be a function that gets linked into the program, without actually adding libsystemd.so.N as a runtime dependency. Recall that libc.so is in fact a linker script:

OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib/x86_64-linux-gnu/libc.so.6 /usr/lib/x86_64-linux-gnu/libc_nonshared.a  AS_NEEDED ( /lib64/ld-linux-x86-64.so.2 ) )

I imagine that a similar linker script could be used for libsystemd.so:

OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /usr/lib/x86_64-linux-gnu/libsystemd_nonshared.a  AS_NEEDED ( /lib/x86_64-linux-gnu/libsystemd.so.0  ) )

libsystemd_nonshared.a would contain sd_notify and all the other "simple" API functions that don't need other shared library dependencies.

libsystemd.so.0 would contain everything, including these "simple" API functions (for binary backward compatibility).

[bluca]

No, we are not adding new libraries. If you want convenience then just use the existing library, otherwise put in some effort and use the copy pastable example.

[Conan-Kudo]

We are not going to split again the library, because that just creates usability, maintainability and integration problems for no gains.

Clearly there are gains or it wouldn't be brought up all the time, and especially now.

The dependencies will all be optional via dlopen, and I'll also document that using sd_journal APIs will result in optional dependencies to be loaded.

And you are free to do this, even though it doesn't fix any of the problems people are actually talking about.

[YHNdnzj]

We are not going to split again the library, because that just creates usability, maintainability and integration problems for no gains.

Clearly there are gains or it wouldn't be brought up all the time, and especially now.

The dependencies will all be optional via dlopen, and I'll also document that using sd_journal APIs will result in optional dependencies to be loaded.

And you are free to do this, even though it doesn't fix any of the problems people are actually talking about.

Sorry, but I really don't see the point here. If people don't use sd-journal in the first place, no extra deps will be loaded. If sd-journal is used and we split the library, you are still linking to sd-journal anyway. So what's the real gain? Essentially this is status quo, except that by splitting libsystemd we create a huge compat break.

[ericcurtin]

IIRC MacOS has weak library deps, it would be great if we had the same on Linux/ELF: i.e a way to declare that certain symbols shall be backed by a library that is laoded the moment the symbol is resolved only, and ensuring the symbol resolves gracefully to NULL if that lib cannot be found. Given we don't have that, we just build the concept manually via dlopen() right now.

Support adding this to Linux/ELF 👍 Would be a neat feature...

[ratchetfreak]

If IFUNC ever gets changed to allow a dlopen to replace already loaded symbols then we are back at square one regarding this.

That change might happen when someone half a decade from now decides that windows' automatic delay load using stubs is a great thing to have for linux.

And I very much doubt that people will have a long enough memory to remember this backdoor and be load enough to add sufficient protections (aka explicit opt-in for the delay loaded functions checked at load times).

[ericcurtin]

If IFUNC ever gets changed to allow a dlopen to replace already loaded symbols then we are back at square one regarding this.

Ultimately we have to trust some libraries/maintainers. Every library/binary/subsystem can theoretically be infiltrated. Searching for silver bullets is hard.

That change might happen when someone half a decade from now decides that windows' automatic delay load using stubs is a great thing to have for linux.

And I very much doubt that people will have a long enough memory to remember this backdoor and be load enough to add sufficient protections (aka explicit opt-in for the delay loaded functions checked at load times).

[bhaible]

@poettering

sd_notify() as API for systemd's simple ready notification protocol

How about creating a libsystemd_nonshared.a library, that contains just the 3 sd_notify* functions, and install it as described above #32028 (comment) ?

It would

• have no impact on libsystemd.so.0 and its exported symbols,
• have no impact on programs/libraries that need other sd_* symbols that these 3 ones,
• remove from /usr/sbin/sshd the DT_NEEDED to libsystemd.so.0, thus reducing the attack surface and additionally improving sshd's startup time.

And the size of libsystemd_nonshared.a would be small (something like 1-2 KB).

[bluca]

I really want this to be the last time it needs to be repeated:

As already mentioned: there is no point in going in circles. The answer is no, we are not splitting anything, and no amount of back and forth will change that. Sorry if it's not the answer you were hoping for, but let's stop here, and focus keep this ticket on the topic of what we want to do: document dlopen, remove gcrypt and cap2 linking, provide a notify self contained example

We have some real work to do, and this insistence from the peanut gallery on things that have already been clearly and unambiguously rejected is wasting everyone's time and verging on the spam side. Please stop. Thank you.

[poettering]

@poettering

sd_notify() as API for systemd's simple ready notification protocol

How about creating a libsystemd_nonshared.a library, that contains just the 3 sd_notify* functions, and install it as described above #32028 (comment) ?

It would

* have no impact on libsystemd.so.0 and its exported symbols,

* have no impact on programs/libraries that need other sd_* symbols that these 3 ones,

* remove from `/usr/sbin/sshd` the DT_NEEDED to `libsystemd.so.0`, thus reducing the attack surface and additionally improving sshd's startup time.

And the size of libsystemd_nonshared.a would be small (something like 1-2 KB).

Why though?

As it appears sshd is implementing sd_notify() on their own. Which is what I'd recommend anyway for a project like that.

https://bugzilla.mindrot.org/show_bug.cgi?id=2641

So maybe let's end this specific discussion now? I mean, let's summarize:

1. libsystemd doesn't have the compression lib hard deps anymore in git main
2. libsystemd won't even have gcrypt hard dep anymore more soon, see gcrypt: dlopenify for libsystemd #32019
3. it appears that sshd is just implementing the thing on its own anyway, which I'd recommend anyway.

So, however you want to spin this, it's handled from both directions now: from sshd, from systemd.

If you want us to expose the dlopen information in way programs can read it from ELF metadata, then we are on board, write a spec, we'll help from our side, and make sure to get at least one relevant potential user of this data on board too (dpkg, rpm, ldd, initrd builders, …).

But for me that settles things pretty comprehensibly.

[YHNdnzj]

remove from /usr/sbin/sshd the DT_NEEDED to libsystemd.so.0, thus reducing the attack surface and additionally improving sshd's startup time.

The openssh upstream is already reimplementing the notification protocol on their own: https://bugzilla.mindrot.org/show_bug.cgi?id=2641#c13

[bhaible]

As it appears sshd is implementing sd_notify() on their own.

Indeed; a good move.

Why though?

As a convenience for the other packages, other than openssh, that need sd_notify:

389-ds-base
amavisd-new
apache2
apertium-apy
apt-cacher-ng
asterisk
autofs
avahi
biboumi
bind9
bluez
brltty
budgie-session
casync
chromium
conntrack-tools
corosync-qdevice
crun
cubemap
davmail
dbus
dbus-broker
dlm
dlt-daemon
dnsdist
dolphin-emu
dovecot
dpkg
drbd-utils
efl
ell
elogind
emacs
finit
fluidsynth
freeradius
frr
fwupd
gamemode
getdns
gitlab
gnome-shell
gobgp
gunicorn
haproxy
hddemux
i3-wm
icinga2
inn
inn2
iwd
jackd2
kamailio
keepalived
knot
knot-resolver
knxd
lbcd
libconfig-model-systemd-perl
libinfinity
liblinux-systemd-perl
libosmocore
libpod
libreswan
libzorpll
lirc
lomiri
lomiri-session
lua-systemd
lvm2
manpages-l10n
mariadb
mosquitto
mpd
mpdscribble
multipath-tools
net-snmp
networkd-dispatcher
network-manager
nsca-ng
nsd
nut
nvme-stas
ofono
openbsd-inetd
openfortivpn
open-iscsi
openldap
opensearch
openvpn
osmo-bts
osmo-msc
owfs
pcp
pdns
pdns-recursor
pgagroal
pgbouncer
postgresql-16
ppp
psi-notify
pulseaudio
puma
pystemd
python-oslo.service
python-systemd
qt6-webengine
qtwebengine-opensource-src
rbldnsd
rdma-core
redis
refpolicy
remctl
rpcbind
rsyslog
rtkit
rtpengine
ruby-sd-notify
ruby-sidekiq
rust-libsystemd
rust-sd-notify
s6
samba
shibboleth-sp
smartmontools
smcroute
snapd
squid
sssd
strongswan
suricata
swupdate
syslog-ng
tgt
tor
transmission
triggerhappy
unbound
uptimed
vast
vdr
weston
w-scan-cpp
xen
xorg-server
xorp
xpra
xtide
xwayland
zeroc-ice

A good portion of these packages certainly does not need the full libsystemd.so.0 either.

[ericcurtin]

As it appears sshd is implementing sd_notify() on their own.

Indeed; a good move.

Why though?

As a convenience for the other packages, other than openssh, that need sd_notify:

389-ds-base
amavisd-new
apache2
apertium-apy
apt-cacher-ng
asterisk
autofs
avahi
biboumi
bind9
bluez
brltty
budgie-session
casync
chromium
conntrack-tools
corosync-qdevice
crun
cubemap
davmail
dbus
dbus-broker
dlm
dlt-daemon
dnsdist
dolphin-emu
dovecot
dpkg
drbd-utils
efl
ell
elogind
emacs
finit
fluidsynth
freeradius
frr
fwupd
gamemode
getdns
gitlab
gnome-shell
gobgp
gunicorn
haproxy
hddemux
i3-wm
icinga2
inn
inn2
iwd
jackd2
kamailio
keepalived
knot
knot-resolver
knxd
lbcd
libconfig-model-systemd-perl
libinfinity
liblinux-systemd-perl
libosmocore
libpod
libreswan
libzorpll
lirc
lomiri
lomiri-session
lua-systemd
lvm2
manpages-l10n
mariadb
mosquitto
mpd
mpdscribble
multipath-tools
net-snmp
networkd-dispatcher
network-manager
nsca-ng
nsd
nut
nvme-stas
ofono
openbsd-inetd
openfortivpn
open-iscsi
openldap
opensearch
openvpn
osmo-bts
osmo-msc
owfs
pcp
pdns
pdns-recursor
pgagroal
pgbouncer
postgresql-16
ppp
psi-notify
pulseaudio
puma
pystemd
python-oslo.service
python-systemd
qt6-webengine
qtwebengine-opensource-src
rbldnsd
rdma-core
redis
refpolicy
remctl
rpcbind
rsyslog
rtkit
rtpengine
ruby-sd-notify
ruby-sidekiq
rust-libsystemd
rust-sd-notify
s6
samba
shibboleth-sp
smartmontools
smcroute
snapd
squid
sssd
strongswan
suricata
swupdate
syslog-ng
tgt
tor
transmission
triggerhappy
unbound
uptimed
vast
vdr
weston
w-scan-cpp
xen
xorg-server
xorp
xpra
xtide
xwayland
zeroc-ice

A good portion of these packages certainly does not need the full libsystemd.so.0 either.

Another thing that could be useful is a systemd api that you can use as either an allowlist or blocklist for dlopen'ing libraries.

Because lets say we have all the systemd optional dependencies installed on a machine. But lets take binary sshd as an example where we need to be extra secure. We need/want to use libsystemd in sshd... But... We do not want to use liblzma in sshd because we do not need it or trust it, this could be useful:

if (use_liblzma)
   dlopen(liblzma);

[RussNelson]

libsystemd contains at least three APIs that are just the client-side for IPC:

1. sd-bus as API for D-Bus
2. sd_notify() as API for systemd's simple ready notification protocol
3. sd_journal_print() as API for structured logging to journald

Then they don't need to be part of libsystemd, do they? They each be in a separate library. Given the problem of "libsystemd links to many things", the solution seems to me that libsystemd solves too many problems, creating an insecurity.

OR, if that's simply unacceptable to you, then restructure libsystemd so that every function is in its own .o file, so that when someone links their program to libsystemd, ONLY the code they actually use becomes part of their executable. Static linking and small .o files are your friend.

[poettering]

Another thing that could be useful is a systemd api that you can use as either an allowlist or blocklist for dlopen'ing libraries.

Because lets say we have all the systemd optional dependencies installed on a machine. But lets take binary sshd as an example where we need to be extra secure. We need/want to use libsystemd in sshd... But... We do not want to use liblzma in sshd because we do not need it or trust it, this could be useful:

if (use_liblzma)
   dlopen(liblzma);

Such an API would certainly useful (though I don't see how it would it help in any way in the xzpocalipse). But I'd argue this should probably be prat of glibc/libd/dl.so. i.e. another LD_XYZ env var or so.

[poettering]

Then they don't need to be part of libsystemd, do they? They each be in a separate library. Given the problem of "libsystemd links to many things", the solution seems to me that libsystemd solves too many problems, creating an insecurity.

I don't think oyu know what you are talking of. there's not that much left if you remove the IPC clients.

Also, please read up, let's not repeat this discussion over and over again.

OR, if that's simply unacceptable to you, then restructure libsystemd so that every function is in its own .o file, so that when someone links their program to libsystemd, ONLY the code they actually use becomes part of their executable. Static linking and small .o files are your friend.

Yeah, but no. Sorry.

Please do not add further noise to this thread. Unless you are very sure that what you are saying hasn't been said before here, don't write it. I'll start to aggressively hide further comments to keep the noise down if they just reiterate the same things over and over again.

So, for anyone else: if you want to suggest splitting up the lib, just shut up, please. It has been proposed before, and has been rejected before. Go and read up.

[ericcurtin]

Another thing that could be useful is a systemd api that you can use as either an allowlist or blocklist for dlopen'ing libraries.
Because lets say we have all the systemd optional dependencies installed on a machine. But lets take binary sshd as an example where we need to be extra secure. We need/want to use libsystemd in sshd... But... We do not want to use liblzma in sshd because we do not need it or trust it, this could be useful:

if (use_liblzma)
   dlopen(liblzma);

Such an API would certainly useful (though I don't see how it would it help in any way in the xzpocalipse). But I'd argue this should probably be prat of glibc/libd/dl.so. i.e. another LD_XYZ env var or so.

Well lets say sshd or a similar project wants to use libsystemd but not liblzma, but they expect in many cases liblzma will be installed on the OS regardless. It closes up an attack vector for sshd in this case.

[poettering]

@bluca why did you reopen? is there anything left to address?

[bluca]

The gcrypt dlopenify PR, will work on that next

[bluca]

I have regenerated the docs so the self-contained example is published at: https://www.freedesktop.org/software/systemd/man/devel/sd_notify.html#Notes

[topimiettinen]

Regarding the original issue, "Reduce dependencies": downstream distros have the full capability to reduce the dependencies they don't trust, with meson options such as -Dxz=false. I'd even argue that that kind of QA is one of their major responsibilities. They also make choices whether to package the dependencies at all.

Of course this all doesn't mean that upstreams get to be careless when choosing to integrate with 3rd party components. There are also non-optional dependencies like glibc/ld.so (which BTW introduced the GNU IFUNC feature used in the exploit).

[bluca]

I have regenerated the docs so the self-contained example is published at: https://www.freedesktop.org/software/systemd/man/devel/sd_notify.html#Notes

This is now also linked from https://systemd.io/PORTABILITY_AND_STABILITY/

[bluca]

Then they don't need to be part of libsystemd, do they? They each be in a separate library. Given the problem of "libsystemd links to many things", the solution seems to me that libsystemd solves too many problems, creating an insecurity.

I don't think oyu know what you are talking of. there's not that much left if you remove the IPC clients.

Also, please read up, let's not repeat this discussion over and over again.

OR, if that's simply unacceptable to you, then restructure libsystemd so that every function is in its own .o file, so that when someone links their program to libsystemd, ONLY the code they actually use becomes part of their executable. Static linking and small .o files are your friend.

Yeah, but no. Sorry.

Please do not add further noise to this thread. Unless you are very sure that what you are saying hasn't been said before here, don't write it. I'll start to aggressively hide further comments to keep the noise down if they just reiterate the same things over and over again.

So, for anyone else: if you want to suggest splitting up the lib, just shut up, please. It has been proposed before, and has been rejected before. Go and read up.

I guess this was linked on some social media - time for lock&roll