Dynamically load compression libraries

[teknoraver]

Dynamically load compression libraries (LZ4, ZSTD, LZMA) so we can reduce the size of the initram images by omitting libraries which aren't really used.

[YHNdnzj]

I'd really like to know if we can replace all these with libarchive first (see also #31131 (comment))

[poettering]

I'd really like to know if we can replace all these with libarchive first (see also #31131 (comment))

hmm, so we link against these libraries in journald for example to compress individual fields, does it really make sense to involve libarchive in that?

does libarchive support uncompressing blobs that aren't tarballs/zipfiles?

i kinda like the fact that if we make the individual compression libs dlopen-deps we truly minimize the dep tree, because people can pick one compressor only. After all libarchive itself does not employ dlopen() afaics.

I mean:

 lddtree /usr/lib64/libarchive.so
libarchive.so => /usr/lib64/libarchive.so (interpreter => none)
    libcrypto.so.3 => /lib64/libcrypto.so.3
    libacl.so.1 => /lib64/libacl.so.1
        libattr.so.1 => /lib64/libattr.so.1
    liblzma.so.5 => /lib64/liblzma.so.5
    libzstd.so.1 => /lib64/libzstd.so.1
    liblz4.so.1 => /lib64/liblz4.so.1
    libbz2.so.1 => /lib64/libbz2.so.1
    libz.so.1 => /lib64/libz.so.1
    libxml2.so.2 => /lib64/libxml2.so.2
        libm.so.6 => /lib64/libm.so.6
            ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2
    libc.so.6 => /lib64/libc.so.6

Hence I am very much inclined to merging this PR.

In particular libbz2 I think is something we should try to get out of the deptree. By using libarchive for everyting we'd still link to it indirectly.

[poettering]

lgtm, but some issues, see above.

And I guess the CIs need some updating before we can merge this, because the deps are now implicit rather than explicit.

@mrc0mmand any chance you can look into that?

[poettering]

Please also add an extra commit that deletes/updates the relevant lines in the TODO file in the toplevel dir. the one containing "bzip2, xz, lz4"

[YHNdnzj]

does libarchive support uncompressing blobs that aren't tarballs/zipfiles?

Yes, see archive_write_set_format_raw.

i kinda like the fact that if we make the individual compression libs dlopen-deps we truly minimize the dep tree, because people can pick one compressor only. After all libarchive itself does not employ dlopen() afaics.

Indeed. However, as discussed in the other PR, libkmod depends on some of those libs anyway too. And bzip2 is required by things like gnupg/ffmpeg/...

While it would be great if bzip2 can be eliminated from the whole system, that won't happen any time soon. So instead of adding more dlopen()-ed deps, which are hard to manage, it seems more practical to me to reduce the number of direct deps first.

[YHNdnzj]

IOW, the reduced dependency tree isn't really observable by users, in which case adding more code and introducing more packaging/CI changes really sounds like a pure burden.

[poettering]

Indeed. However, as discussed in the other PR, libkmod depends on some of those libs anyway too. And bzip2 is required by things like gnupg/ffmpeg/...

well, the focus here is really on building tiny systems for initrds for example. I don't see why you'd need gunpg/ffmpeg/… in an initrd...

i mean, sure, on a full blown desktop they will end up int he full install, but even then, i see no reason they need to be in the initrd.

While it would be great if bzip2 can be eliminated from the whole system, that won't happen any time soon. So instead of adding more dlopen()-ed deps, which are hard to manage, it seems more practical to me to reduce the number of direct deps first.

but libarchive itself doesn#t need to be in the initrd, no?

[teknoraver]

Should I call dlopen_* functions also in decompress_startswith_* functions?

This could be the cause of systemd:libsystemd/test-journal-enum failing since they're are referenced in src/libsystemd/sd-journal/journal-file.c

[DaanDeMeyer]

test-dlopen-so.c should be modified as well to test the new dlopen functions

[eworm-de]

Wondering if we need changes for the initrd generation now... Are these libraries required at early boot?

[poettering]

Wondering if we need changes for the initrd generation now... Are these libraries required at early boot?

I think you might get away without them for now.

[aafeijoo-suse]

Is this failing only for me? shouldn't libdl be added to the libbasic deps?

[495/2818] Linking target libsystemd.so.0.38.0
FAILED: libsystemd.so.0.38.0 
cc  -o libsystemd.so.0.38.0  -Wl,--as-needed -Wl,--no-undefined -shared -fPIC -Wl,--start-group -Wl,-soname,libsystemd.so.0 -Wl,--whole-archive src/libsystemd/libsystemd_static.a -Wl,--no-whole-archive -fstack-protector src/basic/libbasic.a src/basic/libbasic-gcrypt.a -shared -Wl,--version-script=/mnt/work/src/systemd/upstream-fork/main/src/libsystemd/libsystemd.sym -lrt -pthread -Wl,--fatal-warnings -Wl,-z,now -Wl,-z,relro -Wl,--warn-common /usr/lib64/libcap.so -lm -Wl,--end-group -Wl,--fatal-warnings -Wl,-z,now -Wl,-z,relro -Wl,--warn-common -Wl,--fatal-warnings -Wl,-z,now -Wl,-z,relro -Wl,--warn-common -Wl,--fatal-warnings -Wl,-z,now -Wl,-z,relro -Wl,--warn-common
/usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld: src/basic/libbasic.a.p/dlfcn-util.c.o: in function `safe_dlclose':
/mnt/work/src/systemd/upstream-fork/main/build/../src/basic/dlfcn-util.h:12:(.text+0x22): undefined reference to `dlclose'
/usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld: src/basic/libbasic.a.p/dlfcn-util.c.o: in function `dlsym_many_or_warnv':
/mnt/work/src/systemd/upstream-fork/main/build/../src/basic/dlfcn-util.c:18:(.text+0xf1): undefined reference to `dlsym'
/usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld: /mnt/work/src/systemd/upstream-fork/main/build/../src/basic/dlfcn-util.c:20:(.text+0x15e): undefined reference to `dlerror'
/usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld: src/basic/libbasic.a.p/dlfcn-util.c.o: in function `dlopen_many_sym_or_warn_sentinel':
/mnt/work/src/systemd/upstream-fork/main/build/../src/basic/dlfcn-util.c:47:(.text+0x3a4): undefined reference to `dlopen'
/usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld: /mnt/work/src/systemd/upstream-fork/main/build/../src/basic/dlfcn-util.c:49:(.text+0x42c): undefined reference to `dlerror'
collect2: error: ld returned 1 exit status

[yuwata]

@aafeijoo-suse Maybe #31652 fixes your issue?

[aafeijoo-suse]

@aafeijoo-suse Maybe #31652 fixes your issue?

Yes, definitely. I had already tried it :)

[yuwata]

@aafeijoo-suse Maybe #31652 fixes your issue?

Yes, definitely. I had already tried it :)

Sorry for interrupting your work :-p

[aafeijoo-suse]

Sorry for interrupting your work :-p

No problem at all! Thanks for fixing it so quickly.