Dynamically load compression libraries

[teknoraver]

Dynamically load compression libraries (LZ4, ZSTD, LZMA) so we can reduce the size of the initram images by omitting libraries which aren't really used.

[YHNdnzj]

I'd really like to know if we can replace all these with libarchive first (see also #31131 (comment))

[poettering]

I'd really like to know if we can replace all these with libarchive first (see also #31131 (comment))

hmm, so we link against these libraries in journald for example to compress individual fields, does it really make sense to involve libarchive in that?

does libarchive support uncompressing blobs that aren't tarballs/zipfiles?

i kinda like the fact that if we make the individual compression libs dlopen-deps we truly minimize the dep tree, because people can pick one compressor only. After all libarchive itself does not employ dlopen() afaics.

I mean:

 lddtree /usr/lib64/libarchive.so
libarchive.so => /usr/lib64/libarchive.so (interpreter => none)
    libcrypto.so.3 => /lib64/libcrypto.so.3
    libacl.so.1 => /lib64/libacl.so.1
        libattr.so.1 => /lib64/libattr.so.1
    liblzma.so.5 => /lib64/liblzma.so.5
    libzstd.so.1 => /lib64/libzstd.so.1
    liblz4.so.1 => /lib64/liblz4.so.1
    libbz2.so.1 => /lib64/libbz2.so.1
    libz.so.1 => /lib64/libz.so.1
    libxml2.so.2 => /lib64/libxml2.so.2
        libm.so.6 => /lib64/libm.so.6
            ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2
    libc.so.6 => /lib64/libc.so.6

Hence I am very much inclined to merging this PR.

In particular libbz2 I think is something we should try to get out of the deptree. By using libarchive for everyting we'd still link to it indirectly.

[poettering]

lgtm, but some issues, see above.

And I guess the CIs need some updating before we can merge this, because the deps are now implicit rather than explicit.

@mrc0mmand any chance you can look into that?

[poettering]

Please also add an extra commit that deletes/updates the relevant lines in the TODO file in the toplevel dir. the one containing "bzip2, xz, lz4"

[YHNdnzj]

does libarchive support uncompressing blobs that aren't tarballs/zipfiles?

Yes, see archive_write_set_format_raw.

i kinda like the fact that if we make the individual compression libs dlopen-deps we truly minimize the dep tree, because people can pick one compressor only. After all libarchive itself does not employ dlopen() afaics.

Indeed. However, as discussed in the other PR, libkmod depends on some of those libs anyway too. And bzip2 is required by things like gnupg/ffmpeg/...

While it would be great if bzip2 can be eliminated from the whole system, that won't happen any time soon. So instead of adding more dlopen()-ed deps, which are hard to manage, it seems more practical to me to reduce the number of direct deps first.

[YHNdnzj]

IOW, the reduced dependency tree isn't really observable by users, in which case adding more code and introducing more packaging/CI changes really sounds like a pure burden.

[poettering]

Indeed. However, as discussed in the other PR, libkmod depends on some of those libs anyway too. And bzip2 is required by things like gnupg/ffmpeg/...

well, the focus here is really on building tiny systems for initrds for example. I don't see why you'd need gunpg/ffmpeg/… in an initrd...

i mean, sure, on a full blown desktop they will end up int he full install, but even then, i see no reason they need to be in the initrd.

While it would be great if bzip2 can be eliminated from the whole system, that won't happen any time soon. So instead of adding more dlopen()-ed deps, which are hard to manage, it seems more practical to me to reduce the number of direct deps first.

but libarchive itself doesn#t need to be in the initrd, no?

[teknoraver]

Should I call dlopen_* functions also in decompress_startswith_* functions?

This could be the cause of systemd:libsystemd/test-journal-enum failing since they're are referenced in src/libsystemd/sd-journal/journal-file.c

[daandemeyer]

test-dlopen-so.c should be modified as well to test the new dlopen functions

[aafeijoo-suse]

Sorry for interrupting your work :-p

No problem at all! Thanks for fixing it so quickly.

[nullbio]

Thanks for saving the world @teknoraver - not sure if you've seen the latest Veritasium video. :-)

[bmkmanoj]

Thanks for saving the world @teknoraver - not sure if you've seen the latest Veritasium video. :-)

This is the video - The Internet Was Weeks Away From Disaster and No One Knew :)

[miladsoltany]

Jesus Christ! I just saw the video. Thank you @teknoraver.

[vinciest]

At first people thought it was just a normal PR.
Thank you! Matteo
Thank you! Andres
Salute from Veritasium's video! o7