Debian Bullseye, fluent-bit 1.9 support, shrink images

[ReillyBrogan]

• Update base image to Debian Bullseye (including distroless image)
• fluent-bit 1.9.x support (which adds yaml config file support and thus needs libyaml)

Reduced image size
I noticed that the old copy commands were very inefficient. For instance a Bullseye system will have the below setup for libgnutls:

lrwxrwxrwx  1 root root   20 Jun 14 16:55 libgnutls.so.30 -> libgnutls.so.30.29.1
-rw-r--r--  1 root root 2.0M Jun 14 16:55 libgnutls.so.30.29.1

Note that libgnutls.so.30 is a symlink to libgnutls.so.30.29.1 (for information on why this is the case lookup "shared library soname"). However, the previous copy commands would create the following:

-rw-r--r--  1 root root 2.0M Jun 14 16:55 libgnutls.so.30
-rw-r--r--  1 root root 2.0M Jun 14 16:55 libgnutls.so.30.29.1

This is because docker copy does not preserve that the file is a symlink and instead copies the file that the symlink is pointing to. This creates duplicate copies of many libs (some libraries have two separate symlinks, so three copies of the same library), significantly bloating image size.

In addition copy commands like this COPY --from=builder /usr/lib/x86_64-linux-gnu/libssl.so* /usr/lib/x86_64-linux-gnu/ would very frequently copy static libraries (.a files) into the image. These are completely unnecessary at runtime (they would only ever be used during compile-time if we were creating a statically linked binary) and often larger than the .so files themselves.

So to solve both of these issues I went through with ld and identified every .so that the fluent-bit binary was actually (recursively) linked to, and just copied those. To avoid having to do any LD_PRELOAD fuckery I checked for which file the binary was actually linked against (which was always the soname file), and just copied that file across (abusing the "docker copy dereferences symlinks" behavior that was previously mentioned).

Note that the format of the copy command is now very specific. You'd think that this would work:

 COPY --from=builder /usr/lib/x86_64-linux-gnu/libgnutls.so.30 /usr/lib/x86_64-linux-gnu/

But instead it creates the following file:

-rw-r--r--  1 root root 2.0M Jun 14 16:55 libgnutls.so.30.29.1

So it's dereferencing the symlink but also carrying over the filename that the symlink points to?!?!? Very... interesting.... behavior.

So instead we need to use this form:

 COPY --from=builder /usr/lib/x86_64-linux-gnu/libgnutls.so.30 /usr/lib/x86_64-linux-gnu/libgnutls.so.30

Which creates the expected output:

-rw-r--r--  1 root root 2.0M Jun 14 16:55 libgnutls.so.30

In addition, I removed the copy commands for libssl and libcrypto, as these copied shared libs from openssl which were already part of the distroless base image (the "old" copies of them stuck around in the previous layer of the docker image). I also removed libselinux and libpcre because neither showed up in the output of ld and they are likely unnecessary (however they could be used via dlopen on certain systems, if they are needed they can be re-added). I added libzstd which is a new dependency of libsystemd in Bullseye.

To verify that the shared library changes here work you can build the image and then run the following (note that this can be run in the debug image or the normal image, no shell is required for this):

 docker run --rm -it $IMAGE_TAG /lib/x86_64-linux-gnu/ld-2.31.so --list /fluent-bit/bin/fluent-bit

Which will produce the following output:

        linux-vdso.so.1 (0x00007ffe3bef3000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fccebbf3000)
        libyaml-0.so.2 => /usr/lib/x86_64-linux-gnu/libyaml-0.so.2 (0x00007fccebbd1000)
        libsystemd.so.0 => /usr/lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007fccebb1c000)
        libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fcceba89000)
        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fcceb795000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fcceb778000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fcceb759000)
        libpq.so.5 => /usr/lib/x86_64-linux-gnu/libpq.so.5 (0x00007fcceb706000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fcceb5c2000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fcceb5bc000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fcceb5a2000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcceb3dd000)
        /lib64/ld-linux-x86-64.so.2 => /lib/x86_64-linux-gnu/ld-2.31.so (0x00007fccec6e8000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fcceb3d0000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fcceb3a8000)
        libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007fcceb2cd000)
        liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x00007fcceb2aa000)
        libgcrypt.so.20 => /usr/lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007fcceb18a000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fcceb137000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007fcceb0df000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fcceb0b9000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fcceafdf000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fcceafaf000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fcceafa9000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fcceaf98000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007fcceaf87000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fcceaf6d000)
        libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fccead6d000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fccead66000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fcceac30000)
        libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007fcceac0f000)
        libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007fcceaa8d000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fcceaa77000)
        libnettle.so.8 => /usr/lib/x86_64-linux-gnu/libnettle.so.8 (0x00007fcceaa2f000)
        libhogweed.so.6 => /usr/lib/x86_64-linux-gnu/libhogweed.so.6 (0x00007fccea9e6000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fccea963000)
        libffi.so.7 => /usr/lib/x86_64-linux-gnu/libffi.so.7 (0x00007fccea957000)

This indicates that the dynamic loader was able to fully resolve all linked libraries. You would receive an error message and a non-0 output code if this wasn't true.

The image size after making all of these changes goes from 133MB to 62MB, a 54% reduction in image size.