distroless image not really distroless? Not running under alpine..

[narcoticfresh]

hi guys

my struggle is with dockerizing envoy. specifially, i have a component that needs an alpine base image.

i am aware that there was envoy-alpine builds, but those are deprecated in favor of the distroless images (as mentioned in #21758). but it seems they are not really distroless, as they are not able to run inside an alpine container.

first, we start with a simple dockerfile:

FROM alpine:3

COPY --from=envoyproxy/envoy:distroless-v1.30.3 /usr/local/bin/envoy /usr/local/bin/envoy

RUN chmod +x /usr/local/bin/envoy

let's build and try to use it

$ docker build -t alp -f Dockerfile-alpine .
$ docker run --rm -ti alp /usr/local/bin/envoy --help
exec /usr/local/bin/envoy: no such file or directory

hm, let's see ldd output:

$ docker run --rm -ti alp ldd /usr/local/bin/envoy   
	/lib64/ld-linux-x86-64.so.2 (0x7ff9fd73f000)
	libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7ff9fd73f000)
	librt.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7ff9fd73f000)
	libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7ff9fd73f000)
	libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7ff9fd73f000)
	libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7ff9fd73f000)
Error loading shared library 
: No such file or directory (needed by /usr/local/bin/envoy)
Error relocating /usr/local/bin/envoy: __memcpy_chk: symbol not found
Error relocating /usr/local/bin/envoy: __memset_chk: symbol not found
Error relocating /usr/local/bin/envoy: __strcpy_chk: symbol not found
Error relocating /usr/local/bin/envoy: __strncpy_chk: symbol not found
Error relocating /usr/local/bin/envoy: __fdelt_chk: symbol not found
Error relocating /usr/local/bin/envoy: fcntl64: symbol not found
Error relocating /usr/local/bin/envoy: pthread_cond_clockwait: symbol not found
Error relocating /usr/local/bin/envoy: __memmove_chk: symbol not found

ok - let's try to add some musl/glibc/compat things as suggested over the internet

apk add --no-cache libc6-compat gcompat musl-dev

build and run again

$ docker run --rm -ti alp ldd /usr/local/bin/envoy 
	/lib64/ld-linux-x86-64.so.2 (0x7f70eed88000)
	libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f70eed88000)
	librt.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f70eed88000)
	libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f70eed88000)
	libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f70eed88000)
	libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f70eed88000)
	ld-linux-x86-64.so.2 => /lib/ld-linux-x86-64.so.2 (0x7f70e9e99000)
Error relocating /usr/local/bin/envoy: __memcpy_chk: symbol not found
Error relocating /usr/local/bin/envoy: __memset_chk: symbol not found
Error relocating /usr/local/bin/envoy: __strcpy_chk: symbol not found
Error relocating /usr/local/bin/envoy: __strncpy_chk: symbol not found
Error relocating /usr/local/bin/envoy: __fdelt_chk: symbol not found
Error relocating /usr/local/bin/envoy: fcntl64: symbol not found
Error relocating /usr/local/bin/envoy: pthread_cond_clockwait: symbol not found
Error relocating /usr/local/bin/envoy: __memmove_chk: symbol not found
Error relocating /usr/local/bin/envoy: __vsnprintf_chk: symbol not found
Error relocating /usr/local/bin/envoy: backtrace: symbol not found
Error relocating /usr/local/bin/envoy: backtrace_symbols: symbol not found
Error relocating /usr/local/bin/envoy: __longjmp_chk: symbol not found
Error relocating /usr/local/bin/envoy: strtoll_l: symbol not found
Error relocating /usr/local/bin/envoy: strtoull_l: symbol not found
Error relocating /usr/local/bin/envoy: __cxa_thread_atexit_impl: symbol not found
Error relocating /usr/local/bin/envoy: __register_atfork: symbol not found
Error relocating /usr/local/bin/envoy: __libc_stack_end: symbol not found

now it seems all *.so files are found but we still have those symbol not found errors.

---

Assumption?

The name distroless suggests that it is truly a portable, statically linked binary. Many (especially golang) projects provide similar binaries which run on alpine without any issues. Is this assumption incorrect? Shouldn't it also run under alpine?

Thanks so much for all your efforts, love the project!

[htuch]

@phlax

[phlax]

it seems they are not really distroless ... truly a portable, statically linked binary

it really is - that is exactly what it provides - glibc and nothing else - we (envoy) add a static binary that uses just that

if the expectation is for some kind of universal distro then i think there is some misunderstanding - not sure that it would be even possible

our binary is not universally portable - it requires a minimum glibc version - but its pretty portable - ie to any environment that meets the glibc environment

iirc alpine uses musl - im not totally clear on the compatibility there - but in the main i believe it should work

i think this question is not really about whether the container is "distroless" its more about binary compatibility with musl

i think we dont want to bring back the alpine distro - we had it mostly to serve the purpose that the distroless container serves far far better (ie minimum included environment, smaller and more lightweight container image)

this seems more like an issue with musl/alpine glibc compatibility tbh - i would raise the issue there

[narcoticfresh]

@phlax
hm, yes, i see.. "distroless" is maybe a misnomer in this regard. it is also not clear from an image consumer side, what version requirements there are (which glibc version for example)

i also don't know how musl tracks its compatibility to glibc, i guess there must some kind of API versioning to ensure it.

do you see any chance that envoy could produce a "musl" binary? or is it simply the case that musl does not provide all the features needed?

---> i'm just baffled that not even a year ago there was a "alpine distro" build and now it seems to be fully incompatible?

[phlax]

the (upstream) container/image is designed to be distro agnostic - i think the name is correct - although i guess its not entirely distro agnostic - which in docker terms would just be the scratch image

i also dont know exactly how musl tracks/maps to glibc - im aware there is some compatibility, but we had issues previously which was one of our reasons for dropping support - the main reason being that the distroless image provides a better base for what we were using it for.

im not 100% about which glibc version we require - but i think when i looked at this previously i came to the conclusion that is set here in our repo:

distribution/debian/packages.bzl:GLIBC_MIN_VERSION = "2.27"

alpine has not been supported for ~2 years - i dont rem exactly the state when we dropped support, but i would be surprised if it cant be made to work if that is a hard requirement. We are unlikely to bring back support for it as we now provide the distroless which, as mentioned above, better serves our needs and requires much less maintenance

[narcoticfresh]

closing as documented, thanks @phlax