Investigate docker image size reduction #1428
Comments
|
my 2c:
|
|
as @om26er mentions, we need to make sure it works not only for crossbar, but also for crossbarfx, and for all supported architectures. I have spent countless hours in fiddling around with this to make it work (see ) unless and until s.o. comes up with a PR that provably works for all of above, we won't merge, and I won't spend more time on it. sorry also, I never actually understood why reducing the size is important. disk is cheap, who cares. fwiw, we ourself are using snaps and single-file executables https://download.crossbario.com/?prefix=crossbarfx/linux-amd64/ a lot with the latter, crossbar with all dependencies boils down to 40MB. actually, crossbarfx has a lot more than crossbar (eg the 40MB include a complete embedded numpy etc). and works on anything Linux 2.6+, any distro, no docker needed. |
|
Interesting input about snaps. In my fishbowl everything goes docker now :) I can provide my reasons for reducing image sizes for our app:
|
|
Thanks for laying out the reasons why image size matters to you! In absolute terms, and discounting other aspects, the smallest possible image size we'll be able to bake is likely using the EXE (40MB) and a scratch container ( The only downside: the EXE is baked on top of CPython, not PyPy. We haven't managed to bake the EXE on top of PyPy, and this is likely a non-trivial research like endeavor rgd snaps "vs" docker: yeah, docker rules for anything cloud, but for devices (IoT, edge, desktop, etc), snaps bring a couple of very substantial advantages compared to docker. in my view. anyways, surely we want to support both systems in the best possible way. rgd our single-file EXE approach. check this out (this should work on any linux 2.6+ .. regardless of distro or whatever): |
|
Great. Managed to run it on my Ubuntu 16.04. Generally, 40MB + 5MB for alpine is much better than ~430Mb for today's crossbar. I agree that it's too bad to loose PyPy. What tech do you use for baking the executable? |
|
For the EXE, we are using pyinstaller. A Docker image from that can be done it exactly 40MB, no need for Alpine. The base image is |
|
Sorry, I initially confused Well, your binary still needs libz and libc - it's not statically linked. Using just Results in: Even gcr.io/distroless/base (which is 3 times bigger than alpine) is not enough because of libz. Since this particular binary was compiled against glibc it does not work on alpine as well. If rebuilt on alpine the binary itself would've been smaller. Or, if bundled with libc/z, |
|
try this: |
|
rgd "bundling" libc: not possible. the C runtime library (and the shared object loader) are the actual interface to the kernel, and cannot be statically linked (as far as I know / am aware) anyways. it works: starting the node: |
|
Well, you are cheating :)
A glimpse of my past memories relates to what you say, but I can't remember details and looks like at least with "Hello World" it works fine (see below). Also have a look at https://github.com/GoogleContainerTools/distroless/blob/master/base/README.md - |
currently yes, but we could build on an old centos, and make any kernel since 2.6.23 work from a single EXE. the linux kernel api is supposed to allow that (https://liquidat.wordpress.com/2007/07/21/linux-kernel-2623-to-have-stable-userspace-driver-api/) - though I haven't tested that. |
|
Yes, kernel API is stable, but you are relying on glibc API, namely requiring host to have glibc (not just libc, bug GNU libc). Mounting /lib and /tmp into container will surely raise eyebrows. I'm not a security expert, but you need to be very careful with it - by default container processes run as root, hence by default crossbar will be able to modify system libraries and have a root access to /tmp directory. Anyhow, going static looks easy - there is staticx project that does exactly that - stati'fies existing binaries. PyInstaller docs refer to it. I've tried it on crossbar and after patching staticx as suggested it worked (the resulting library is only 600k bigger than the dynamic one): I've used alpine as a base image because crossbar seems to require some temp directories. Once you have them, I think |
|
@haizaar ok, yes, you are right rgd kernel API and tmp. I agree. (note: this thread now has gotten a bit OT, as we definitely do want great docker images building on upstream pypy image eg - so we will follow up on this as well). what you found with staticx: awesome! thank you very much for pointing me there=) I can confirm, this works. It increases the startup time (which doesn't matter much, because crossbar is a long running server thingy anyways), and the size a bit (expected). That's fine. means: we will add that staticx libc/libdl bundling as a build step. (follow up issue on private repo https://github.com/crossbario/crossbarfx/issues/176) thanks again, cool;) |
fwiw, the need for |
|
You are welcome. P.S. Is there a chance someone can follow up on Autobahn issue? - https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/autobahnws/4vnuE8Q7yZ8/Xnb532ykBQAJ? (Sorry for abusing the discussion :) |
|
sure! I replied on the list. let me know if that helps, I'll answer |
|
ok, I spent a couple of hours fiddling around and testing our current EXE (everything static but libc/libdl) and fully static EXE (using staticx). turns out as this: summary: we will keep our current build approach for the EXE, and not statically link libc/libdl the issues with the 100% static EXE are all similar to this: after compiling 3 versions of glibc from source and trying I gave up on CentOS 6 (kernel 2.6.32 and oldish glibc) |
|
I guess that's some glitch between PyInstaller and staticx. I can only suggest opening a ticket in staticx - the maintainer seems to be interested to get these issues solved. You can still use staticx just for building small docker images as I tried earlier - results in 47Mb Docker image Out of curiosity I built alpine based docker image of crossbar (not fx obviously) using this Dockerfile - 104Mb |
|
crossbar docker is ~500MB https://hub.docker.com/r/crossbario/crossbar/tags |
The last time we tried that, we had to revert it because it broke crossbarfx images. This time lets be better prepared and make sure to test everything and do the required changes.
The text was updated successfully, but these errors were encountered: