Decreasing the final image sizes

[coffee-cup]

The final image sizes produced by Nixpacks are unfortunately fairly large. Especially when you look at images for compiled languages such as Rust (around ~3GB 😮). For these languages, we can ideally build a static binary that does not need to link to any system libraries such as glibc. The go provider by default compiles with cgo disabled which produces a small binary that we copy to a fresh debian slim image. The resulting image size is ~85MB compared to the ~1.4GB without copying to slim image.

On a platform like Railway, the various docker layers can be cached which can vastly improve push time. But these large images are far from ideal and we should find a way to decrease them.

The vast majority of the image size comes from deps in the /nix/store. For example

Not many of these these deps are needed at runtime, but some are. We can see which files are necessary by looking at the shared object deps.

For rust

If the Nix store is not present, we see that GLIBC is not found

and when trying to run the binary, we can't

How can we improve?

There are a few possibilities for how we can fix this. Many compiled languages have a way to create a static binary by linking with musl instead of glibc. However, this isn't always that straight forward and often requires installing a bunch of system libraries (e.g. the crystal dockerfile https://github.com/crystal-lang/distribution-scripts/blob/master/docker/alpine.Dockerfile)

We could manually edit the interpreter, but that seems like it will cause many issues.

We could remove all packages in the Nix store besides the one that are referenced by ldd. But that feels wrong and error prone.

There are also a few options that move us more into Nix land. There is an experimental nix bundle command that can "bundle an application so that it works outside of the Nix store". For that we would have to auto gen a derivation for the app so we can bundle it. I've never used this before so not sure how effective it will be.

We could build the images with dockerTools.buildImage as opposed to generating a Dockerfile that we then build with Docker. This can result in smaller images but may cause issues with non-Nix deps (sadly some exist) or building languages that step outside of the pure Nix way of doing things.

Final thoughts

The solution implemented must work for ~100% of compiled languages that can produce a single binary. Our solution can't be language aware.

I will keep exploring solutions, but am opening this up for feedback/suggestions.

Edit

Upon further digging I have found a decent way to build rust projects with musl using the Nix overlay. PR up soon.

[flybayer]

@coffee-cup would integrating https://github.com/docker-slim/docker-slim be an option? It seems to massively reduce image size. I haven't tried it, but it's been on my radar for awhile.

[Milo123459]

I think that's a good idea - we could use it on the base image.

[coffee-cup]

Yeah that is a good idea @flybayer. We can look into integrating it with the base image as well as each individually created image. I've never used docker-slim either but can take a look this week.

[coffee-cup]

Just wanted to update you that I looked into docker-slim briefly and it is fairly involved as it will analyze the container as it is running to determine what can be removed in the final image. This means it can't be used on just the Nixpacks base image, but we could use it on the generated images. I am unsure if it should be apart of Nixpacks core as opposed to something that is controlled by the Nixpacks operator (e.g. Railway, CLI user). Thoughts?

I also think there are a other things we can do to decrease the image size (such as removing unused nix/system files). At the moment the Railway team is focused on a few other Nixpacks features and will take prioritize the image size afterward. PR or ideas related to this are also welcome though 😄

[flybayer]

I am unsure if it should be apart of Nixpacks core as opposed to something that is controlled by the Nixpacks operator

My feeling is that if it's a robust solution, it should be part of nixpacks so everyone benefits without having to do the complex work to integrate it themselves. Especially since it's not just perf at play but it also reduces the security footprint.

[dzmitry-lahoda]

image which mounts host /nix/store could reduce OCI size. it will be OCI compliant BTW.

interesting option to hook nix so that docker can build into host nix store (e.g. linux docker on mac darwin machine), so that OCI image just link to nix hook.