Docker Images got Big

[daniel-j-h]

Looks like our release images went up from ~10 MB to ~30 MB. Looks suspicious.

Could be related to #3850 - don't know why yet. cc @danpat

[danpat]

The latest images are being built with alpine:3.5 as the base, but the v5.6.x builds are all coming from https://github.com/Project-OSRM/osrm-backend-docker/blob/master/Dockerfile, which is based on alpine:3.5.

I can't imagine that Alpine added 20MB to their base images, but the upgrade may change how our code compiles, or what dependencies we add during the build, and we might not be cleaning everything up.

[daniel-j-h]

(You probably wanted to say alpine 3.5 vs 3.4 ^)

I checked that, but even the osrm-backend-ci images we built for testing purpose were already using alpine:3.5 and they are only 13 MB

https://hub.docker.com/r/osrm/osrm-backend-ci/tags/

I changed the Alpine version to 3.5 when integrating the Dockerfile into the osrm-backend repo. - nothing else changed, except the switch from Docker Hub to Docker Cloud.

[danpat]

Ah - I bet it's the COPY command we use in the main package now - previously, the Dockerfile cloned the repo inside the image, then cleaned up.

By using COPY, the source code is added as a layer - even though we have the rm -rf /src cleanup, the data is still there in the image. Not sure if there's a good way to fix this, I'll need to do some reading.

[danpat]

Ref: moby/moby#12169

[daniel-j-h]

But the osrm-backend-ci Dockerfile did exactly the same, check the PR

https://github.com/Project-OSRM/osrm-backend/pull/3804/files#diff-3254677a7917c6c01f55212f86c57fbfR4

[danpat]

Hmm. We've unlinked the osrm-backend-ci dockerhub repo, so I'm not 100% sure what Dockerfile it was using. From memory you're correct, but the file sizes seem odd given all the other evidence. Maybe osrm-backend-ci wasn't building what we thought it was?

If I build locally, I get about a 50MB image from the osrm-backend/master/docker/Dockerfile, similar to what we're auto-publishing.

I ran the images through https://github.com/goldmann/docker-squash and they shrink back down to:

osrm/osrm-backend   squashed            216f82f1588c        7 seconds ago       21.6 MB
osrm/osrm-backend   master              cdee0af2fcf2        10 minutes ago      54.6 MB

This is a hint that my COPY hunch is right - docker-squash removes data from layers that's not visible in the final image - the COPY . /src data (about 35MB of it) fits this description....

I'm not sure what the next action here is - AFAIK there's no easy way to run something like docker-squash as part of the build process, and if using COPY is the root cause of the bloat, there doesn't seem to be a built-in way to remove COPY-ed layers from the final image.

[daniel-j-h]

We're up to almost 200 MB in the latest release image https://hub.docker.com/r/osrm/osrm-backend/tags/

[xydinesh]

@daniel-j-h Is there way to bring down the size ? v5.7.4 was only ~52MB. What changed ?

[daniel-j-h]

Check two commits above. Try building from the Dockerfile we provide locally.

What changed is we're using Docker Cloud to automatically build docker images by now.

And Docker Cloud does not squash:

https://forums.docker.com/t/automatic-squash-for-docker-hub-builds/4148

[xydinesh]

Thanks @daniel-j-h. I will give it a try.

[daniel-j-h]

In #4648 (comment) @danpat used the new multistage feature to reduce the Docker image size. We're now down to 9 MB for the latest tag which is quite impressive imho:

https://hub.docker.com/r/osrm/osrm-backend/tags/

Closing this as resolved.