Proposal: early stage layers from multi-stage builds should not be considered dangling

[daveisfera]

This is a bit of the opposite request from #34151 and #34513 because currently we have to explicitly tag the early stages so that docker image prune doesn't clean them up and blow away build cache that should still be valid.

The current workaround is to run the following:

docker build --target build -t myimage_build myimage
docker build -t myimage myimage

This is inefficient because the build for the build stage has to happen twice. The second time it's all cached, but it means that the context has to be reloaded and this explicit tagging is required when it shouldn't be necessary because the image isn't dangling and is part of a tagged build.

[cpuguy83]

I think with buildkit enabled this may not be an issue, but I'm not positive.
Have you tried, with Docker 18.06 in experimental mode, with DOCKER_BUILDKIT=1 docker build <args>?

[thaJeztah]

Yes, with buildkit, the build-cache no longer creates those images; build-cache is kept separate from the image store, and can be pruned separately (docker builder prune; added through docker/cli#1295 in docker 18.09).

Docker 18.09 also brings a configurable garbage-collect policy for the build-cache (see #37846 for details)

[daveisfera]

Is there documentation on this functionality somewhere? I'd like to understand how it works and make sure that it will meet out needs.

[thaJeztah]

Most information right now can be found in those pull requests (more documentation still has to be written), but you can try a release candidate of 18.09 to test those features, and check if the behavior matches your expectations / use-case.

[daveisfera]

Is there a way to view the build-cache?

[thaJeztah]

The files themselves? Not sure tbh (@tiborvass @tonistiigi @AkihiroSuda should know). If I'm not mistaken, the size will show up in docker system df

[tiborvass]

@daveisfera Build cache can be viewed with docker system df -v, and in master docker cli (or 18.09) there is a way to get --format {{json .}} in conjunction with -v, so you could do docker system df -v --format {{json .}} | jq .BuildCache

[tianon]

I can confirm that BuildKit handles this case correctly. 👍

(Probably safe to close since 23.0.0 will have BuildKit enabled by default?)