Allow controlling cache mounts storage location

Allowing exporting contents of the type=cache mounts have been asked many times in different repositories and slack.

Regular remote instruction cache does not work for cache mounts that are not tracked by cache key and are just a location on disk that can be shared by multiple builds.

Currently, the best approach to maintain this cache between nodes is to do it as part of the build. docker/buildx#244 (comment)

I don't think we should try to combine cache mounts with the remote cache backends. Usually, cache mounts are for throwaway cache and restoring it would take a similar time to just recreating it.

What we could do is to allow users to control where the cache location is on disk, in case it is not on top of the snapshots.

We can introduce a cache mount backend concept behind a go interface that different implementation can implement.

Eg. for a Dockerfile like

RUN --mount=type=cache,target=/root/.cache,id=gocache go build ...

you could invoke a build with

docker build --cache-mount id=gocache,type=volume,volume=myvolume .

In that case, the cache could use a Docker volume as a backend. I guess good drivers would be volume in Docker and bind mount from the host for non-docker. If no --cache-mount is specified, the usual snapshotter-based location is used.

From the security perspective, BuildKit API is considered secure by default for the host, so I think this would require daemon side configuration to enable what paths can be bound.

Another complexity is buildx container driver as we can't easily add new mounts to a running container atm. Possible solutions are to force these paths to be set on buildx create or do some hackery with mount propagation.

Just to clarify whether the scope of this proposal covers a use case I have.

Go has content-addressed build and module caches defined via go env GOCACHE and, as of 1.15, go env GOMODCACHE (which was (go env GOPATH)[0]/pkg/mod in previous go versions).

Have read the documentation at https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md, it does not appear possible to delegate a cache to a host directory. Therefore a buildkit Go build and module cache will likely largely duplicate the build and module caches that exists on the host.

However, this proposal seems to be heading in the direction of sharing such a cache:

In that case, the cache could use a Docker volume as a backend

Under this proposal, can I confirm, would it be possible to delegate these caches to host directories?

I note a couple of requirements:

in the case of Go, the location of these directories is not guaranteed to be given by an environment variable, rather the output of go env GOCACHE and go env GOMODCACHE is definitive. Whilst not the end of the world if environment variables were the only way of passing values, support the output of go env GOCACHE and go env GOMODCACHE would be even better
the UID and GID of writes to the cache should default to be that of the caller

Apologies if this covers old ground (covered elsewhere); I'm rather new to the buildkit approach but happened upon this issue and it sounded like exactly the thing I was after.

Many thanks

@tonistiigi

hope this feature could land asap.

it will be useful for ci caching. #1673 (comment)

for host directories, could be hacking by

# ensure host path
$ mkdir -p /tmp/gocache

# create volume
$ docker volume create --driver local \
      --opt type=none \
      --opt device=/tmp/gocache \   
      --opt o=bind \
      myvolume 

$ docker run -it --volume myvolume:/go/pkg/mod busybox touch /go/pkg/mod/test.txt
# test.txt will be created under host dir /tmp/gocache/

# maybe work
$ docker buildx build --cache-mount id=gocache,type=volume,volume=myvolume .

@tonistiigi should we control the mount target too?
--cache-mount id=gocache,type=volume,volume=myvolume,target=/go/pkg/mod

Does the proposed solution cater to a scenario of a buildserver which uses docker-in-docker? I'm not sure tbh.

Any news on this? This shouldn’t be a really big change, right?

This would solve my life.

Would love to see this. Would be a huge win for speeding up builds on CI

This would be brilliant for build systems like Gradle and Maven building on e.g. GitHub Actions.

They typically download all their dependencies to a cache dir. It's hard to benefit from layer caching - dependencies can be expressed in multiple files in a nested folder structure, so to avoid a maintenance nightmare it's generally necessary for the Dockerfile to do a COPY . . before running the Gradle / Maven command that downloads the dependencies, which in turn means the layer cache is invalid nearly every time. Downloading the transitive dependencies is very chatty, can be hundreds of HTTP requests, so it's well worth turning into a single tar ball.

I really want to use the same Dockerfile to build locally and on CI, which I think means I don't want to use the strategy suggested in docker/buildx#244 (comment) of loading & exporting the cache to named locations as commands in the Dockerfile - it might work in CI but would be much less efficient building locally, as well as adding a lot of noise to the Dockerfile.

I'm currently caching the whole of the /var/lib/docker dir (!) and restarting the docker service on the GitHub Action runner, which is also pretty slow and expensive, and generally not a great idea!

I'm guessing it wouldn't be a great place for a buildx AND go newbie to start contributing, though...

I would love to see

RUN --mount=type=cache,target=/root/.m2,id=mvncache mvn package

be exported as a own layer/blob in the --cache-to and --cache-from arguments like --cache-to type=registry,ref=myregistry/myapp:cache.

Perhaps even better if it got it's own argument like --cache-id id=mvncache,type=registry,ref=myregistry/dependency-cache:mvn and thus you can share maven dependencies between similar projects and avoiding downloading so much from the Internet.

I have hacked together a solution that seems to work for including cache mounts in the GitHub actions/cache action. It works by dumping the entire buildkit state dir to a tar file and archiving that in the cache, similar to the approach described here. I think this dump also includes the instruction cache, so this should not be archived separately if using the action.

Because this cache will grow on every run, we use docker buildx prune --force --keep-storage to remove everything but the cache before archiving. You will need to adjust the cache-max-size var to suit your needs, up to GitHub's limit of 5g. It's a dirty hack, and could probably be improved by exporting only the relevant cache mounts, but I am new to both buildkit and GitHub Actions so this is what I came up with. Comments and suggestions for different approaches are welcome.

It would be nice to allow setting the from parameter to the build context.

hello, is that feature currently planned or being worked on?

I think the design is accepted. Not being worked on atm.

@tonistiigi

Usually, cache mounts are for throwaway cache and restoring it would take a similar time to just recreating it.

@tonistiigi can you clarify what you mean by this? This sounds like it is only considering use cases such as caching dependency installation where downloading the dependencies from a repository would take the same time to download the cache itself. What about situations where regenerating the cache takes significantly longer than importing it (e.g. code building)? The desire is to re-use parts of the cache without erasing it entirely, something that the remote instruction cache does not support.

@chris13524

@chris13524 That's what "usually" means there, "not always". Even with code building cache you might have mixed results. Importing and exporting cache is not free. In addition you need something that can clean up the older cache from your mount or your cache just infinitely grows with every invocation. You can try this today by importing/exporting cache out of the builder into some persistent storage(or additional image) with just an additional built request. docker/buildx#244 (comment)

@thompson-shaun

I see this issue has being added to v0.future milestone by @thompson-shaun . Hopefully this can be prioritize and move to v.0.17 milestone given the huge interest in this feature from the community.

Also in my case the lack of this functionality is making the docker builds much slower in CI then in local environments...

bump

I would love if this was included ASAP to save me from using hacky workarounds in CI

bump

Could save so much time for everyone.

bump, waiting for this

Is this feature already implemented by other OCI Containerfile implementations?

mount=type=bind,read-write,source=/path/on/host,destination=/path/in/container,relabel=shared

https://github.com/containers/common/blob/main/docs/Containerfile.5.md#format

Need this now that github action caching is broken.

@tonistiigi

@tonistiigi could you please add https://github.com/reproducible-containers/buildkit-cache-dance to the issue description? it is mentioned in several comments, but they are hidden by default because this thread is very long

For anyone hitting this with Docker Desktop on macOS the build cache mount problem compounds with Docker's own disk image growth. Here is a practical workaround for auditing and managing the damage:

Audit your Docker disk usage on macOS:

# Docker's virtual disk image (biggest offender)
ls -lh ~/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw 2>/dev/null

# BuildKit cache specifically
docker system df -v 2>/dev/null | head -30

# Build cache breakdown
docker buildx du 2>/dev/null

Staged cleanup (least to most aggressive):

# 1. Remove only dangling build cache
docker builder prune -f

# 2. Remove all build cache (keeps images)
docker builder prune -a -f

# 3. Nuclear: reclaim everything
docker system prune -a --volumes -f

The real problem on macOS: Even after pruning, Docker Desktop's Docker.raw disk image does NOT shrink automatically. Since Docker 4.34+, VirtioFS helps, but the image still grows over time. You need to either:

Reset Docker Desktop (Settings > Resources > Disk image size)
Or manually compact: docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -u -i -n fstrim /

I maintain ClearDisk which gives a quick visual audit of Docker + all other developer caches (build cache, images, volumes) alongside Xcode, Homebrew, node_modules, etc. -- helpful because on most Mac dev machines, Docker is just one of 5-6 things silently eating disk. I also wrote a Docker disk cleanup guide with more details on the Docker.raw issue.

build: drop BuildKit cache mounts, use cargo-chef layer caching properly

To anyone running into this as an issue, I have an alternative BuildKit driver that implements this, docs are here https://clipper.dev/docs/buildkit

It uses a lazy loading FUSE driver to only pull the cache files your build uses, so it should be fast.

perf(ci): backend 改 cargo-chef、快取搬去 GHCR——相依的編譯結果在 CI 上根本沒被存下來

🌱 CI wall-clock: shard tests + fold coverage into PR gate, Go cache m…

Navigation Menu

Repository navigation

Allow controlling cache mounts storage location #1512

43 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow controlling cache mounts storage location #1512

Description

Activity

myitcv commented on Sep 26, 2020

morlay commented on Sep 27, 2020

hansbogert commented on Nov 20, 2020

ties-s commented on Mar 4, 2021

rhyek commented on Mar 4, 2021

summera commented on Mar 20, 2021

Mahoney commented on Apr 13, 2021

Raboo commented on Apr 21, 2021

strophy commented on Jul 2, 2021

speller commented on Aug 26, 2021

clemsos commented on Sep 1, 2021

tonistiigi commented on Sep 1, 2021

chris13524 commented on Sep 20, 2021

tonistiigi commented on Sep 20, 2021

43 remaining items

massimeddu-sj commented on Sep 24, 2024

bjf5201 commented on Nov 24, 2024

royreznik commented on Jan 27, 2025

moddi3 commented on Apr 10, 2025

ruffsl commented on Apr 11, 2025

JasonDictos commented on Apr 24, 2025

ei-grad commented on Nov 18, 2025

bysiber commented on Mar 2, 2026

kyle-basis commented on Jun 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions