WITH DOCKER with cached docker root dir
#3399
Assignees
Labels
type:proposal
A proposal for a new feature
Comments
|
For now I was able to make really fast working setup with CACHE + WITH DOCKER seems like its working with overlay really fine. Because we are using cache I think that Buildkit should take care of cleaning it? Something like. Assuming that I have did 2 changes in docker-wrapper.. What I would have to do is to add |
|
Full log of |
|
I suspect that dockerd uses another storage driver in this case (although I could be wrong). Can you show the dockerd logs too? |
Dockerd logs |
|
Yeah, it seems that in this case Dockerd doesn't use overlayfs based on this line This has many disadvantages... the main one being that it tends to fill the disk very quickly as it is not a union filesystem. Read more about it here.
|
alexcb
added a commit
that referenced
this issue
May 3, 2024
Introduces a new `WITH DOCKER --cache-id=<name>` flag, which when specified will cache the daya layers from the docker data-root between runs, even when dependencies have changed. Note that the docker image tags are NOT cached; and a user will still have to re-load the image (via the WITH DOCKER --load or --pull flags); however, since the data layers are cached, the re-tagging should be much faster. fixes #3399 #3485
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Use case
Faster
WITH DOCKERload times (at the cost of slightly less repeatability).Expected Behavior
Currently
WITH DOCKERwipes the Docker root dir on each invocation. This is great for ensuring that builds are repeatable - nothing in the pre-existing Docker state from a previous (like a previously loaded image) can affect the current to cause inconsistency.That, unfortunately, comes at the cost of having to load all the images into the inner Docker on each run, an operation that takes time, even though it's just a local transfer.
This proposal introduces an option for the Docker root dir to be kept around between runs, in a similar manner to which users can use global cache directories via
--id.Possible syntax:
The
--cache-idoption does three things:Possible extensions:
Challenges
An implementation challenge is the fact that keeping the Docker root dirs around requires that they are managed and GC'd appropriately, in a similar manner that Buildkit manages the cache. These root dirs cannot be part of Buildkit's cache management, though, because of the inherent limitation that Buildkit operates on overlayfs, and the Docker root dir has to also be overlayfs, and you can't put overlayfs on top of overlayfs.
For this reason, the lifecycle of these Docker root dirs needs to be managed separately within Earthly's buildkit fork.
A v0 / initial experimental version of this feature could perhaps rely on manual-only clearing of Docker root dirs (via some Earthly command) and not require that we implement automatic GCing for it.
A v1 version could maintain a last-used timestamp in these directories and clean old entries on Buildkit startup (e.g. older than 14 days).
The text was updated successfully, but these errors were encountered: