-
Notifications
You must be signed in to change notification settings - Fork 8.2k
build: garbage collection page #16032
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| --- | ||
| title: Garbage collection | ||
| keywords: build, buildx, buildkit, garbage collection, prune | ||
| --- | ||
|
|
||
| While [`docker build prune`](../../../engine/reference/commandline/builder_prune.md) | ||
| or [`docker buildx prune`](../../../engine/reference/commandline/buildx_prune.md) | ||
| commands run at once, garbage collection runs periodically and follows an | ||
| ordered list of prune policies. | ||
|
|
||
| Garbage collection runs in the BuildKit daemon. The daemon clears the build | ||
| cache when the cache size becomes too big, or when the cache age expires. The | ||
| following sections describe how you can configure both the size and age | ||
| parameters by defining garbage collection policies. | ||
|
|
||
| ## Configuration | ||
|
|
||
| Depending on the [driver](../drivers/index.md) used by your builder instance, | ||
| the garbage collection will use a different configuration file. | ||
|
|
||
| If you're using the [`docker` driver](../drivers/docker.md), garbage collection | ||
| can be configured in the [Docker Daemon configuration](../../../engine/reference/commandline/dockerd.md#daemon-configuration-file). | ||
| file: | ||
|
|
||
| ```json | ||
| { | ||
| "builder": { | ||
| "gc": { | ||
| "enabled": true, | ||
| "defaultKeepStorage": "10GB", | ||
| "policy": [ | ||
| {"keepStorage": "10GB", "filter": ["unused-for=2200h"]}, | ||
| {"keepStorage": "50GB", "filter": ["unused-for=3300h"]}, | ||
| {"keepStorage": "100GB", "all": true} | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| For other drivers, garbage collection can be configured using the | ||
| [BuildKit configuration](../../buildkit/toml-configuration.md) file: | ||
|
|
||
| ```toml | ||
| [worker.oci] | ||
| gc = true | ||
| gckeepstorage = 10000 | ||
| [[worker.oci.gcpolicy]] | ||
| keepBytes = 512000000 | ||
| keepDuration = 172800 | ||
| filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout"] | ||
| [[worker.oci.gcpolicy]] | ||
| all = true | ||
| keepBytes = 1024000000 | ||
| ``` | ||
|
|
||
| ## Default policies | ||
|
|
||
| Default garbage collection policies are applied to all builders if not | ||
| already set: | ||
|
|
||
| ``` | ||
| GC Policy rule#0: | ||
| All: false | ||
| Filters: type==source.local,type==exec.cachemount,type==source.git.checkout | ||
| Keep Duration: 48h0m0s | ||
| Keep Bytes: 512MB | ||
| GC Policy rule#1: | ||
| All: false | ||
| Keep Duration: 1440h0m0s | ||
| Keep Bytes: 26GB | ||
| GC Policy rule#2: | ||
| All: false | ||
| Keep Bytes: 26GB | ||
| GC Policy rule#3: | ||
| All: true | ||
| Keep Bytes: 26GB | ||
| ``` | ||
|
Comment on lines
+62
to
+78
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Where does this come from, it looks like command output? (partly for curiosity, but also we should include the command used so that users can check their own policies themselves).
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm using buildctl to retrieve this info from a BuildKit container: $ docker exec -it buildx_buildkit_builder20 buildctl debug workers -v
ID: vhcmyn5zy0ub763bj4thm0l6c
Platforms: linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/mips64le,linux/mips64,linux/arm/v7,linux/arm/v6
BuildKit: github.com/moby/buildkit v0.10.0-607-gf394afc59.m f394afc591a578537c3242a09c8b876db14a6d7f.m
Labels:
org.mobyproject.buildkit.worker.executor: oci
org.mobyproject.buildkit.worker.hostname: docker-desktop
org.mobyproject.buildkit.worker.network: host
org.mobyproject.buildkit.worker.oci.process-mode: sandbox
org.mobyproject.buildkit.worker.snapshotter: overlayfs
GC Policy rule#0:
All: false
Filters: type==source.local,type==exec.cachemount,type==source.git.checkout
Keep Duration: 48h0m0s
Keep Bytes: 512MB
GC Policy rule#1:
All: false
Keep Duration: 1440h0m0s
Keep Bytes: 26GB
GC Policy rule#2:
All: false
Keep Bytes: 26GB
GC Policy rule#3:
All: true
Keep Bytes: 26GBMaybe we could add this in buildx when inspecting a builder? 🤔
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Default policies:
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh awesome, ok, let's not put that command in then, since it's a bit fiddly 😄
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Yes! That would be really useful info to pull through 🎉 |
||
|
|
||
| * `rule#0`: if build cache uses more than 512MB delete the most easily | ||
| reproducible data after it has not been used for 2 days. | ||
| * `rule#1`: remove any data not used for 60 days. | ||
| * `rule#2`: keep the unshared build cache under cap. | ||
| * `rule#3`: if previous policies were insufficient start deleting internal data | ||
| to keep build cache under cap. | ||
|
|
||
| > **Note** | ||
| > | ||
| > "Keep bytes" defaults to 10% of the size of the disk. If the disk size cannot | ||
| > be determined, it defaults to 2GB. | ||
Uh oh!
There was an error while loading. Please reload this page.