From ee8b43aee6cd343b61827c84dfbd316ec02bbf87 Mon Sep 17 00:00:00 2001 From: adeniyistephen Date: Thu, 3 Dec 2020 16:20:11 +0100 Subject: [PATCH 1/7] Documentation for builder prune Signed-off-by: adeniyistephen --- docs/reference/commandline/builder_prune.md | 323 +++++++++++++++++++- 1 file changed, 322 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/builder_prune.md b/docs/reference/commandline/builder_prune.md index 8dbbfbd8d220..25af67f37beb 100644 --- a/docs/reference/commandline/builder_prune.md +++ b/docs/reference/commandline/builder_prune.md @@ -13,4 +13,325 @@ Remove build cache | `--keep-storage` | `bytes` | `0` | Amount of disk space to keep for cache | - + +--- +title: "builder prune" +description: "The builder prune operations with various inputs and usage" +keywords: "builder, docker, prune" +redirect_from: +- /reference/commandline/builder_prune/ +--- + + + +## Description + +First, when the `BuildKit` garbage collector is enabled it runs periodically in the background and follows an ordered list of policies. Each policy is a prune operation so the order matters. But pruning can also be done manually with `docker builder prune` with is respective prune inputs to remove build cache. + +# builder prune + +```markdown +Usage: docker builder prune [OPTIONS] + +Remove build cache + +Options: + -a, --all Remove all unused build cache, not just dangling ones + --filter filter Provide filter values (e.g. 'until=24h') + -f, --force Do not prompt for confirmation + --keep-storage bytes Amount of disk space to keep for cache +``` + +## Usage + +Build cache records can be inspected with the following command: ` docker system df -v` +(but shows more than just build cache records) or with buildx via: `buildx du --verbose` +with a bit different output. + +```bash +$ docker system df -v + +Build cache usage: 2.97GB + +CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED +y6xp6ur6a7ku regular 265B 2 weeks ago 2 weeks ago 1 false +ace0eda3e3be regular 5.57MB 2 weeks ago 2 weeks ago 2 false +db9945da3e79 regular 4.68kB 2 weeks ago 2 weeks ago 2 false +6a3365937e92 regular 0B 2 weeks ago 2 weeks ago 2 false +271d635f4277 regular 49.9MB 2 weeks ago 2 weeks ago 2 false +2e86a56bd8aa regular 138B 2 weeks ago 2 weeks ago 3 false +rungovuto78m regular 311B 2 weeks ago 2 weeks ago 1 false +nhqetzusaoxp source.local 695B 2 weeks ago 2 weeks ago 1 false +7sg0fy1pmvtt regular 0B 2 weeks ago 2 weeks ago 1 false +gi8wypsigtjm source.local 340B 2 weeks ago 2 weeks ago 1 false +cd21hgno3hax regular 2.42MB 2 weeks ago 2 weeks ago 3 true +36bctagchywz regular 540MB 2 weeks ago 2 weeks ago 3 false +l1wnrjeqm9pa regular 81MB 2 weeks ago 2 weeks ago 3 false +1jr4qt5wqgjl regular 0B 2 weeks ago 2 weeks ago 3 false +9780f6d83e45 regular 114MB 2 weeks ago 2 weeks ago 4 false +5173011923d0 regular 16.5MB 2 weeks ago 2 weeks ago 4 false +4bb57adf9037 regular 17.5MB 2 weeks ago 2 weeks ago 4 false +d5d618196ec3 regular 146MB 1 weeks ago 1 weeks ago 4 false +1a893709dfe5 regular 32.9MB 2 minutes ago 2 minutes ago 4 false +a5998494261f regular 345MB 7 minutes ago 7 minutes ago 4 false +``` + +The [docker builder prune](commandline/builder_prune.md) command line remove all build cache +that are dangling without specifying any of the prune flag [OPTIONS]. + +Before using the `docker builder prune` command, this is what the cache record looks like with the `docker system df -v` command: + +```bash +$ docker system df -v + +Build cache usage: 647MB + +CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED +y6xp6ur6a7ku regular 265MB 2 weeks ago 2 weeks ago 1 false +ace0eda3e3be regular 5.57MB 2 weeks ago 2 weeks ago 0 false +db9945da3e79 regular 254MB 2 weeks ago 2 weeks ago 1 false +6a3365937e92 regular 15.84MB 2 weeks ago 2 weeks ago 0 false +271d635f4277 regular 49.9MB 2 weeks ago 2 weeks ago 2 false +2e86a56bd8aa regular 38MB 2 weeks ago 2 weeks ago 3 false +rungovuto78m regular 311B 2 weeks ago 2 weeks ago 0 false +nhqetzusaoxp source.local 695B 2 weeks ago 2 weeks ago 1 false +7sg0fy1pmvtt regular 10MB 2 weeks ago 2 weeks ago 0 false +gi8wypsigtjm source.local 340B 2 weeks ago 2 weeks ago 1 false +``` +This was the record after using the `docker builder prune` command: + +```bash +$ docker system df -v + +Build cache usage: 647MB + +CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED +y6xp6ur6a7ku regular 265MB 2 weeks ago 2 weeks ago 1 false +db9945da3e79 regular 254MB 2 weeks ago 2 weeks ago 1 false +271d635f4277 regular 49.9MB 2 weeks ago 2 weeks ago 2 false +2e86a56bd8aa regular 38MB 2 weeks ago 2 weeks ago 3 false +nhqetzusaoxp source.local 695B 2 weeks ago 2 weeks ago 1 false +gi8wypsigtjm source.local 340B 2 weeks ago 2 weeks ago 1 false +``` +As noticed, some dangeling cache are deleted from the record based on the USAGE Status. Does cache record showing 0 usage are not +being used and therefore are been deleted, also if a cache is been shared by two or more images i.e SHARED = true, the cache can +not be deleted, but we have all the cache in false (they are not being shared). Cache can not be deleted if `USAGE = 0` +and `SHARED = true` same applies to `USAGE = 1` and `SHARED = false` both of the cache attribute needs to correlate. +i.e `SHARED = false` and `USAGE = 0`. + + +## Examples + +The [docker builder prune --help](commandline/builder_prune.md) shows all the flag / options that can be used for builder prune. +This is what the output looks like: + +```bash +$ docker builder prune --help + +Usage: docker builder prune + +Remove build cache + +Options: + -a, --all Remove all unused build cache, not just dangling ones + --filter filter Provide filter values (e.g. 'until=24h') + -f, --force Do not prompt for confirmation + --keep-storage bytes Amount of disk space to keep for cache +``` + +## Build Prune With (--force) flag + +First of all, the `--force` `-f` flag in [docker builder prune](commandline/builder_prune.md) helps to bypass +the authentication that pops up when we use `docker builder prune` and any of its options. it can be used with other +options, applications and usage examples of `--force` are shown with other options below. + +> **Note** +> +> This feature dose not allow you to force remove a cache that is in use. +> it only bypass authentications. + + +## Build Prune With (--all, -a) flag + +Builder prune has a flag for removing all unused build cache, not just dangling caches, Discription of this flag is below. +First we check for builder caches: + +```bash +$ docker system df -v + +Build cache usage: 941.3MB + +CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED +a5998494261f regular 345MB 2 weeks ago 2 weeks ago 4 false +bdd3e2e51daf regular 38.2MB 2 weeks ago 2 weeks ago 0 false +zii5dwynmvgf regular 3.56kB 2 weeks ago 2 weeks ago 2 false +p04j8skx2cio regular 4.89kB 2 weeks ago 2 weeks ago 2 false +aph1w8hfh3u5 regular 15.4MB 2 weeks ago 2 weeks ago 1 false +nj9l7si31hw7 regular 540MB 2 weeks ago 2 weeks ago 1 false +2s1t824ku5g7 regular 2.42MB 2 weeks ago 2 weeks ago 0 false +qb9txhrba7vx regular 0B 2 weeks ago 2 weeks ago 1 false +nlaulj32zp6e regular 4.89kB 2 weeks ago 2 weeks ago 3 false +kyogwgykftyr regular 0B 2 weeks ago 2 weeks ago 0 false +``` +Now we use the `--all, -a` flag. + +```bash +$ docker builder prune --all + +# A warning notification pops up, that you would need to answer yes or no +# y answers yes to remove all build cache. +# n answers no to decline the removal of build cache. + +WARNING! This will remove all build cache. Are you sure you want to continue? [y/N] y +# y was selected i.e delete all cache. + +Deleted build cache objects: +bdd3e2e51dafnchruxumyash1 +a5998494261fqksutu2630zkk +zii5dwynmvgfdlrdw7vkqup60 +p04j8skx2cio7hsp60k1b7adm +aph1w8hfh3u5r6wxqngb45989 +nj9l7si31hw7ih17y44qe412z +2s1t824ku5g7kthe8ecu6k3xe +qb9txhrba7vxxm9vo7r0p490p +nlaulj32zp6e5zgwl7z95cmk5 +kyogwgykftyrmgvbd3wl3hg9y + +Total reclaimed space: 941.3MB +``` +## Using -force, -f flag with --all flag + +This is an example that shows what an -f flag does in a `docker builder prune --all -f` or `docker builder prune --all --force` +command + +```bash +$ docker system df -v # to check for build cache + +Build cache usage: 112.78MB + +CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED +d0fe97fa8b8c regular 69.2MB 2 weeks ago 13 days ago 5 false +4d6b84966400 regular 41.3MB 2 weeks ago 13 days ago 1 false +bb186d194dec regular 2.28MB 2 weeks ago 13 days ago 2 false + +$ docker builder prune --all -f #to remove all build cache with the force flag. + +# NOTE: No pop up showed up to ask if you approve the removal of the build cache. + +Deleted build cache objects: +d0fe97fa8b8cq19z9hqs5tphe +4d6b84966400179cjn6dgqtaz +bb186d194decu5qrg0jjmw6tp + +Total reclaimed space: 112.78MB +``` + +## Build Prune With (--filter) flag + +The filter flag for `builder prune` is used to `filter` what type of filter you want to remove based on the usage time +i.e the last time it was used, below we will show how the `--filter` flag is been used in our builder prune command. +Command: `docker builder prune --filter unused-for=24h` or `docker builder prune --filter until=24h` + +What the command simply say is that remove every builder cache that has not been used for the past 24 hours or +that are not within 24 hours range. + +**Note** +> Setting `unused-for` or `until` should be in Hours + +```bash +$ docker system df -v ## to check the builder cache + +Build cache usage: 652.98MB + +CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED +e76a5m45h6pw regular 74.5MB 14 hours ago 14 hours ago 1 false +xlpunf7hd5yh regular 36.1MB 17 hours ago 17 hours ago 1 false +rx58kxz9kmlg regular 540MB 22 hours ago 22 hours ago 1 false +08unho1rnguz regular 2.38MB 13 days ago 13 days ago 1 false +i8nfk2fvpyyi regular 0B 2 weeks ago 13 days ago 2 false + +$ docker builder prune --filter until=24h # to remove builder cache with a filter or 24 hours + +WARNING! This will remove all dangling build cache. Are you sure you want to continue? [y/N] y + +# y (yes) was selected. + +Deleted build cache objects: +08unho1rnguz179cjn6dgqtaz +i8nfk2fvpyyiu5qrg0jjmw6tp + +Total reclaimed space: 2.38MB +``` +**Note** +>Two builder cache were deleted, the two were used 13 days ago, but we specify that we want to delete every build +>cache that has not been used for the last 24 hours. + +## Using -f, --force flag with --filter. +By now we know what the `-f` or `--force` flag does, in this example we are going to see how to +implement the `-f` or `--force` flag with our `--filter` flag. + +```bash +$ docker system df -v # to check for builder cache + +Build cache usage: 8.625KB + +CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED +hjjsk55wnswi regular 0B 2 weeks ago 13 days ago 6 false +e6vrmmkb82mm source.local 0B 2 weeks ago 13 days ago 19 false +g7hi043vg9ml source.local 265B 3 days ago 2 hours ago 13 false +8mtuu862jx59 regular 3.5kB 13 days ago 13 days ago 1 false +dalvi76v52o2 regular 4.86kB 11 days ago 54 minutes ago 1 false + +$ docker builder prune --filter unused-for=24h --force + +# NOTE: No pop up showed up to ask if you approve the removal of the build cache. +Deleted build cache objects: +hjjsk55wnswizmxpf551sp2qh +e6vrmmkb82mmffmls87xe41k8 +8mtuu862jx59yyqyguxrhrdv8 + + +Total reclaimed space: 3.5kb +``` + +## Build Prune With (--keep-storage) flag + +Docker as a way to manage the storage space that build cache can consume. The `--keep-storage` command helps +us to manage the amount of storage space that our build cache can have. + +**Note** +>If build cache as been asigned to only have 1GB of storage, i.e it can only store upto 1GB amount of build caches +>and if the storage is full, it can no longer store more build caches therefore any other build will not be stored. +>i.e when the build is been run again, it starts from the begining because the build was not cached on the first run. + +Usage is below: + +```bash +$ docker builder prune --keep-storage 5gb + +WARNING! This will remove all dangling build cache. Are you sure you want to continue? [y/N] y + +# y (yes) was selected. + +Total reclaimed space: 0B +``` +## Using -f, --force flag with --keep-storage. + +In this example we are going to see how to +implement the `-f` or `--force` flag with our `--keep-storage` flag. + +```bash +$ docker builder prune --keep-storage 5gb -f + +# NOTE: No pop up showed up to ask if you approve the removal of the build cache. +Total reclaimed space: 0B +``` +This are what you need to know on how to use `builder prune` on your docker engine. + \ No newline at end of file From 4e67ef057cc39510c8c50f16368215eb746a5202 Mon Sep 17 00:00:00 2001 From: adeniyistephen Date: Sun, 6 Dec 2020 17:10:03 +0100 Subject: [PATCH 2/7] Documentation of builder prune with its inputs and examples Signed-off-by: adeniyistephen --- docs/reference/commandline/builder_prune.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/reference/commandline/builder_prune.md b/docs/reference/commandline/builder_prune.md index 25af67f37beb..cc91608f4bc4 100644 --- a/docs/reference/commandline/builder_prune.md +++ b/docs/reference/commandline/builder_prune.md @@ -33,7 +33,9 @@ redirect_from: ## Description -First, when the `BuildKit` garbage collector is enabled it runs periodically in the background and follows an ordered list of policies. Each policy is a prune operation so the order matters. But pruning can also be done manually with `docker builder prune` with is respective prune inputs to remove build cache. +When the Garbage collector in the buildkit daemon configuration is enabled it runs in the background periodically which it follows an ordered list of +policies, policies which are prune operations, so the order matters. Pruning can be done and tested manually in the cli with `docker builder prune` command, +we are going to look at how pruning helps with build caches including different inputs to configure our prune. # builder prune @@ -51,8 +53,9 @@ Options: ## Usage -Build cache records can be inspected with the following command: ` docker system df -v` -(but shows more than just build cache records) or with buildx via: `buildx du --verbose` +Our build caches records can be inspected with the following command: `docker system df -v` +(it shows image, container and volume caches) +or with buildx command via: `buildx du --verbose` with a bit different output. ```bash From 72ef7f25ba71b8c5748b61dac47a334d53146e64 Mon Sep 17 00:00:00 2001 From: adeniyistephen Date: Tue, 15 Dec 2020 21:07:19 +0100 Subject: [PATCH 3/7] Modified buider prune and add garbage collection config Signed-off-by: adeniyistephen --- docs/reference/commandline/builder_prune.md | 112 ++++++++++++++--- .../commandline/garbagecollection_config.md | 119 ++++++++++++++++++ 2 files changed, 214 insertions(+), 17 deletions(-) create mode 100644 docs/reference/commandline/garbagecollection_config.md diff --git a/docs/reference/commandline/builder_prune.md b/docs/reference/commandline/builder_prune.md index cc91608f4bc4..ada536ec1ed2 100644 --- a/docs/reference/commandline/builder_prune.md +++ b/docs/reference/commandline/builder_prune.md @@ -31,11 +31,27 @@ redirect_from: will be rejected. --> -## Description +# Description -When the Garbage collector in the buildkit daemon configuration is enabled it runs in the background periodically which it follows an ordered list of -policies, policies which are prune operations, so the order matters. Pruning can be done and tested manually in the cli with `docker builder prune` command, -we are going to look at how pruning helps with build caches including different inputs to configure our prune. +Working with Docker, docker have various unused objects hanging around from previous activities, so docker +takes a conservative approach to clean up unused objects, part of the object which we would be talking about is the builder cache +these object are stored when building dockerfiles, there're generally not removed unless you explicitly ask Docker to do so. This +can cause Docker to use extra disk space. Docker provides a prune command `docker builder prune` or `docker buildx prune` to clean +up multiple unused build cache at once. This topic shows how to use these prune command and it different inputs to clean build +cache. + +When we execute the `$ docker builder prune` command, it makes an API call to the Docker daemon, and the daemon searches for all +unused build cache objects on that host and removes those objects from the system. + +The goal of creating cache is to speed up future builds. +Removing unused build cache is very important in docker so as to build effective dockerfiles, pruning removes dead and unused +build cache allowing room for new and useful cache. It also defers bad build cache and promotes the builds to work faster and +effective. +So docker try to remove the cache that is less likely to be used in future builds to keep builds as fast as possible while maintaining cache size under control. + +**Note** +>Pruning does not works if the Total is below what you specified as keep storage option to prune. It needs to go above, we are +>going to see how to use the `--keep-storage` option as part of our examples in this topic. # builder prune @@ -51,8 +67,10 @@ Options: --keep-storage bytes Amount of disk space to keep for cache ``` -## Usage +# Usage +To see how much cache you are using currently you can use `docker system df` to get a total view for what’s being in use including +containers, images, and build cache. Our build caches records can be inspected with the following command: `docker system df -v` (it shows image, container and volume caches) or with buildx command via: `buildx du --verbose` @@ -130,7 +148,7 @@ and `SHARED = true` same applies to `USAGE = 1` and `SHARED = false` both of the i.e `SHARED = false` and `USAGE = 0`. -## Examples +# Examples The [docker builder prune --help](commandline/builder_prune.md) shows all the flag / options that can be used for builder prune. This is what the output looks like: @@ -149,19 +167,18 @@ Options: --keep-storage bytes Amount of disk space to keep for cache ``` -## Build Prune With (--force) flag +# Build Prune With (--force) flag First of all, the `--force` `-f` flag in [docker builder prune](commandline/builder_prune.md) helps to bypass the authentication that pops up when we use `docker builder prune` and any of its options. it can be used with other options, applications and usage examples of `--force` are shown with other options below. > **Note** -> > This feature dose not allow you to force remove a cache that is in use. > it only bypass authentications. -## Build Prune With (--all, -a) flag +# Build Prune With (--all, -a) flag Builder prune has a flag for removing all unused build cache, not just dangling caches, Discription of this flag is below. First we check for builder caches: @@ -236,7 +253,7 @@ bb186d194decu5qrg0jjmw6tp Total reclaimed space: 112.78MB ``` -## Build Prune With (--filter) flag +# Build Prune With (--filter) flag The filter flag for `builder prune` is used to `filter` what type of filter you want to remove based on the usage time i.e the last time it was used, below we will show how the `--filter` flag is been used in our builder prune command. @@ -271,10 +288,41 @@ Deleted build cache objects: i8nfk2fvpyyiu5qrg0jjmw6tp Total reclaimed space: 2.38MB + +# Two builder cache were deleted, the two were used 13 days ago, but we specify that we want to delete every build +# cache that has not been used for the last 24 hours. +``` + +### Example of --all and --filter flag combination. +```bash +$ docker system df -v ## to check the builder cache + +Build cache usage: 336.28MB + +CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED +rkze7nhuwsrh regular 36.1MB 2 weeks ago 12 hours ago 1 false +qemxnvg1n1o8 regular 36.1MB 2 weeks ago 2 weeks ago 2 false +8l4s9m2j2z4b regular 74.5MB 2 weeks ago 2 weeks ago 0 false +d0fe97fa8b8c regular 69.2MB 2 weeks ago 10 days ago 1 false +4d6b84966400 regular 41.3MB 2 weeks ago 13 days ago 0 false +bb186d194dec regular 2.28MB 2 weeks ago 17 hours ago 1 false +216461ee9eb5 regular 76.8MB 2 weeks ago 13 days ago 0 false + +$ docker builder prune --all --filter unused-for = 24h + +WARNING! This will remove all dangling build cache. Are you sure you want to continue? [y/N] y +#y selected. + +Deleted build cache objects: + +8l4s9m2j2z4biu5qrg0jjmw6tp +4d6b84966400yyqyguxrhrdv8k +216461ee9eb5xm9vo7r0p490pg + +Total reclaimed space: 38.38MB + +# only build cache that are unused and are last used for more than 24 hours were deleted. ``` -**Note** ->Two builder cache were deleted, the two were used 13 days ago, but we specify that we want to delete every build ->cache that has not been used for the last 24 hours. ## Using -f, --force flag with --filter. By now we know what the `-f` or `--force` flag does, in this example we are going to see how to @@ -304,7 +352,7 @@ e6vrmmkb82mmffmls87xe41k8 Total reclaimed space: 3.5kb ``` -## Build Prune With (--keep-storage) flag +# Build Prune With (--keep-storage) flag Docker as a way to manage the storage space that build cache can consume. The `--keep-storage` command helps us to manage the amount of storage space that our build cache can have. @@ -312,18 +360,47 @@ us to manage the amount of storage space that our build cache can have. **Note** >If build cache as been asigned to only have 1GB of storage, i.e it can only store upto 1GB amount of build caches >and if the storage is full, it can no longer store more build caches therefore any other build will not be stored. ->i.e when the build is been run again, it starts from the begining because the build was not cached on the first run. +>i.e when the prune, it deletes the ones that are unused until it gets to 1GB. Usage is below: ```bash -$ docker builder prune --keep-storage 5gb +$ docker builder prune --keep-storage 512MB WARNING! This will remove all dangling build cache. Are you sure you want to continue? [y/N] y # y (yes) was selected. Total reclaimed space: 0B + +# now let use prune our build cache + +$ docker system df -v # to check for builder cache + +Build cache usage: 624.5MB + +CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED +nhqetzusaoxp source.local 695B 2 weeks ago 2 weeks ago 0 false +7sg0fy1pmvtt regular 0B 2 weeks ago 2 weeks ago 0 false +gi8wypsigtjm source.local 340B 2 weeks ago 2 weeks ago 1 false +cd21hgno3hax regular 2.42MB 2 weeks ago 2 weeks ago 3 true +36bctagchywz regular 540MB 2 weeks ago 2 weeks ago 0 false +l1wnrjeqm9pa regular 81MB 2 weeks ago 2 weeks ago 3 false +1jr4qt5wqgjl regular 0B 2 weeks ago 2 weeks ago 3 false +bdd3e2e51daf regular 38.2MB 2 weeks ago 2 weeks ago 0 false + +$ docker builder prune --all # Remove all unused cache till its under 512MB(Keep Storage limit) + +WARNING! This will remove all dangling build cache. Are you sure you want to continue? [y/N] y + +nhqetzusaoxpxm9vo7r0p490p +7sg0fy1pmvtt5zgwl7z95cmk5 +36bctagchywzmgvbd3wl3hg9y + +# we have 4 unused cache there, but he removed 3 unused cache, because it has removed the amount of cache down to what we needed +# 512MB keep storage, and based on our filter. + +Total reclaimed space: 540.70MB ``` ## Using -f, --force flag with --keep-storage. @@ -336,5 +413,6 @@ $ docker builder prune --keep-storage 5gb -f # NOTE: No pop up showed up to ask if you approve the removal of the build cache. Total reclaimed space: 0B ``` -This are what you need to know on how to use `builder prune` on your docker engine. +[Garbage collection](commandline/garbagecollection_config.md) is done in the pruning process, its an ordered list of prune +operations, click the link to know more and see how to configure garbage collection. \ No newline at end of file diff --git a/docs/reference/commandline/garbagecollection_config.md b/docs/reference/commandline/garbagecollection_config.md new file mode 100644 index 000000000000..073da7b4dbed --- /dev/null +++ b/docs/reference/commandline/garbagecollection_config.md @@ -0,0 +1,119 @@ +## Description +Garbage collection configuration is an orderd list of prune operations, prune run its operation ones while garbage collection runs +it operation based on its default policy, which we would be looking at in this topic and how to configure them. +Click To know more about [build cache prune.](commandline/builder_prune.md) + +Garbage collector works when it is enabled, it runs periodically in the background and follows an ordered list of policies, this +configuration is done in the `etc/docker/daemon.json` config file, its not a command line configuration. + +## Enable GC + +The following enables the default GC on your docker daemon: + +```bash +"builder": { + "gc": { + "enabled": true + } + } +``` +The garbage collection config file can be created incase you dont have one located in your `etc/docker` folder, and if you do have +one it migth be there and be set to `false` so you need to enable the garbage collection by setting it as true, if you don't see +any gc JSON, copy the above JSON and paste it in the file. That is the very first step to configure your garbage collection. +Garbage collection config JSON are optional, i.e some fields can be left out, like the example above about enabling GC, the policy +field was left out including the defaultKeepStorage. Next we are going to talk about defaultKeepStorage in the JSON file and +it's configuration. + +## defaultKeepStorage +GC works based on `defaultKeepStorage` and the default keep storage is 10% of local disk space which can be configured, +the `defaultKeepStorage` is very useful to measure the amount of cache to be deleted. When the `defaultKeepStorage` is set +it uses the `defaultKeepStorage` value to set a limit to the amount of build cache to be deleted. +If a defaultKeepStorage is set to 10GB and you have a build cache of 20GB the prune policy deletes all the build cache until it +gets to the defaultKeepStorage value of 10GB. + +The following is an example of a defaultKeepStorage GC: + +```bash +"builder": { + "gc": { + "enabled": true, + "defaultKeepStorage": "10GB" + } + } +``` +The example above shows a GC configuration being enabled and the defaultKeepStorage being set to 10GB. + +## Garbage Collection Policy +Gabage collection works in an orderd list of prune operations, They are basically four default policy rules which can be configured +and also be added to or reduced to your preferred choice based on what can handle the build cache. Every policy as we said is a +prune operation with different categories of prune operation, which delete build cache on every stage based on different +conditions. +The reason there are multiple rules, is because the we users needs to be smart about which objects will be used more often in +future builds. + +Below is an illustration of how GC default policy works + +```bash +Step 1 ==> GC enabled +Step 2 ==> Policy rules : + Rule 0 ==> The first policy rule states that: if the build cache uses more than 512MB, delete the most + easily reproducible data after it has not been used for 2 days. + If the first rule is not enough to bring the cache down, it move to the next policy. + + Rule 1 ==> This rule only execute if the first rule is not enough to bring the cache down, + this default rule state that: remove any data not used for 60 days, if this rule + is not enough to bring the cache down, it jumps to the third rule, hence it + terminates. + + Rule 2 ==> The third rule only exexute if the first two rules are not enough to + bring down the build cache. This rule states that: it should keep + every remaining unshared build cache under cap i.e among the remaining + build cache, remove the ones that are unshared. If this rule is not + enough to bring the cache down, it jumps to the fourth rule, hence it terminates. + + Rule 3 ==> This rule only execute if the previous three rules are not enough to bring the down the build + cache. This rule states that: Start deleting internal data to keep build cache under cap. i.e + delete datas to make sure that the build cache is under the keep storage limit. +Step 3 ==> Keep Storage : 512MB +``` +**Note** +>All garbage collection properties are optional, so you can configure the gc to have upto 4 rules or not, also add filters. +>Some of the properties can be left out, maybe step 3 etc + +Below is the JSON configuration of the GC default policy: + +```bash +"gc": { + "enabled": true, + "policy": [ + {"keepStorage": "512MB", "filter": ["unused-for=48h"]}, + {"keepStorage": "512MB", "filter": {"unused-for": {"1440h": true}}}, # days converted to hours + {"keepStorage": "512MB"}, + {"keepStorage": "512MB", "all": true} + ] + "defaultKeepStorage": "512MB" + } +``` + +## more example of a random GC Config + +Now we are going to look at a new garbage collection config that is not the default policy. + +```bash +"gc": { + "enabled": true, + "policy": [ + {"keepStorage": "10GB", "filter": ["unused-for=2200h"]}, + {"keepStorage": "50GB", "filter": {"unused-for": {"3300h": true}}}, + {"keepStorage": "100GB", "all": true} + ] + } +``` +The configuration above shows that the garbage collection is on, and it follows three rules. +First rule state that, if the build cache is more than 10GB delete every unused build cache that are more than 92 days old +(converted to days), if the first rule is not enough to bring the cache down to 10GB it jumps to the next rule, stating that it +should remove every cache that are more than 136 days old, if the second rule is not enough to bring the cache down to 50GB, then +it would apply the third rule that state that, it should remove all the build cache data until it the keep storage reaches 100GB. +For every state once the condition is meant, it will terminate and not move to the other condition. + +Go on to configure your gabage collection for a better build. \ No newline at end of file From e586375e899fdd3bd86549b175c1dc4ea6d0c434 Mon Sep 17 00:00:00 2001 From: Adeniyi Stephen Oluwatola <41240528+adeniyistephen@users.noreply.github.com> Date: Sat, 28 Jan 2023 15:38:09 +0100 Subject: [PATCH 4/7] misspelled errors: incase ~ in case --- docs/reference/commandline/garbagecollection_config.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/garbagecollection_config.md b/docs/reference/commandline/garbagecollection_config.md index 073da7b4dbed..e528f7cdb9ed 100644 --- a/docs/reference/commandline/garbagecollection_config.md +++ b/docs/reference/commandline/garbagecollection_config.md @@ -17,7 +17,7 @@ The following enables the default GC on your docker daemon: } } ``` -The garbage collection config file can be created incase you dont have one located in your `etc/docker` folder, and if you do have +The garbage collection config file can be created in case you dont have one located in your `etc/docker` folder, and if you do have one it migth be there and be set to `false` so you need to enable the garbage collection by setting it as true, if you don't see any gc JSON, copy the above JSON and paste it in the file. That is the very first step to configure your garbage collection. Garbage collection config JSON are optional, i.e some fields can be left out, like the example above about enabling GC, the policy @@ -116,4 +116,4 @@ should remove every cache that are more than 136 days old, if the second rule is it would apply the third rule that state that, it should remove all the build cache data until it the keep storage reaches 100GB. For every state once the condition is meant, it will terminate and not move to the other condition. -Go on to configure your gabage collection for a better build. \ No newline at end of file +Go on to configure your gabage collection for a better build. From 11b616c8acb070af61208483b9c7f35b301748d5 Mon Sep 17 00:00:00 2001 From: adeniyistephen Date: Tue, 22 Apr 2025 18:30:21 +0100 Subject: [PATCH 5/7] correct spellings and some descriptions --- docs/reference/commandline/builder_prune.md | 4 +-- .../commandline/garbagecollection_config.md | 31 +++++++++++-------- 2 files changed, 20 insertions(+), 15 deletions(-) diff --git a/docs/reference/commandline/builder_prune.md b/docs/reference/commandline/builder_prune.md index ada536ec1ed2..98484e6bcb84 100644 --- a/docs/reference/commandline/builder_prune.md +++ b/docs/reference/commandline/builder_prune.md @@ -50,7 +50,7 @@ effective. So docker try to remove the cache that is less likely to be used in future builds to keep builds as fast as possible while maintaining cache size under control. **Note** ->Pruning does not works if the Total is below what you specified as keep storage option to prune. It needs to go above, we are +>Pruning does not work if the Total is below what you specified as keep storage option to prune. It needs to go above, we are >going to see how to use the `--keep-storage` option as part of our examples in this topic. # builder prune @@ -277,7 +277,7 @@ rx58kxz9kmlg regular 540MB 22 hours ago 08unho1rnguz regular 2.38MB 13 days ago 13 days ago 1 false i8nfk2fvpyyi regular 0B 2 weeks ago 13 days ago 2 false -$ docker builder prune --filter until=24h # to remove builder cache with a filter or 24 hours +$ docker builder prune --filter until=24h # to remove builder cache with a filter of 24 hours WARNING! This will remove all dangling build cache. Are you sure you want to continue? [y/N] y diff --git a/docs/reference/commandline/garbagecollection_config.md b/docs/reference/commandline/garbagecollection_config.md index e528f7cdb9ed..5b9429b6543c 100644 --- a/docs/reference/commandline/garbagecollection_config.md +++ b/docs/reference/commandline/garbagecollection_config.md @@ -101,19 +101,24 @@ Now we are going to look at a new garbage collection config that is not the defa ```bash "gc": { - "enabled": true, - "policy": [ - {"keepStorage": "10GB", "filter": ["unused-for=2200h"]}, - {"keepStorage": "50GB", "filter": {"unused-for": {"3300h": true}}}, - {"keepStorage": "100GB", "all": true} - ] - } + "enabled": true, + "policy": [ + {"keepStorage": "10GB", "filter": ["unused-for=2200h"]}, + {"keepStorage": "50GB", "filter": {"unused-for": {"3300h": true}}}, + {"keepStorage": "100GB", "all": true} + ] +} ``` -The configuration above shows that the garbage collection is on, and it follows three rules. -First rule state that, if the build cache is more than 10GB delete every unused build cache that are more than 92 days old -(converted to days), if the first rule is not enough to bring the cache down to 10GB it jumps to the next rule, stating that it -should remove every cache that are more than 136 days old, if the second rule is not enough to bring the cache down to 50GB, then -it would apply the third rule that state that, it should remove all the build cache data until it the keep storage reaches 100GB. -For every state once the condition is meant, it will terminate and not move to the other condition. +The configuration above enables Docker CLI garbage collection and defines a tiered cleanup policy with three rules: + +1. First rule: If the total build cache exceeds 10GB, delete any unused build cache entries that are older than 2200 hours (~92 days). + +2. Second rule: If the first rule does not reduce the cache size below 10GB, the system moves on. Now, if the total cache size exceeds 50GB, it deletes entries that are unused and older than 3300 hours (~137 days). + +3. Third rule: If the cache is still larger than 50GB, the last rule applies. It deletes any build cache (regardless of usage or age) until the cache is brought below 100GB. + +Each rule is evaluated in order, and once a rule successfully reduces the cache size to the defined keepStorage threshold, garbage collection stops and the following rules are not evaluated. + +This configuration ensures the system aggressively prunes older and unused cache layers first, and only performs more general deletions if absolutely necessary. Go on to configure your gabage collection for a better build. From c1a1bb2aec18c4ff047de6919aefced22e48f9c4 Mon Sep 17 00:00:00 2001 From: adeniyistephen Date: Tue, 22 Apr 2025 18:38:30 +0100 Subject: [PATCH 6/7] correct spellings and some descriptions --- .../commandline/garbagecollection_config.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/reference/commandline/garbagecollection_config.md b/docs/reference/commandline/garbagecollection_config.md index 5b9429b6543c..f2404b1eb4d2 100644 --- a/docs/reference/commandline/garbagecollection_config.md +++ b/docs/reference/commandline/garbagecollection_config.md @@ -122,3 +122,20 @@ Each rule is evaluated in order, and once a rule successfully reduces the cache This configuration ensures the system aggressively prunes older and unused cache layers first, and only performs more general deletions if absolutely necessary. Go on to configure your gabage collection for a better build. + +### builder prune + + +Remove build cache + +### Options + +| Name | Type | Default | Description | +|:-----------------|:---------|:--------|:------------------------------------------------------| +| `-a`, `--all` | `bool` | | Remove all unused build cache, not just dangling ones | +| `--filter` | `filter` | | Provide filter values (e.g. `until=24h`) | +| `-f`, `--force` | `bool` | | Do not prompt for confirmation | +| `--keep-storage` | `bytes` | `0` | Amount of disk space to keep for cache | + + + From c0394a1bd32b55b83fb16b5726c0458c53800257 Mon Sep 17 00:00:00 2001 From: adeniyistephen Date: Tue, 22 Apr 2025 19:05:48 +0100 Subject: [PATCH 7/7] correct spellings and some descriptions --- docs/reference/commandline/builder_prune.md | 13 ------------- .../commandline/garbagecollection_config.md | 17 ----------------- 2 files changed, 30 deletions(-) diff --git a/docs/reference/commandline/builder_prune.md b/docs/reference/commandline/builder_prune.md index 98484e6bcb84..4dd90ffec2a9 100644 --- a/docs/reference/commandline/builder_prune.md +++ b/docs/reference/commandline/builder_prune.md @@ -53,19 +53,6 @@ So docker try to remove the cache that is less likely to be used in future build >Pruning does not work if the Total is below what you specified as keep storage option to prune. It needs to go above, we are >going to see how to use the `--keep-storage` option as part of our examples in this topic. -# builder prune - -```markdown -Usage: docker builder prune [OPTIONS] - -Remove build cache - -Options: - -a, --all Remove all unused build cache, not just dangling ones - --filter filter Provide filter values (e.g. 'until=24h') - -f, --force Do not prompt for confirmation - --keep-storage bytes Amount of disk space to keep for cache -``` # Usage diff --git a/docs/reference/commandline/garbagecollection_config.md b/docs/reference/commandline/garbagecollection_config.md index f2404b1eb4d2..5b9429b6543c 100644 --- a/docs/reference/commandline/garbagecollection_config.md +++ b/docs/reference/commandline/garbagecollection_config.md @@ -122,20 +122,3 @@ Each rule is evaluated in order, and once a rule successfully reduces the cache This configuration ensures the system aggressively prunes older and unused cache layers first, and only performs more general deletions if absolutely necessary. Go on to configure your gabage collection for a better build. - -### builder prune - - -Remove build cache - -### Options - -| Name | Type | Default | Description | -|:-----------------|:---------|:--------|:------------------------------------------------------| -| `-a`, `--all` | `bool` | | Remove all unused build cache, not just dangling ones | -| `--filter` | `filter` | | Provide filter values (e.g. `until=24h`) | -| `-f`, `--force` | `bool` | | Do not prompt for confirmation | -| `--keep-storage` | `bytes` | `0` | Amount of disk space to keep for cache | - - -