Pretty big cache

[MarcelCoding]

Description

The cache can grow very quickly with large images, since old entries are not deleted.

Configuration

• Repository URL (if public): https://github.com/MarcelCoding/docker-desktop-vnc/blob/02fb4a97d5a75e40c467b856538efa443d1077cc/.github/workflows/docker-publish.yml#L28-L60
• Build URL (if public): https://github.com/MarcelCoding/docker-desktop-vnc/runs/1549446282?check_suite_focus=true#logs

[...]

      - name: Cache Docker layers
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-

[...]

      - name: Build
        uses: docker/build-push-action@v2
        with:
          push: false
          tags: ${{ steps.prepare.outputs.image }}
          platforms: ${{ env.DOCKER_PLATFORMS }}
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache
          context: .

[...]

Logs

logs_37.zip

My solution

Add a clean-cache configuration option that runs the following command before exporting the layers docker system prune -f --filter "until=5h".

[crazy-max]

@MarcelCoding

The cache can grow very quickly with large images, since old entries are not deleted.

Yes you're right atm caches are copied over the existing cache so it keeps growing. Can you open an issue on buildkit repo about that please? In the meantime you can do this:

[...]

      - name: Cache Docker layers
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-

[...]

      - name: Build
        uses: docker/build-push-action@v2
        with:
          push: false
          tags: ${{ steps.prepare.outputs.image }}
          platforms: ${{ env.DOCKER_PLATFORMS }}
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new
          context: .

[...]

      - name: Move cache
        run:
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

cc. @tonistiigi

[MarcelCoding]

@crazy-max that means that the cache-to option does not export the entire Docker cache, just the one for the current images?

[crazy-max]

@MarcelCoding

that means that the cache-to option does not export the entire Docker cache, just the one for the current images?

cache-to exports the build cache for the current image being built with buildkit yes. If you are implying that the cache can be shared between different images, this is not the case. See #153 (comment).

[MarcelCoding]

Ok, thanks for the help, I will create an issue in the builtkit repo. For implementing an option to clean the cache/remove old versions of the cache.

[zero88]

Hi @crazy-max
Can I use buildkit registry cache within local registry by GitHub service container, then caching registry volume as GitHub cache.
After each run, just prune cache in local registry to keep as small as possible.

However, I'm not sure about GitHub service container is able to create volume for service container and your GitHub action support that (because I'm only see caching inline option)

[malobre]

docker/buildx#535 should fix this and make using github cache a breeze:

[...]

      - name: Build
        uses: docker/build-push-action@v2
        with:
          tags: user/app:latest
          cache-from: type=gha
          cache-to: type=gha

[zero88]

I used it for caching
https://github.com/zero88/gh-registry/blob/main/README.md#usage

[rafis]

@malobre how to fix this?

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@master
        with:
          install: true

moby/buildkit:buildx-stable-1 => buildkitd github.com/moby/buildkit v0.8.3 81c2cbd8a418918d62b71e347a00034189eea455

error: failed to solve: rpc error: code = Unknown desc = unknown cache exporter: "gha"
102
Error: buildx call failed with: error: failed to solve: rpc error: code = Unknown desc = unknown cache exporter: "gha"

[malobre]

The server side is not implemented yet: moby/buildkit#1974.

[jauderho]

FWIW, there's a syntax error in the code block below. There is a missing "|" that took me a while to find and correct.

This should be the correct invocation:

  - name: Move cache
    run: |
      rm -rf /tmp/.buildx-cache
      mv /tmp/.buildx-cache-new /tmp/.buildx-cache

@MarcelCoding

The cache can grow very quickly with large images, since old entries are not deleted.

Yes you're right atm caches are copied over the existing cache so it keeps growing. Can you open an issue on buildkit repo about that please? In the meantime you can do this:

[...]

      - name: Move cache
        run:
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

cc. @tonistiigi

[crazy-max]

@jauderho https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md#github-cache

[ahgraber]

Given that moby/buildkit#1974 was just merged, what does the timeline look like to use cache type=gha in build-push-action?

[crazy-max]

You can test the gha cache exporter using this workflow while waiting for buildx 0.6 and BuildKit 0.9 to be GA. Feel free to give us your feedback, thanks!

[jauderho]

I just tried this out on one of my workflows. See https://github.com/jauderho/dockerfiles/actions/workflows/cloudflared.yml

Looks like the "Setup Buildx" step now takes longer but I'm assuming that's due to the rest not yet being merged in. More importantly, the "Build and push" step looks to be much faster.

Nice job!

[crazy-max]

@jauderho

Looks like the "Setup Buildx" step now takes longer but I'm assuming that's due to the rest not yet being merged in. More importantly, the "Build and push" step looks to be much faster.

Yes that's it, buildx is built on-fly atm, that's why it takes more time for the setup step.

[jauderho]

Awesome. Looking forward to everything being merged in.

[crazy-max]

@jauderho

Looks like the "Setup Buildx" step now takes longer

Buildx 0.6.0-rc1 has been released. I've updated the workflow to use it so now it should be faster than building from source.

[jauderho]

Hmm, not sure if I am doing something wrong here but after updating to call buildx 0.6.0-rc1, it does not seem to trigger the caching.

Here is my action: https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/cloudflared.yml

With buildx 0.6.0-rc1

• First build (40m, uncached but expected) https://github.com/jauderho/dockerfiles/actions/runs/1034517548
• Second build (should show as cached in logs but looks like it is rebuilding from scratch) https://github.com/jauderho/dockerfiles/actions/runs/1034655356

Compare this to 2 days ago (which has the expected behavior)

• First build (34m, uncached but expected) https://github.com/jauderho/dockerfiles/actions/runs/1027888442
• Second build (2m, cached) https://github.com/jauderho/dockerfiles/actions/runs/1027997222

[crazy-max]

@jauderho

Use image=moby/buildkit:master instead of image=moby/buildkit:v0.9.0-rc1.

[jauderho]

@crazy-max

Per your suggestion, updated to use image=moby/buildkit:master but it does not appear to make a difference.

• First build (31m) https://github.com/jauderho/dockerfiles/actions/runs/1035009487
• Second build (33m not single digit mins) https://github.com/jauderho/dockerfiles/actions/runs/1035310379

[crazy-max]

@jauderho

As you're on a monorepo building multi Docker images with different context, you should use a specific scope for each one of them to avoid cache collision/invalidation. For example in your cloudflared workflow:

cache-from: type=gha,scope=cloudflared
cache-to: type=gha,scope=cloudflared

[MichaelVoelkel]

#756 mentions a comment where the "old" way is used. I switched to cache-from/cache-to:gha BUT if the whole job fails (or the build is not entirely successful), the cache does not seem to be used.

In that regard, it seems still suboptimal or am I potentially doing anything wrong?

[Laubeee]

@MarcelCoding

The cache can grow very quickly with large images, since old entries are not deleted.

Yes you're right atm caches are copied over the existing cache so it keeps growing. Can you open an issue on buildkit repo about that please? In the meantime you can do this:

[...]

  - name: Cache Docker layers
    uses: actions/cache@v2
    with:
      path: /tmp/.buildx-cache
      key: ${{ runner.os }}-buildx-${{ github.sha }}
      restore-keys: |
        ${{ runner.os }}-buildx-

[...]

  - name: Build
    uses: docker/build-push-action@v2
    with:
      push: false
      tags: ${{ steps.prepare.outputs.image }}
      platforms: ${{ env.DOCKER_PLATFORMS }}
      cache-from: type=local,src=/tmp/.buildx-cache
      cache-to: type=local,dest=/tmp/.buildx-cache-new
      context: .

[...]

  - name: Move cache
    run:
      rm -rf /tmp/.buildx-cache
      mv /tmp/.buildx-cache-new /tmp/.buildx-cache

cc. @tonistiigi

If the cache hits then it won't write a new entry and then overwrites the existing cache with an empty dir. Adding an if statement checking the new directory has contents may help with this.