Multi-stage build doesn't reuse previously made local cache from Github Cache Action

[lucernae]

Troubleshooting

Before sumbitting a bug report please read the Troubleshooting doc.

Behaviour

I am using Multi-stage Dockerfile.
There are 3 stages, base, prod and test.
I'm also using Github cache action.
In the workflow, I build the target base, then prod, then test.
This is because I want to run the test first then store local cache.
When the tests finished, I want to build base and prod image with push: true to the registry, reusing the previous cache.
I tried to control the caching. I put .dockerignore in such a way it ignores everything else except the Dockerfile. This is combined with cache key using build-${{ hashFiles('Dockerfile') }} in order for it to ensure a cache hit.

I'm hoping that with this usage, whenever I change unittests, or github workflow files, the builder will reuse existing cache because obviously the image itself should not change.

However, what happens is:

1. The cache hits (cache key works)
2. Base image build uses the cache (from previous build)
3. Prod image target doesn't use the cache (even though it's the same thing, and reuse previous base image cache)
4. Test image target doesn't use the cache, but this time, I assume because the prod image (the base for test image) was rebuilt.

Even without changing any files and rerun the workflow. This always happens.

Steps to reproduce this issue

1. Create simple Dockerfile with 3 stages, each using previous stage:
2. Make sure .dockerignore filters out all irrelevant files with the build. Ensuring exact cache hit
3. My workflow, using github action cache, and 3 stage build with each stage as separate steps
4. Run the workflow twice at minimum to check if build uses previous cache

Expected behaviour

Second workflow run should build all local images using the cache entirely, because no files are changed.
Workflow run after new commit should use cache entirely if build input doesn't change (files included in the docker context).

Actual behaviour

Base stage uses cache, but prod stage doesn't use prod cache, however it still uses base stage cache.
Test stage uses prod cache, but doesn't use test stage cache.

Configuration

• Repository URL (if public): https://github.com/lucernae/docker-build-cache-action-test
• Build URL (if public): https://github.com/lucernae/docker-build-cache-action-test/actions/runs/523080913

name: build-latest
on:
  push:
jobs:
  build-image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      - name: Get build cache
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-cache
          key: buildx-${{ hashFiles('Dockerfile') }}
          restore-keys: |
            buildx-

      - name: Build base image
        id: docker_build_base
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,mode=max,dest=/tmp/.buildx-cache
          target: base

      - name: Build prod image
        id: docker_build_prod
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,mode=max,dest=/tmp/.buildx-cache
          target: prod

      - name: Build image for testing
        id: docker_build_testing_image
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,mode=max,dest=/tmp/.buildx-cache
          target: test

Logs

logs_17.zip

[lucernae]

I also create an alternative use case, just to check if I use the action correctly.
This is on a separate branch:

Behaviour

Same idea. I want to reuse the local build cache. However this time I build all the stages in one action. Then build again separate stage (base, prod, and test) in the hope of doing a local retag to be used for docker testing and at the same time reuse existing cache.

Steps to reproduce this issue

1. Create simple Dockerfile with 3 stages, each using previous stage (same as before).
2. Make sure .dockerignore filters out all irrelevant files with the build. Ensuring exact cache hit
3. My workflow, using github action cache. First, it builds all stages. Then build each stage as separate steps.
4. Run the workflow twice at minimum to check if build uses previous cache

Expected behaviour

Second workflow run would reuse all cache from previous build. Each build stage would reuse these caches.

Actual behaviour

Interestingly for me. The step for building all stages reuse all the cache for all stage.
Base stage build in the next step (same run) reuse cache.
Prod stage build in the next step (same run) didn't use the cache. Even though in the all stage build it use the cache?
Test stage also didn't use the cache, even if the all stage build uses the cache.

Configuration

• Repository URL (if public): https://github.com/lucernae/docker-build-cache-action-test/tree/build-all-stages
• Build URL (if public): https://github.com/lucernae/docker-build-cache-action-test/actions/runs/523085312

name: build-latest
on:
  push:
jobs:
  build-image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      - name: Get build cache
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-cache
          key: buildx-${{ hashFiles('Dockerfile') }}
          restore-keys: |
            buildx-

      - name: Build all stages
        id: docker_build_all
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,mode=max,dest=/tmp/.buildx-cache

      - name: Build base image
        id: docker_build_base
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,mode=max,dest=/tmp/.buildx-cache
          target: base

      - name: Build prod image
        id: docker_build_prod
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,mode=max,dest=/tmp/.buildx-cache
          target: prod

      - name: Build image for testing
        id: docker_build_testing_image
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,mode=max,dest=/tmp/.buildx-cache
          target: test

Logs

logs_19.zip

[lucernae]

I'm fine if the accepted use case is to actually build all stages to reuse the cache.
But I want to know how to use that to load separate stage into docker to run tests in the next job within the same workflow run (cache are passed via Github Cache action).

[hiaselhans]

I encountered the same problem even without cache-from and cache-to.

I think it relates to context: .
Without local context cached layers were working better.

[jfabre]

@lucernae Any news on this? I'm in the same boat.

[gilesknap]

I'm experiencing the same problem here https://github.com/epics-containers/epics-modules/blob/main/.github/workflows/buiild.yml

[lucernae]

@jfabre I don't know man. I just decided that it's beyond my power and knowledge.
If you think about it, we are trying to save the planet by making sure that the cache hits so that no computes is wasted on docker build. Well, at least the action runs using Github's money not mine 😄 .
This is also an official docker repo, so I don't know if this repo is company managed or community managed, or does it need funding? The maintainer team doesn't seem to reply yet.

[crazy-max]

@lucernae Sorry for the delay. There might be some cache invalidation with actions/cache (GC) but I think you're hitting the following behavior where BuildKit will only build stages that are needed for the final target for your Build all stages and not all stages as you may think (see #377 (comment) about the detailed explanations).

So smth like this might be better in your case:

name: build-latest
on:
  push:
jobs:
  build-image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      
      - name: Get build cache all
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-all-cache
          key: buildx-all-${{ hashFiles('Dockerfile') }}
          restore-keys: |
            buildx-all-

      - name: Get build cache base
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-base-cache
          key: buildx-base-${{ hashFiles('Dockerfile') }}
          restore-keys: |
            buildx-base-

      - name: Get build cache prod
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-prod-cache
          key: buildx-prod-${{ hashFiles('Dockerfile') }}
          restore-keys: |
            buildx-prod-

      - name: Get build cache test
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-test-cache
          key: buildx-test-${{ hashFiles('Dockerfile') }}
          restore-keys: |
            buildx-test-

      - name: Build all stages
        id: docker_build_all
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=local,src=/tmp/.buildx-all-cache
          cache-to: |
            type=local,mode=max,dest=/tmp/.buildx-all-cache-new

      - name: Build base image
        id: docker_build_base
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=local,src=/tmp/.buildx-all-cache
            type=local,src=/tmp/.buildx-base-cache
          cache-to: |
            type=local,mode=max,dest=/tmp/.buildx-base-cache-new
          target: base

      - name: Build prod image
        id: docker_build_prod
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=local,src=/tmp/.buildx-all-cache
            type=local,src=/tmp/.buildx-base-cache
            type=local,src=/tmp/.buildx-prod-cache
          cache-to: |
            type=local,mode=max,dest=/tmp/.buildx-prod-cache-new
          target: prod

      - name: Build image for testing
        id: docker_build_testing_image
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=local,src=/tmp/.buildx-all-cache
            type=local,src=/tmp/.buildx-base-cache
            type=local,src=/tmp/.buildx-prod-cache
            type=local,src=/tmp/.buildx-test-cache
          cache-to: |
            type=local,mode=max,dest=/tmp/.buildx-prod-cache-new
          target: test
      -
        # Temp fix
        # https://github.com/docker/build-push-action/issues/252
        # https://github.com/moby/buildkit/issues/1896
        name: Move cache
        run: |
          rm -rf /tmp/.buildx-all-cache /tmp/.buildx-base-cache /tmp/.buildx-prod-cache /tmp/.buildx-test-cache
          mv /tmp/.buildx-all-cache-new /tmp/.buildx-all-cache
          mv /tmp/.buildx-base-cache-new /tmp/.buildx-base-cache
          mv /tmp/.buildx-prod-cache-new /tmp/.buildx-prod-cache
          mv /tmp/.buildx-test-cache-new /tmp/.buildx-test-cache

You can also try with the new GitHub Action cache backend:

name: build-latest
on:
  push:
jobs:
  build-image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      
      - name: Build base image
        id: docker_build_base
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=gha,scope=base
          cache-to: |
            type=gha,scope=base,mode=max
          target: base

      - name: Build prod image
        id: docker_build_prod
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=gha,scope=prod
            type=gha,scope=base
          cache-to: |
            type=gha,scope=prod,mode=max
          target: prod

      - name: Build image for testing
        id: docker_build_testing_image
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: false
          load: false
          cache-from: |
            type=gha,scope=test
            type=gha,scope=prod
            type=gha,scope=base
          cache-to: |
            type=gha,scope=test,mode=max
          target: test

[lucernae]

Thank you @crazy-max !

In my Build all stages example, my final target is test and it was supposed to build both base and prod as it's stage dependency. I was under the impression if the stage dependency is like this: base -> prod -> test , then if I specify the target test using buildx, the cache will contain both base and prod stages. Is my assumption correct?

From the example you provide above, my use case fit the second yaml file using the new GH cache backend, so I will try that first. It's also nice because it's less verbose.

[crazy-max]

@lucernae

In my Build all stages example, my final target is test and it was supposed to build both base and prod as it's stage dependency.

Ah yes indeed I didn't look at the right Dockerfile...

I was under the impression if the stage dependency is like this: base -> prod -> test , then if I specify the target test using buildx, the cache will contain both base and prod stages. Is my assumption correct?

Yes that's correct

From the example you provide above, my use case fit the second yaml file using the new GH cache backend, so I will try that first. It's also nice because it's less verbose.

Yes try it. I will also repro your use case and keep you in touch.

[crazy-max]

@jfabre Ok was able to reproduce your initial use case and from what I see cache is invalidated because you're using the same cache folder for different targets which overrides the cache index (pretty much the same thing as #153 (comment)). One of the solutions in my comment above should fix your issue.

[lucernae]

Thanks @crazy-max and sorry to necro the thread a little bit.

The GH cache backend works nicely. For others having the same issue, the key solution is to provide a different cache-to location for each different target. If you want to reuse any target, include the possible caches in cache-from (can be multiple lines). The problem is caused by cache invalidation if multiple target uses the same cache-to location.

I've made an example action that build all the stages and then reuse it in the same run:
https://github.com/lucernae/docker-build-cache-action-test/blob/gh-action-cache-backend/.github/workflows/build-load-test-push.yaml
It behaves like this:

• In the same (initial run), different stage caches can be reused (prod), because the test stage also build prod stage
• Second workflow run also reuse the cache, so all the stages uses caches. Very useful if you didn't change the build. But change other files in repo, like a deployment scripts.

Again, thanks for the quick response.

[huksley]

Effectively it means multi-stage builds are broken (it requires external configuration so it makes it useless, similar to running X docker buildx build commands)

Is there any roadmap to fixing this cache issue? type=local, type=gha and type=registry,ref=ghcr.io are all broken as they do not cache properly previous stage in GitHub Actions.

[trondhindenes]

we're struggling with this too. I can't get over how hard it is to do something as mundane as building a docker image in the most efficient way using github actions. This shouldn't be this hard.