Support pull-through caches

[wichert]

Troubleshooting

Before sumbitting a bug report please read the Troubleshooting doc.

Behaviour

Steps to reproduce this issue

1. Setup a GitHub Action workflow using docker/build-action
2. Observe hitting a pull rate limit

Expected behaviour

Be able to use an existing Artifactory instance as a pull-through cache.

Actual behaviour

No cache ability, causing us to hit rate limits.

jobs:
  python:
    runs-on: self-hosted

    name: Build Python ${{ matrix.python }}

    strategy:
      matrix:
        python: ["3.8", "3.9"]

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Set up Docker Buildx
        uses: synced-actions/docker-setup-buildx-action@v1

      - name: Login to Artifactory
        uses: synced-actions/docker-login-action@v1
        with:
          registry: docker.artifactory.tri-ad.tech
          username: ${{ secrets.ARTIFACTORY_USER }}
          password: ${{ secrets.ARTIFACTORY_API_KEY }}

      - name: Build and push
        uses: synced-actions/docker-build-push-action@v2
        with:
          cache-from: type=registry,ref=user/app:latest
          push: true
          tags: repository/python-ci:${{ matrix.python }}
          context: ./python-ci
          file: ./python-ci/Dockerfile
          build-args: |
            PYTHON_VERSION=${{ matrix.python }}

Please note that synced-actions/docker-login-action and synced-actions/docker-build-push-action are synced copies of the official actions, running on a GHE instance.

[crazy-max]

@wichert

Observe hitting a pull rate limit

Can you post logs please?

running on a GHE instance.

If you are on GHE I assume you're running a self-hosted GitHub Runner. If that's the case you can configure docker.artifactory.tri-ad.tech (assuming it's configured on Artifactory as a remote repository to Docker Hub) to run as a pull through cache through the Docker daemon:

{
  "registry-mirrors": ["https://docker.artifactory.tri-ad.tech"]
}

Or to increase your pull rate limits you can upgrade your account to a Docker Pro or Team subscription and add another Login step in your workflow to connect to Docker Hub:

jobs:
  python:
    runs-on: self-hosted

    name: Build Python ${{ matrix.python }}

    strategy:
      matrix:
        python: ["3.8", "3.9"]

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Login to Docker Hub
        uses: synced-actions/docker-login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USER }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Set up Docker Buildx
        uses: synced-actions/docker-setup-buildx-action@v1

      - name: Login to Artifactory
        uses: synced-actions/docker-login-action@v1
        with:
          registry: docker.artifactory.tri-ad.tech
          username: ${{ secrets.ARTIFACTORY_USER }}
          password: ${{ secrets.ARTIFACTORY_API_KEY }}

      - name: Build and push
        uses: synced-actions/docker-build-push-action@v2
        with:
          cache-from: type=registry,ref=user/app:latest
          push: true
          tags: repository/python-ci:${{ matrix.python }}
          context: ./python-ci
          file: ./python-ci/Dockerfile
          build-args: |
            PYTHON_VERSION=${{ matrix.python }}

[wichert]

2021-01-23T22:08:05.5360930Z ##[section]Starting: Request a runner to run this job
2021-01-23T22:08:05.6563283Z Can't find any online and idle self-hosted runner in current repository that matches the required labels: 'self-hosted'
2021-01-23T22:08:05.7360724Z Found online and idle self-hosted runner in current repository's organization/enterprise account that matches the required labels: 'self-hosted'
2021-01-23T22:08:06.0353674Z ##[section]Finishing: Request a runner to run this job
2021-01-23T22:08:10.1060530Z Current runner version: '2.275.1'
2021-01-23T22:08:10.1063379Z Runner name: 'prod-swp-ie-ghe-runner-H3OgmFWwGOnKg'
2021-01-23T22:08:10.1064173Z Machine name: 'ip-172-24-9-242'
2021-01-23T22:08:10.1067258Z Prepare workflow directory
2021-01-23T22:08:10.1613455Z Prepare all required actions
2021-01-23T22:08:10.1660097Z Getting action download info
2021-01-23T22:08:10.5489373Z Download action repository 'actions/checkout@v2'
2021-01-23T22:08:10.8905561Z Download action repository 'synced-actions/docker-setup-buildx-action@v1'
2021-01-23T22:08:11.0234830Z Download action repository 'synced-actions/docker-login-action@v1'
2021-01-23T22:08:11.1466406Z Download action repository 'synced-actions/docker-build-push-action@v2'
2021-01-23T22:08:11.4384827Z ##[group]Run actions/checkout@v2
2021-01-23T22:08:11.4385122Z with:
2021-01-23T22:08:11.4385353Z   repository: amp-portal/docker
2021-01-23T22:08:11.4385647Z   token: ***
2021-01-23T22:08:11.4385798Z   ssh-strict: true
2021-01-23T22:08:11.4385931Z   persist-credentials: true
2021-01-23T22:08:11.4386083Z   clean: true
2021-01-23T22:08:11.4386221Z   fetch-depth: 1
2021-01-23T22:08:11.4386346Z   lfs: false
2021-01-23T22:08:11.4386472Z   submodules: false
2021-01-23T22:08:11.4386599Z ##[endgroup]
2021-01-23T22:08:11.5427311Z Syncing repository: amp-portal/docker
2021-01-23T22:08:11.5428113Z ##[group]Getting Git version info
2021-01-23T22:08:11.5428976Z Working directory is '/home/github-runner/runner/_work/docker/docker'
2021-01-23T22:08:11.5429602Z [command]/usr/bin/git version
2021-01-23T22:08:11.5462127Z git version 2.17.1
2021-01-23T22:08:11.5479374Z ##[endgroup]
2021-01-23T22:08:11.5482667Z Deleting the contents of '/home/github-runner/runner/_work/docker/docker'
2021-01-23T22:08:11.5484854Z The repository will be downloaded using the GitHub REST API
2021-01-23T22:08:11.5485199Z To create a local Git repository instead, add Git 2.18 or higher to the PATH
2021-01-23T22:08:11.5489187Z Downloading the archive
2021-01-23T22:08:11.7291324Z Writing archive to disk
2021-01-23T22:08:11.7301629Z Extracting the archive
2021-01-23T22:08:11.7314426Z [command]/bin/tar xz -C /home/github-runner/runner/_work/docker/docker/f22de485-fa51-41c5-a2c3-3ef623590fae -f /home/github-runner/runner/_work/docker/docker/f22de485-fa51-41c5-a2c3-3ef623590fae.tar.gz
2021-01-23T22:08:11.7362233Z Resolved version amp-portal-docker-eb786099653009366ec3830096bbd5f06e250e9f
2021-01-23T22:08:11.7492508Z ##[group]Run synced-actions/docker-setup-buildx-action@v1
2021-01-23T22:08:11.7492682Z with:
2021-01-23T22:08:11.7492823Z   driver: docker-container
2021-01-23T22:08:11.7493010Z   buildkitd-flags: --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host
2021-01-23T22:08:11.7493181Z   install: false
2021-01-23T22:08:11.7493321Z   use: true
2021-01-23T22:08:11.7493454Z ##[endgroup]
2021-01-23T22:08:11.9561766Z 📣 Buildx version: 0.5.1
2021-01-23T22:08:11.9566891Z ##[group]🔨 Creating a new builder instance
2021-01-23T22:08:11.9576131Z [command]/usr/bin/docker buildx create --name builder-f8ce65ae-7c82-4109-ad95-b5cc86f6c1e8 --driver docker-container --buildkitd-flags --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host --use
2021-01-23T22:08:12.7459392Z builder-f8ce65ae-7c82-4109-ad95-b5cc86f6c1e8
2021-01-23T22:08:12.7459915Z ##[endgroup]
2021-01-23T22:08:12.7460691Z ##[group]🏃 Booting builder
2021-01-23T22:08:12.7461312Z [command]/usr/bin/docker buildx inspect --bootstrap --builder builder-f8ce65ae-7c82-4109-ad95-b5cc86f6c1e8
2021-01-23T22:08:12.7461602Z #1 [internal] booting buildkit
2021-01-23T22:08:12.7461802Z #1 sha256:6e4683524e4733c8e5940a3fbc889720c2e5716fe6fa79711bd2a68eb319ee73
2021-01-23T22:08:12.7462225Z #1 pulling image moby/buildkit:buildx-stable-1
2021-01-23T22:08:22.5906962Z #1 pulling image moby/buildkit:buildx-stable-1 10.5s done
2021-01-23T22:08:22.7399075Z #1 creating container buildx_buildkit_builder-f8ce65ae-7c82-4109-ad95-b5cc86f6c1e80
2021-01-23T22:08:23.2279528Z #1 creating container buildx_buildkit_builder-f8ce65ae-7c82-4109-ad95-b5cc86f6c1e80 0.6s done
2021-01-23T22:08:23.2281109Z #1 DONE 11.1s
2021-01-23T22:08:23.3177849Z Name:   builder-f8ce65ae-7c82-4109-ad95-b5cc86f6c1e8
2021-01-23T22:08:23.3179077Z Driver: docker-container
2021-01-23T22:08:23.3179498Z 
2021-01-23T22:08:23.3180033Z Nodes:
2021-01-23T22:08:23.3192383Z Name:      builder-f8ce65ae-7c82-4109-ad95-b5cc86f6c1e80
2021-01-23T22:08:23.3197004Z Endpoint:  unix:///var/run/docker.sock
2021-01-23T22:08:23.3199241Z Status:    running
2021-01-23T22:08:23.3202244Z Flags:     --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host
2021-01-23T22:08:23.3206025Z Platforms: linux/amd64, linux/386
2021-01-23T22:08:23.3207642Z ##[endgroup]
2021-01-23T22:08:23.3209242Z ##[group]🛒 Extracting available platforms
2021-01-23T22:08:23.4396230Z linux/amd64,linux/386
2021-01-23T22:08:23.4401021Z ##[endgroup]
2021-01-23T22:08:23.4577651Z ##[group]Run synced-actions/docker-login-action@v1
2021-01-23T22:08:23.4577918Z with:
2021-01-23T22:08:23.4578058Z   registry: docker.artifactory.tri-ad.tech
2021-01-23T22:08:23.4578609Z   username: ***
2021-01-23T22:08:23.4578791Z   password: ***
2021-01-23T22:08:23.4578914Z   logout: true
2021-01-23T22:08:23.4579040Z ##[endgroup]
2021-01-23T22:08:23.4950736Z 🔑 Logging into docker.artifactory.tri-ad.tech...
2021-01-23T22:08:23.5731573Z 🎉 Login Succeeded!
2021-01-23T22:08:23.5808452Z ##[group]Run synced-actions/docker-build-push-action@v2
2021-01-23T22:08:23.5808653Z with:
2021-01-23T22:08:23.5808812Z   push: false
2021-01-23T22:08:23.5808988Z   tags: docker.artifactory.tri-ad.tech/amp-portal/sdk:latest
2021-01-23T22:08:23.5809212Z   context: ./sdk
2021-01-23T22:08:23.5809386Z   file: ./sdk/Dockerfile
2021-01-23T22:08:23.5809570Z   pull: false
2021-01-23T22:08:23.5809732Z   no-cache: false
2021-01-23T22:08:23.5809889Z   load: false
2021-01-23T22:08:23.5810209Z   github-token: ***
2021-01-23T22:08:23.5810419Z ##[endgroup]
2021-01-23T22:08:23.7833306Z 📣 Buildx version: 0.5.1
2021-01-23T22:08:23.7865787Z 🏃 Starting build...
2021-01-23T22:08:23.7881460Z [command]/usr/bin/docker buildx build --tag docker.artifactory.tri-ad.tech/amp-portal/sdk:latest --iidfile /tmp/docker-build-push-AFVHbW/iidfile --file ./sdk/Dockerfile ./sdk
2021-01-23T22:08:23.8573921Z time="2021-01-23T22:08:23Z" level=warning msg="No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load"
2021-01-23T22:08:23.9543340Z #1 [internal] load build definition from Dockerfile
2021-01-23T22:08:23.9543986Z #1 sha256:44aa4c1c5fcb879319d55b05da6f9654b928b7b925646cef04592b866da22bf0
2021-01-23T22:08:23.9545243Z #1 DONE 0.0s
2021-01-23T22:08:24.1054329Z 
2021-01-23T22:08:24.1054788Z #1 [internal] load build definition from Dockerfile
2021-01-23T22:08:24.1055118Z #1 sha256:44aa4c1c5fcb879319d55b05da6f9654b928b7b925646cef04592b866da22bf0
2021-01-23T22:08:24.1055628Z #1 transferring dockerfile: 1.25kB done
2021-01-23T22:08:24.1055992Z #1 DONE 0.0s
2021-01-23T22:08:24.1056168Z 
2021-01-23T22:08:24.1056410Z #2 [internal] load .dockerignore
2021-01-23T22:08:24.1056724Z #2 sha256:4357111c37c77eacab7955a6a242f9c517e96468823eb1f65c5cf619033b2e87
2021-01-23T22:08:24.1057106Z #2 transferring context: 2B done
2021-01-23T22:08:24.1057374Z #2 DONE 0.0s
2021-01-23T22:08:24.1057444Z 
2021-01-23T22:08:24.1058418Z #3 [internal] load metadata for docker.io/library/ubuntu:xenial-20201030
2021-01-23T22:08:24.1058811Z #3 sha256:4b2b257c52fdd43e9dd212adca823983a4c857f3639b1e4a9afeeabcc6c6b7fc
2021-01-23T22:08:29.2109219Z #3 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/ubuntu/manifests/sha256:3355b6e4ba1b12071ba5fe9742042a2f10b257c908fbdfac81912a16eb463879: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2021-01-23T22:08:29.2113319Z ------
2021-01-23T22:08:29.2132366Z  > [internal] load metadata for docker.io/library/ubuntu:xenial-20201030:
2021-01-23T22:08:29.2143589Z ------
2021-01-23T22:08:29.2147741Z error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/ubuntu/manifests/sha256:3355b6e4ba1b12071ba5fe9742042a2f10b257c908fbdfac81912a16eb463879: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2021-01-23T22:08:29.2196994Z ##[error]buildx call failed with: error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/ubuntu/manifests/sha256:3355b6e4ba1b12071ba5fe9742042a2f10b257c908fbdfac81912a16eb463879: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2021-01-23T22:08:29.2378827Z Post job cleanup.
2021-01-23T22:08:29.2923539Z 🚿 Removing temp folder /tmp/docker-build-push-AFVHbW
2021-01-23T22:08:29.2974671Z Post job cleanup.
2021-01-23T22:08:29.3338254Z [command]/usr/bin/docker logout docker.artifactory.tri-ad.tech
2021-01-23T22:08:29.3692436Z Removing login credentials for docker.artifactory.tri-ad.tech
2021-01-23T22:08:29.3779533Z Post job cleanup.
2021-01-23T22:08:29.4244905Z [command]/usr/bin/docker buildx rm builder-f8ce65ae-7c82-4109-ad95-b5cc86f6c1e8
2021-01-23T22:08:29.7227857Z Post job cleanup.
2021-01-23T22:08:29.8102997Z Cleaning up orphan processes

I have no access rights to manage the runner, so I can not change the runner configuration. Switching to a paid account is likewise not an option, since we already have a local registry that we should use instead of paying for an extra service that should not be needed.

[crazy-max]

@wichert

I have no access rights to manage the runner, so I can not change the runner configuration.

Ok so in this case you have to handle the FROM instructions to pull from Artifactory instead of Docker Hub. So for example, assuming dockerhub.artifactory.tri-ad.tech is configured on Artifactory as a remote repository for Docker Hub, you have to replace in ./python-ci/Dockerfile:

FROM ubuntu:xenial-20201030

with

ARG REGISTRY
FROM ${REGISTRY}ubuntu:xenial-20201030

and use the following workflow:

jobs:
  python:
    runs-on: self-hosted

    name: Build Python ${{ matrix.python }}

    strategy:
      matrix:
        python: ["3.8", "3.9"]

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Set up Docker Buildx
        uses: synced-actions/docker-setup-buildx-action@v1

      - name: Login to Artifactory
        uses: synced-actions/docker-login-action@v1
        with:
          registry: docker.artifactory.tri-ad.tech
          username: ${{ secrets.ARTIFACTORY_USER }}
          password: ${{ secrets.ARTIFACTORY_API_KEY }}

      - name: Build and push
        uses: synced-actions/docker-build-push-action@v2
        with:
          cache-from: type=registry,ref=user/app:latest
          push: true
          tags: repository/python-ci:${{ matrix.python }}
          context: ./python-ci
          file: ./python-ci/Dockerfile
          build-args: |
            REGISTRY=dockerhub.artifactory.tri-ad.tech/
            PYTHON_VERSION=${{ matrix.python }}

[netcaptors]

@crazy-max to add to @wichert comments, we have already configured:

{
  "registry-mirrors": ["https://docker.artifactory.tri-ad.tech"]
}

as as a pull-through cache, if we use build-push-action it uses docker.io registry to pull, if we use docker without this action its pulling through our artifactory registry, so i assume there is issue with build-push-action

[crazy-max]

@netcaptors Hum yes indeed, I think you need at least Docker 20.10 to support mirrors with containerd (moby/buildkit#1787). WDYT @tonistiigi?

[ScOut3R]

@crazy-max we are running Docker 20.10.7 as have a registry mirror configured yet the action is still pulling from Docker Hub directly.

[crazy-max]

Sorry for the delay. It seems you're running a builder using the docker-container driver. In this case you have to define the mirror configuration for this builder with buildkitd.toml (not just in the engine daemon config):

jobs:
  python:
    runs-on: self-hosted

    name: Build Python ${{ matrix.python }}

    strategy:
      matrix:
        python: ["3.8", "3.9"]

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Create buildkitd conf
        run: |
          cat > /tmp/buildkitd.toml <<EOL
          [registry."docker.io"]
            mirrors = ["docker.artifactory.tri-ad.tech"]
          EOL

      - name: Set up Docker Buildx
        uses: synced-actions/docker-setup-buildx-action@v1
        with:
          config: /tmp/buildkitd.toml

      - name: Login to Artifactory
        uses: synced-actions/docker-login-action@v1
        with:
          registry: docker.artifactory.tri-ad.tech
          username: ${{ secrets.ARTIFACTORY_USER }}
          password: ${{ secrets.ARTIFACTORY_API_KEY }}

      - name: Build and push
        uses: synced-actions/docker-build-push-action@v2
        with:
          cache-from: type=registry,ref=user/app:latest
          push: true
          tags: repository/python-ci:${{ matrix.python }}
          context: ./python-ci
          file: ./python-ci/Dockerfile
          build-args: |
            REGISTRY=dockerhub.artifactory.tri-ad.tech/
            PYTHON_VERSION=${{ matrix.python }}

I will add some documentation on setup-buildx-action repo. I also think I will add an inline config input for the setup-buildx-action to avoid creating the file in a dedicated step.

[crazy-max]

Documentation available: https://github.com/docker/setup-buildx-action#registry-mirror