Unable to Use Authentication Against Registries Using kubernetes driver

[davejhahn]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

I have build agents running in Kubernetes. When they run a build, it uses docker buildx create using the kubernetes drive, and then docker buildx build to run the build.

I ran into some issues because it was using anonymous authentication against Docker.io. The limit is 100 per 6 hours. My goal is to have buildkit authenticate against Docker.Io so there is no rate limit (using our licensed user and API auth token).

What I did:

• Created a Kubernetes secret (generic) with the docker.config file containing the auth token in the buildkit namespace
• Inject the volume mount on any pod container buildkitd in the buildkit namespace to /root/.docker
• Inject an ENV variable DOCKER_CONFIG=/root/.docker same as above
• Added --bootstrap to the docker buildx create so that I can add a sleep and access the buildkit pod as it is created.

I see:

• /root/.docker/config.json with the expected and validated auth
• ENV variable DOCKER_CONFIG set to /root/.docker

Prior to running the build, I obtain the rate limit and print it in the build logs (which uses Anonymous auth):

      TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
      curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest

What I observe is that after each build, the ratelmit decreases indicating it is using the Anonymous authentication.

I have also tried creating a .toml config file and passing it in with --config:

   [registry."docker.io"]
     config = "/root/.docker/config.json"

Using docker buiildx inspect to confirm that it is being passed and validating in /etc/ folder that the .toml file with the expected name is there and has the expected values.

Maybe I'm doing something wrong, most likely, but I am out of ideas. It seems that buildkit does not adhere to any sort of authentication provided.

Added information:

• Running Azure DevOps
• Each Build Job Creates a Keda Scaled Job for a new Build Agent Pod
• The build runs on the above pod, and then buildkit creates a subsequent pod in a different namespace for the build

I would also add that the creation of the buildkit pod (which also pulls from docker.io is NOT a problem). In this case, I've created a K8S secret in the buildkit namespace and added a pullsecret to the default service account in the buildkit namespace. I know this works because we were at a point where the rate limit was 0 and adding this immediately began working. But that only helps for the buildkit pod, not the builds within the buildkit pod.

Expected behaviour

Authentication against registry using provided credentials

Actual behaviour

Ignores provided authentication and defaults to Anonymous authentication, which is limited to 100 pulls per 6 hours.

Buildx version

github.com/docker/buildx v0.20.0 8e30c46

Docker info

Client: Docker Engine - Community
 Version:    27.5.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.20.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.4
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Builders list

Client: Docker Engine - Community
 Version:    27.5.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.20.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.4
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Configuration

Dockerfile can is not anything special.

Build logs

Additional info

No response

[tonistiigi]

The registry credentials are loaded on the client side, so on the environment where you call buildx. It does not depend on the buildx driver you are using or need any buildkitd configuration (there is no such thing as registry->config in toml).

[davejhahn]

@tonistiigi just to clarify, when I call docker buildx on the client side using the kubernetes driver, it creates a buildkit pod which the build runs on. You are saying that if I authenticate on the client those credentials are used inside the buildkit pod that is created, correct?

[tonistiigi]

Yes, credentials are on the client side. If buildkit inside k8s needs to authenticate against a registry it will call back to client. Client will ask registry auth service for a temporary token for the required repo and scope(push/pull) using your credentials and then send that token to buildkit daemon.

You will see smth like

#3 [auth] library/alpine:pull token for registry-1.docker.io
#3 DONE 0.0s

In progress stream when client sends a token to buildkit daemon.

[davejhahn]

@tonistiigi ok, makes sense, but what I don't understand is how this works if there are separate repositories.

Example: Docker file pulls from docker.io to create a new image, and then docker buildx pushes the image to a private registry

Currently, we are authenticating against our private registry prior to running docker buildx build. So really there are two separate repositories that require credentials, our private registry (for pushing) and the docker.io registry (for pulling).

[tonistiigi]

Docker config file does not support different credentials per repository. Only one set per registry hostname. The tokens are still scoped by repository so buildx will create a separate token for each image and send it to buildkit daemon(in k8s) but if all these images are from Docker Hub then all token will be created based on the same credentials.