Build Multi Arch Image Error: server message: insufficient_scope: authorization failed

[wangcanfengxs]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

docker buildx build --pull -f ./Dockerfile --build-arg BASE_IMAGE=harbor.example.com/devops/jdk8-openjdk-skiff:v1.6 --platform linux/amd64,linux/arm64 --push . -t harbor.example.com/lctest/test:4dcd8eac

Expected behaviour

push successfully

Actual behaviour

#23 [auth] lcdev/playwright-java8:pull lctest/test:pull,push token for harbor.example.com
[2024-03-28 10:46:05] #23 sha256:d0078f5ccd4e4b94fbcb708aceebf38d48339d9d0fb6e5a308bc05fc0860e58c
[2024-03-28 10:46:05] #23 DONE 0.0s
[2024-03-28 10:46:05]
[2024-03-28 10:46:05] #24 [auth] lcdev/playwright-java8:pull lctest/test:pull,push token for harbor.example.com
[2024-03-28 10:46:05] #24 sha256:d6813da4a1ba3d97cfaac03368aca522c1145b52ed917f1a1034fab9dea8f8f0
[2024-03-28 10:46:05] #24 DONE 0.0s
[2024-03-28 10:46:05]
[2024-03-28 10:46:05] #21 exporting to image
[2024-03-28 10:46:05] #21 sha256:d1f58143e758915860091f89ba9a815b655303b4a9da7e6ca9b9aa917ab657f9
[2024-03-28 10:46:05] #21 pushing layers 0.4s done
[2024-03-28 10:46:05] #21 ERROR: failed to push harbor.example.com/lctest/test: server message: insufficient_scope: authorization failed
[2024-03-28 10:46:05] ------
[2024-03-28 10:46:05] > exporting to image:
[2024-03-28 10:46:05] ------
[2024-03-28 10:46:05] error: failed to solve: rpc error: code = Unknown desc = failed to push harbor.cloud.netease.com/qztest/cicd-testng:4dcd8eac: server message: insufficient_scope: authorization failed

Buildx version

github.com/docker/buildx v0.11.2 9872040

Docker info

Client:
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.11.2)

Server:
 Containers: 37
  Running: 16
  Paused: 0
  Stopped: 21
 Images: 141
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.268-1.el7.elrepo.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.62GiB
 Name: node20240130-015
 ID: YPCI:5MHR:M67F:PBWU:YTF5:GXOA:KTMR:VQW6:4CRC:TDAF:5Q37:Q5YH
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 110
  Goroutines: 105
  System Time: 2024-03-28T10:55:28.268585796+08:00
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  harbor.example.com
  127.0.0.0/8
 Live Restore Enabled: true

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Builders list

NAME/NODE                      DRIVER/ENDPOINT             STATUS  BUILDKIT       PLATFORMS
cicd_multi_platform_builder *  docker-container
  cicd_multi_platform_builder0 unix:///var/run/docker.sock running v0.12.5        linux/amd64*, linux/amd64/v2*, linux/amd64/v3*, linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/386*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*
default                        docker
  default                      default                     running v0.6.4+df89d4d linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Configuration

none

Build logs

No response

Additional info

No response

[wangcanfengxs]

I find when push image，buildx will auth another repo

#24 [auth] lcdev/playwright-java8:pull lctest/cicd-testng:pull,push token for harbor.example.com

I indeed no permission for lcdev repo.
But why buildx would check pull permission for lcdev/playwright-java8

[Nox-404]

got the same issue that I could reproduce 100%:

1. create a registry A, build and push an image on registry A -> OK
2. remove registry A
3. create a registry B, build and push the same image as before but tag with registry B -> KO (asking for a token to pull from registry A which fails as registry A is no longer allowed)

[Nox-404]

The only way I could find was to purge any docker buildx cache

[Nox-404]

When using a docker distribution with an S3 backend I get this call which respond with 401 and asks for a bearer for both repo:
https://[my-s3-api]/[repository-b/image-b]/blobs/uploads/?mount=sha256:[somesha256]&from=[repository-a/image-a]

[Sryther]

I'm facing the same issue.

The only way I could find was to purge any docker buildx cache

It's working but I can't implement it that way in my situation.

https://[my-s3-api]/[repository-b/image-b]/blobs/uploads/?mount=sha256:[somesha256]&from=[repository-a/image-a]

Looks like there is only one token asked and used by buildx while two different repositories with different permissions are used.

[Sryther]

I find when push image，buildx will auth another repo

#24 [auth] lcdev/playwright-java8:pull lctest/cicd-testng:pull,push token for harbor.example.com

I indeed no permission for lcdev repo. But why buildx would check pull permission for lcdev/playwright-java8

@wangcanfengxs, I think buildx tries to pull cached layers from lcdev repo before pushing to lctest because it probably already pushed some of them before.