OSX: manifest blob unknown: blob unknown to registry pushing linux image

[andig]

Trying to push a multi-arch image from OSX:

❯ docker manifest create --amend andig/evcc:latest andig/evcc:latest-amd64 andig/evcc:latest-arm.v6
Created manifest list docker.io/andig/evcc:latest

~/htdocs/evcc master*
❯ docker manifest annotate andig/evcc:latest andig/evcc:latest-amd64 --os=linux --arch=amd64

~/htdocs/evcc master*
❯ docker manifest annotate andig/evcc:latest andig/evcc:latest-arm.v6 --os=linux --arch=arm --variant=v6

~/htdocs/evcc master*
❯ docker manifest push andig/evcc:latest

failed to put manifest docker.io/andig/evcc:latest: errors:
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry

docker info:

❯ docker info
Client:
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)
  app: Docker Application (Docker Inc., v0.8.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 3
 Total Memory: 7.778GiB
 Name: docker-desktop
 ID: PD5J:E3CB:VKTS:6OKZ:DX3G:LRZL:5JPC:UTMD:BBHU:2UKZ:GLKV:ETL6
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 34
  Goroutines: 51
  System Time: 2020-02-23T19:51:27.920620011Z
  EventsListeners: 3
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Refs ldez/seihon#6, any hint appreciated...

[andig]

Any help with this? The exact same commands work when run against andig/evcc1, but not andig/evcc when using the public docker registry. Even when deleting andig/evcc, the problem shows again.

[andig]

Looks like https://bugzilla.redhat.com/show_bug.cgi?id=1408661, openshift/origin#12389 and openshift/origin#13001.

[caarlos0]

FWIW, this seems to happen only if the tag is latest for some reason:

$ docker manifest create localhost:5000/goreleaser/test_multiarch --amend localhost:5000/goreleaser/test_multiarch:latest-amd64 --amend localhost:5000/goreleaser/test_multiarch:latest-arm64v8 --insecure
Created manifest list localhost:5000/goreleaser/test_multiarch:latest

$ docker manifest push localhost:5000/goreleaser/test_multiarch --insecure
failed to put manifest localhost:5000/goreleaser/test_multiarch:latest: errors:
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry

$ docker manifest create localhost:5000/goreleaser/test_multiarch:foo --amend localhost:5000/goreleaser/test_multiarch:latest-amd64 --amend localhost:5000/goreleaser/test_multiarch:latest-arm64v8 --insecure
Created manifest list localhost:5000/goreleaser/test_multiarch:foo

$ docker manifest push localhost:5000/goreleaser/test_multiarch:foo --insecure
sha256:d58cf5f2b3512a396e5736e152e29a6343bd0c5cda10c3f4862860508f872609

---

EDIT: also happens with other tags, eventually.