Silent error on failing to mirror images

[aiyengar2]

The following logs are observed on https://drone-publish.rancher.io/rancher/image-mirror/1058/1/3 on trying to mirror the image added in the PR rancher/rancher#43149.

While the Job passed, the image was never actually mirrored to https://hub.docker.com/r/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook/tags.

Line: registry.k8s.io/gmsa-webhook/k8s-gmsa-webhook rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook v0.7.0
registry.k8s.io/gmsa-webhook/k8s-gmsa-webhook rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook v0.7.0
registry.k8s.io/gmsa-webhook/k8s-gmsa-webhook:v0.7.0 is schemaVersion 2
registry.k8s.io/gmsa-webhook/k8s-gmsa-webhook:v0.7.0 is mediaType application/vnd.docker.distribution.manifest.list.v2+json
	Copying registry.k8s.io/gmsa-webhook/k8s-gmsa-webhook@sha256:38c99a8efc9c3c4c0dca50a95f26fdcc053e84b960d114b93f9e5a426f51478d => docker.io/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook:v0.7.0-amd64
	        sha256:38c99a8efc9c3c4c0dca50a95f26fdcc053e84b960d114b93f9e5a426f51478d => MISSING
Getting image source signatures
Copying blob sha256:3144a634950d742e6a2c0d51958e0d81d516fc99868dedc6dcc7391fccce318e
Copying blob sha256:d5696692a0e2e08ec9abe07946c01d5d05ebf39bc7f85a0f6343e2b95d78790f
time="2023-10-16T17:10:49Z" level=fatal msg="writing blob: initiating layer upload to /v2/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook/blobs/uploads/ in registry-1.docker.io: requested access to the resource is denied"
===
Failed copying image for rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook
===
	Copying registry.k8s.io/gmsa-webhook/k8s-gmsa-webhook@sha256:9d59b3f577a6b0383978f1f957b0930c28580f84c74eaa823ebf27be715d96aa => docker.io/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook:v0.7.0-arm64
	        sha256:9d59b3f577a6b0383978f1f957b0930c28580f84c74eaa823ebf27be715d96aa => MISSING
Getting image source signatures
Copying blob sha256:3144a634950d742e6a2c0d51958e0d81d516fc99868dedc6dcc7391fccce318e
Copying blob sha256:d5696692a0e2e08ec9abe07946c01d5d05ebf39bc7f85a0f6343e2b95d78790f
time="2023-10-16T17:10:49Z" level=fatal msg="writing blob: initiating layer upload to /v2/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook/blobs/uploads/ in registry-1.docker.io: requested access to the resource is denied"
===
Failed copying image for rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook
===
	arm NOT FOUND
	s390x NOT FOUND
	Writing manifest list to docker.io/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook:v0.7.0
sha256:38c99a8efc9c3c4c0dca50a95f26fdcc053e84b960d114b93f9e5a426f51478d
sha256:9d59b3f577a6b0383978f1f957b0930c28580f84c74eaa823ebf27be715d96aa
httpReaderSeeker: failed open: content at https://registry-1.docker.io/v2/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook/manifests/sha256:9d59b3f577a6b0383978f1f957b0930c28580f84c74eaa823ebf27be715d96aa not found: not found
===
Failed copying image for rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook
===
Updating description for docker.io/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook

For context, here is how the manifest looks like:

$ docker manifest inspect registry.k8s.io/gmsa-webhook/k8s-gmsa-webhook:v0.7.0
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 698,
         "digest": "sha256:38c99a8efc9c3c4c0dca50a95f26fdcc053e84b960d114b93f9e5a426f51478d",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 698,
         "digest": "sha256:9d59b3f577a6b0383978f1f957b0930c28580f84c74eaa823ebf27be715d96aa",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

[aiyengar2]

This seems like it is tied to #379

[superseb]

@aiyengar2 This seems to be a permission issue, was the new repository created with the permissions so that the DockerHub user can push to it from here?

time="2023-10-16T17:10:49Z" level=fatal msg="writing blob: initiating layer upload to /v2/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook/blobs/uploads/ in registry-1.docker.io: requested access to the resource is denied"

[aiyengar2]

@aiyengar2 This seems to be a permission issue, was the new repository created with the permissions so that the DockerHub user can push to it from here?

time="2023-10-16T17:10:49Z" level=fatal msg="writing blob: initiating layer upload to /v2/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook/blobs/uploads/ in registry-1.docker.io: requested access to the resource is denied"

@Anarkis can we double check the permissions on the DockerHub repository?

[Anarkis]

@aiyengar2 This seems to be a permission issue, was the new repository created with the permissions so that the DockerHub user can push to it from here?

time="2023-10-16T17:10:49Z" level=fatal msg="writing blob: initiating layer upload to /v2/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook/blobs/uploads/ in registry-1.docker.io: requested access to the resource is denied"

@Anarkis can we double check the permissions on the DockerHub repository?

The repo rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook is not found into the list https://github.com/rancherlabs/drone-repo-management/blob/master/repos.txt so it's not enabled for publishing images.

[aiyengar2]

@Anarkis I'm not sure what you mean.

The repository that is pushing this image is rancher/image-mirror, which is in that list:

https://github.com/rancherlabs/drone-repo-management/blob/b1a229369cf24611d9f6e99a31eb81110c3ab1e6/repos.txt#L75

[superseb]

@Anarkis Can you check if the Docker Hub user that is configured in rancher/image-mirror has permissions to push/write to https://hub.docker.com/r/rancher/mirrored-gmsa-webhook-k8s-gmsa-webhook/ ?

[Anarkis]

@aiyengar2 @superseb the new repo requested is not configured to "be pushed" from an outside tool. Do you know what I mean?
It's not the repo which is pushing, is the DockerHub repo which will contain the image
https://github.com/rancherlabs/drone-repo-management#dockerhub-image-publishing

Please @aiyengar2 open a new issue with this

[superseb]

This is working now, I created a follow-up for the silent error: #496