Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error pulling image on GKE in Singapore #116

Closed
BenTheElder opened this issue Oct 13, 2022 · 10 comments
Closed

error pulling image on GKE in Singapore #116

BenTheElder opened this issue Oct 13, 2022 · 10 comments
Assignees
@BenTheElder @ameukam
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra.

Comments

@BenTheElder
Copy link
Member

Following #115 (comment) (which seems unrelated to the original issue, which was seemingly an issue with the loadbalancer when removing europe-west3), I couldn't reproduce locally, but I can reproduce the issue on a standard GKE cluster in asia-southeast1-a

  1. create a standard cluster in asia-southeast1-a
  2. kubectl run --image=registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 test
  3. kubectl describe pod test

observe events:

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  4m53s                  default-scheduler  Successfully assigned default/test to gke-sg-test-default-pool-0f72f8d8-znhx
  Normal   Pulling    3m24s (x4 over 4m52s)  kubelet            Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47"
  Warning  Failed     3m24s (x4 over 4m51s)  kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47": rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47": failed to resolve reference "registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47": failed to do request: Head "https://asia-southeast1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47": failed to authorize redirect: failed to fetch anonymous token: unexpected status: 403 Forbidden
  Warning  Failed     3m24s (x4 over 4m51s)  kubelet            Error: ErrImagePull
  Warning  Failed     3m12s (x6 over 4m50s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    2m59s (x7 over 4m50s)  kubelet            Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47"

cc @winggundamth

/assign
/priority critical-urgent
@k8s-ci-robot k8s-ci-robot added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Oct 13, 2022
@BenTheElder BenTheElder added kind/bug Categorizes issue or PR as related to a bug. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. labels Oct 13, 2022
@BenTheElder BenTheElder mentioned this issue Oct 13, 2022
Closed
@BenTheElder BenTheElder changed the title error pulling image on GKE in singapore error pulling image on GKE in Singapore Oct 13, 2022
@BenTheElder
Copy link
Member Author

BenTheElder commented Oct 13, 2022 •
edited
Loading

On another, all-default cluster in us-central1-c, I do not see this issue.

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  32s                default-scheduler  Successfully assigned default/test to gke-all-default-default-pool-b2407068-43mm
  Normal   Pulling    31s                kubelet            Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47"
  Normal   Pulled     29s                kubelet            Successfully pulled image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47" in 1.527250893s
  Normal   Created    14s (x3 over 29s)  kubelet            Created container test
  Normal   Started    14s (x3 over 29s)  kubelet            Started container test
  Normal   Pulled     14s (x2 over 28s)  kubelet            Container image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47" already present on machine
  Warning  BackOff    13s (x3 over 27s)  kubelet            Back-off restarting failed container

@BenTheElder
Copy link
Member Author

From a workstation on the US west-coast I can pull the backing image for this region:

$ docker pull asia-southeast1-docker.pkg.dev/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
asia-southeast1-docker.pkg.dev/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47: Pulling from k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen
Digest: sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
Status: Downloaded newer image for asia-southeast1-docker.pkg.dev/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
asia-southeast1-docker.pkg.dev/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47

@BenTheElder
Copy link
Member Author

In asia-southeast1:

bentheelder@gke-sg-test-default-pool-0f72f8d8-znhx ~ $ docker run --rm gcr.io/go-containerregistry/crane pull --verbose registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 /dev/null
2022/10/13 06:23:07 --> GET https://registry.k8s.io/v2/
2022/10/13 06:23:07 GET /v2/ HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Accept-Encoding: gzip


2022/10/13 06:23:07 <-- 200 https://registry.k8s.io/v2/ (35.648316ms)
2022/10/13 06:23:07 HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html
Date: Thu, 13 Oct 2022 06:23:07 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 3091c6aff00fc0e764dd54d338c32805
Content-Length: 0


2022/10/13 06:23:07 --> GET https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
2022/10/13 06:23:07 GET /v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Accept-Encoding: gzip


2022/10/13 06:23:07 <-- 307 https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 (9.980463ms)
2022/10/13 06:23:07 HTTP/2.0 307 Temporary Redirect
Content-Length: 220
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 13 Oct 2022 06:23:07 GMT
Location: https://asia-southeast1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 5f20a4450fb25b434d48ff5d94f0dedb

<a href="https://asia-southeast1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47">Temporary Redirect</a>.


2022/10/13 06:23:07 --> GET https://asia-southeast1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
2022/10/13 06:23:07 GET /v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 HTTP/1.1
Host: asia-southeast1-docker.pkg.dev
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Referer: https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
Accept-Encoding: gzip


2022/10/13 06:23:07 <-- 401 https://asia-southeast1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 (26.412454ms)
2022/10/13 06:23:07 HTTP/2.0 401 Unauthorized
Content-Length: 102
Content-Type: application/json
Date: Thu, 13 Oct 2022 06:23:07 GMT
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://asia-southeast1-docker.pkg.dev/v2/token",service="asia-southeast1-docker.pkg.dev",scope="repository:k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen:pull"
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{"errors":[{"code":"UNAUTHORIZED","message":"not authenticated: No valid credential was supplied."}]}

Error: GET https://asia-southeast1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47: UNAUTHORIZED: not authenticated: No valid credential was supplied.

@BenTheElder
Copy link
Member Author

@ameukam found this is because asia-southeast1 has a cloud run region pointing at the matching AR region, but we have no AR instance there.

We need to audit that all the cloud run regions do have matching AR, we missed this one at least flipping from GCR to AR.

@BenTheElder
Copy link
Member Author

kubernetes/k8s.io#4337 is where we moved to AR

@BenTheElder BenTheElder mentioned this issue Oct 13, 2022
Merged
@BenTheElder
Copy link
Member Author

Confirmed @ameukam has rolled out a fix for this:

bentheelder@gke-sg-test-default-pool-0f72f8d8-znhx ~ $ docker run --rm gcr.io/go-containerregistry/crane pull --verbose registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 /dev/null
2022/10/13 06:40:42 --> GET https://registry.k8s.io/v2/
2022/10/13 06:40:42 GET /v2/ HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Accept-Encoding: gzip


2022/10/13 06:40:42 <-- 200 https://registry.k8s.io/v2/ (82.255525ms)
2022/10/13 06:40:42 HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html
Date: Thu, 13 Oct 2022 06:40:42 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 848a238b2567047c5dfc4f14407ec5e8
Content-Length: 0


2022/10/13 06:40:42 --> GET https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
2022/10/13 06:40:42 GET /v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Accept-Encoding: gzip


2022/10/13 06:40:42 <-- 307 https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 (58.164282ms)
2022/10/13 06:40:42 HTTP/2.0 307 Temporary Redirect
Content-Length: 215
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 13 Oct 2022 06:40:42 GMT
Location: https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 9fe2e51fba9f0c1363b97cf0e4b07a80

<a href="https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47">Temporary Redirect</a>.


2022/10/13 06:40:42 --> GET https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
2022/10/13 06:40:42 GET /v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 HTTP/1.1
Host: asia-east1-docker.pkg.dev
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Referer: https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
Accept-Encoding: gzip


2022/10/13 06:40:42 <-- 200 https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 (278.914493ms)
2022/10/13 06:40:42 HTTP/2.0 200 OK
Content-Length: 1384
Content-Type: application/vnd.docker.distribution.manifest.list.v2+json
Date: Thu, 13 Oct 2022 06:40:42 GMT
Docker-Content-Digest: sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "schemaVersion": 2,
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:7bbcbd4232c692a36f8796e9602c14f2ec3b6c638d2974ea81b736ee2bd6e279",
         "size": 739,
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:4e635eb9c52380df7b2f96ad6a7784aa383a6fd252c525aa99eedacb1ebccdc1",
         "size": 739,
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:c0f20cf81d2ce84fab15e78d68e9ded03948aa408fd7f1a6a125a84ccf3c915c",
         "size": 739,
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:deac4e784150f494119317b1ed02f4af8167c9305e91504ddef6f70358e7eeea",
         "size": 739,
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}
2022/10/13 06:40:42 --> GET https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:7bbcbd4232c692a36f8796e9602c14f2ec3b6c638d2974ea81b736ee2bd6e279
2022/10/13 06:40:42 GET /v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:7bbcbd4232c692a36f8796e9602c14f2ec3b6c638d2974ea81b736ee2bd6e279 HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v2+json
Accept-Encoding: gzip


2022/10/13 06:40:42 <-- 307 https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:7bbcbd4232c692a36f8796e9602c14f2ec3b6c638d2974ea81b736ee2bd6e279 (56.790168ms)
2022/10/13 06:40:42 HTTP/2.0 307 Temporary Redirect
Content-Length: 215
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 13 Oct 2022 06:40:42 GMT
Location: https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:7bbcbd4232c692a36f8796e9602c14f2ec3b6c638d2974ea81b736ee2bd6e279
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 61f3dca521431b684915c7db9f89cc15

<a href="https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:7bbcbd4232c692a36f8796e9602c14f2ec3b6c638d2974ea81b736ee2bd6e279">Temporary Redirect</a>.


2022/10/13 06:40:42 --> GET https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:7bbcbd4232c692a36f8796e9602c14f2ec3b6c638d2974ea81b736ee2bd6e279
2022/10/13 06:40:42 GET /v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:7bbcbd4232c692a36f8796e9602c14f2ec3b6c638d2974ea81b736ee2bd6e279 HTTP/1.1
Host: asia-east1-docker.pkg.dev
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v2+json
Referer: https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:7bbcbd4232c692a36f8796e9602c14f2ec3b6c638d2974ea81b736ee2bd6e279
Accept-Encoding: gzip


2022/10/13 06:40:42 <-- 200 https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:7bbcbd4232c692a36f8796e9602c14f2ec3b6c638d2974ea81b736ee2bd6e279 (60.192053ms)
2022/10/13 06:40:42 HTTP/2.0 200 OK
Content-Length: 739
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Date: Thu, 13 Oct 2022 06:40:42 GMT
Docker-Content-Digest: sha256:7bbcbd4232c692a36f8796e9602c14f2ec3b6c638d2974ea81b736ee2bd6e279
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "schemaVersion": 2,
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "digest": "sha256:69547dffc18fcddd4f41b7696a046ad7252ceeabce944106da94fa4bfb3c6b24",
      "size": 1916
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:b9f88661235d25835ef747dab426861d51c4e9923b92623d422d7ac58eb123e9",
         "size": 804101
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:23691ac0df28d0d74653810f3cde139697129a2b46102af61f38691bb4262422",
         "size": 18581579
      }
   ]
}
2022/10/13 06:40:42 --> GET https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:69547dffc18fcddd4f41b7696a046ad7252ceeabce944106da94fa4bfb3c6b24
2022/10/13 06:40:42 GET /v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:69547dffc18fcddd4f41b7696a046ad7252ceeabce944106da94fa4bfb3c6b24 HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Accept-Encoding: gzip


2022/10/13 06:40:42 <-- 307 https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:69547dffc18fcddd4f41b7696a046ad7252ceeabce944106da94fa4bfb3c6b24 (61.908492ms)
2022/10/13 06:40:42 HTTP/2.0 307 Temporary Redirect
Content-Length: 211
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 13 Oct 2022 06:40:42 GMT
Location: https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:69547dffc18fcddd4f41b7696a046ad7252ceeabce944106da94fa4bfb3c6b24
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: f07540a8ecdd8aaa200bc2bbd039b4f6

<a href="https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:69547dffc18fcddd4f41b7696a046ad7252ceeabce944106da94fa4bfb3c6b24">Temporary Redirect</a>.


2022/10/13 06:40:42 --> GET https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:69547dffc18fcddd4f41b7696a046ad7252ceeabce944106da94fa4bfb3c6b24
2022/10/13 06:40:42 GET /v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:69547dffc18fcddd4f41b7696a046ad7252ceeabce944106da94fa4bfb3c6b24 HTTP/1.1
Host: asia-east1-docker.pkg.dev
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Referer: https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:69547dffc18fcddd4f41b7696a046ad7252ceeabce944106da94fa4bfb3c6b24
Accept-Encoding: gzip


2022/10/13 06:40:43 <-- 302 https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:69547dffc18fcddd4f41b7696a046ad7252ceeabce944106da94fa4bfb3c6b24 (172.572738ms)
2022/10/13 06:40:43 HTTP/2.0 302 Found
Content-Length: 828
Content-Type: text/html; charset=utf-8
Date: Thu, 13 Oct 2022 06:40:43 GMT
Docker-Distribution-Api-Version: registry/2.0
Location: /artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JDT1xsTt_IhY1GFGXPSDYEuJPUXsUg3W7_BwatIXtOIj7Ov3RkXpQgydp6ObPimmI6v3klgPjkna4EdqZKi862alaYAZwSBl7TZyuUlFtQNKry7lQlcZ39mjRqXE-ACJDybboFob-_rmgvjGDs1fS9J5aZ9scurOCowF-gu-OSwo8uK5QrVb56RDHnlAbko9Ry6bLq1_31M5JCQvO9hbqlomUgnMYWqyCZ43ojCdaQqq_N7DHCr5qknqJ33bzbCEaJgxNsLeWPCCbIse8rPCSgV9PbvXazDtou1_ynA1Be5R7MR6U3vpYZ5HP41sJbZoBkhohKK_GeijQbJwMBqVHcroLcmlhJZLjY8wUQzhhHABYQNiDe2B0GTCXKXXWTUFjyGdbQFGXmLKejLIR6YUaPqA20gbwvYhsaCfvCmBjewONemzympLvUhzw-qZIEhF71L7TkN6d5KQ_PnfaqMCLnVo7ci4SMM9P-WdWoy-sLXXC4GRQmK2Qv8go54S_13hlYRTlzJ_PUKhPuOzD8DK9SWcDOdHh7C10Ldnege5IvB_ViDZdbh6rwIot5Vrd_xEIM4v3o_2dA5s1g4vV3f_GsSwDU43k2X5WuLaeCuq9RKUVzsna0oKuJB87P0ENkQeeoAbH7fSYxhUZ_XR5jSju54QbSSMnkC01-iDjfO-dCzQXSHDgHIIWAf_H450-MFC8BoWRll1SgG4w==
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

<a href="/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JDT1xsTt_IhY1GFGXPSDYEuJPUXsUg3W7_BwatIXtOIj7Ov3RkXpQgydp6ObPimmI6v3klgPjkna4EdqZKi862alaYAZwSBl7TZyuUlFtQNKry7lQlcZ39mjRqXE-ACJDybboFob-_rmgvjGDs1fS9J5aZ9scurOCowF-gu-OSwo8uK5QrVb56RDHnlAbko9Ry6bLq1_31M5JCQvO9hbqlomUgnMYWqyCZ43ojCdaQqq_N7DHCr5qknqJ33bzbCEaJgxNsLeWPCCbIse8rPCSgV9PbvXazDtou1_ynA1Be5R7MR6U3vpYZ5HP41sJbZoBkhohKK_GeijQbJwMBqVHcroLcmlhJZLjY8wUQzhhHABYQNiDe2B0GTCXKXXWTUFjyGdbQFGXmLKejLIR6YUaPqA20gbwvYhsaCfvCmBjewONemzympLvUhzw-qZIEhF71L7TkN6d5KQ_PnfaqMCLnVo7ci4SMM9P-WdWoy-sLXXC4GRQmK2Qv8go54S_13hlYRTlzJ_PUKhPuOzD8DK9SWcDOdHh7C10Ldnege5IvB_ViDZdbh6rwIot5Vrd_xEIM4v3o_2dA5s1g4vV3f_GsSwDU43k2X5WuLaeCuq9RKUVzsna0oKuJB87P0ENkQeeoAbH7fSYxhUZ_XR5jSju54QbSSMnkC01-iDjfO-dCzQXSHDgHIIWAf_H450-MFC8BoWRll1SgG4w==">Found</a>.


2022/10/13 06:40:43 --> GET https://asia-east1-docker.pkg.dev/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JDT1xsTt_IhY1GFGXPSDYEuJPUXsUg3W7_BwatIXtOIj7Ov3RkXpQgydp6ObPimmI6v3klgPjkna4EdqZKi862alaYAZwSBl7TZyuUlFtQNKry7lQlcZ39mjRqXE-ACJDybboFob-_rmgvjGDs1fS9J5aZ9scurOCowF-gu-OSwo8uK5QrVb56RDHnlAbko9Ry6bLq1_31M5JCQvO9hbqlomUgnMYWqyCZ43ojCdaQqq_N7DHCr5qknqJ33bzbCEaJgxNsLeWPCCbIse8rPCSgV9PbvXazDtou1_ynA1Be5R7MR6U3vpYZ5HP41sJbZoBkhohKK_GeijQbJwMBqVHcroLcmlhJZLjY8wUQzhhHABYQNiDe2B0GTCXKXXWTUFjyGdbQFGXmLKejLIR6YUaPqA20gbwvYhsaCfvCmBjewONemzympLvUhzw-qZIEhF71L7TkN6d5KQ_PnfaqMCLnVo7ci4SMM9P-WdWoy-sLXXC4GRQmK2Qv8go54S_13hlYRTlzJ_PUKhPuOzD8DK9SWcDOdHh7C10Ldnege5IvB_ViDZdbh6rwIot5Vrd_xEIM4v3o_2dA5s1g4vV3f_GsSwDU43k2X5WuLaeCuq9RKUVzsna0oKuJB87P0ENkQeeoAbH7fSYxhUZ_XR5jSju54QbSSMnkC01-iDjfO-dCzQXSHDgHIIWAf_H450-MFC8BoWRll1SgG4w==
2022/10/13 06:40:43 GET /artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JDT1xsTt_IhY1GFGXPSDYEuJPUXsUg3W7_BwatIXtOIj7Ov3RkXpQgydp6ObPimmI6v3klgPjkna4EdqZKi862alaYAZwSBl7TZyuUlFtQNKry7lQlcZ39mjRqXE-ACJDybboFob-_rmgvjGDs1fS9J5aZ9scurOCowF-gu-OSwo8uK5QrVb56RDHnlAbko9Ry6bLq1_31M5JCQvO9hbqlomUgnMYWqyCZ43ojCdaQqq_N7DHCr5qknqJ33bzbCEaJgxNsLeWPCCbIse8rPCSgV9PbvXazDtou1_ynA1Be5R7MR6U3vpYZ5HP41sJbZoBkhohKK_GeijQbJwMBqVHcroLcmlhJZLjY8wUQzhhHABYQNiDe2B0GTCXKXXWTUFjyGdbQFGXmLKejLIR6YUaPqA20gbwvYhsaCfvCmBjewONemzympLvUhzw-qZIEhF71L7TkN6d5KQ_PnfaqMCLnVo7ci4SMM9P-WdWoy-sLXXC4GRQmK2Qv8go54S_13hlYRTlzJ_PUKhPuOzD8DK9SWcDOdHh7C10Ldnege5IvB_ViDZdbh6rwIot5Vrd_xEIM4v3o_2dA5s1g4vV3f_GsSwDU43k2X5WuLaeCuq9RKUVzsna0oKuJB87P0ENkQeeoAbH7fSYxhUZ_XR5jSju54QbSSMnkC01-iDjfO-dCzQXSHDgHIIWAf_H450-MFC8BoWRll1SgG4w== HTTP/1.1
Host: asia-east1-docker.pkg.dev
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Referer: https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:69547dffc18fcddd4f41b7696a046ad7252ceeabce944106da94fa4bfb3c6b24
Accept-Encoding: gzip


2022/10/13 06:40:43 <-- 200 https://asia-east1-docker.pkg.dev/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JDT1xsTt_IhY1GFGXPSDYEuJPUXsUg3W7_BwatIXtOIj7Ov3RkXpQgydp6ObPimmI6v3klgPjkna4EdqZKi862alaYAZwSBl7TZyuUlFtQNKry7lQlcZ39mjRqXE-ACJDybboFob-_rmgvjGDs1fS9J5aZ9scurOCowF-gu-OSwo8uK5QrVb56RDHnlAbko9Ry6bLq1_31M5JCQvO9hbqlomUgnMYWqyCZ43ojCdaQqq_N7DHCr5qknqJ33bzbCEaJgxNsLeWPCCbIse8rPCSgV9PbvXazDtou1_ynA1Be5R7MR6U3vpYZ5HP41sJbZoBkhohKK_GeijQbJwMBqVHcroLcmlhJZLjY8wUQzhhHABYQNiDe2B0GTCXKXXWTUFjyGdbQFGXmLKejLIR6YUaPqA20gbwvYhsaCfvCmBjewONemzympLvUhzw-qZIEhF71L7TkN6d5KQ_PnfaqMCLnVo7ci4SMM9P-WdWoy-sLXXC4GRQmK2Qv8go54S_13hlYRTlzJ_PUKhPuOzD8DK9SWcDOdHh7C10Ldnege5IvB_ViDZdbh6rwIot5Vrd_xEIM4v3o_2dA5s1g4vV3f_GsSwDU43k2X5WuLaeCuq9RKUVzsna0oKuJB87P0ENkQeeoAbH7fSYxhUZ_XR5jSju54QbSSMnkC01-iDjfO-dCzQXSHDgHIIWAf_H450-MFC8BoWRll1SgG4w== (300.761215ms)
2022/10/13 06:40:43 HTTP/2.0 200 OK
Content-Length: 1916
Accept-Ranges: bytes
Cache-Control: private, max-age=0
Content-Type: application/octet-stream
Date: Thu, 13 Oct 2022 06:40:43 GMT
Expires: Thu, 13 Oct 2022 06:40:43 GMT
Server: UploadServer
X-Goog-Hash: crc32c=2RHsDQ==
X-Guploader-Uploadid: ADPycdsfafDpHCLv9Y25Ty5k-rT2lv5I6bBFLXlpzKbWQ8KhdqLaAjKr668Z2dErAhyEX0M1e7yjeyYnTNw_vrCfNRqlgQ

{"architecture":"amd64","author":"Bazel","config":{"User":"65532:65532","Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"],"Entrypoint":["/kube-webhook-certgen"],"WorkingDir":"/","OnBuild":null},"created":"2022-08-01T08:13:20.280753359Z","history":[{"created":"1970-01-01T00:00:00Z","created_by":"bazel build ...","author":"Bazel"},{"created":"2022-08-01T08:13:20.280753359Z","created_by":"ARG BUILDPLATFORM","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2022-08-01T08:13:20.280753359Z","created_by":"ARG TARGETARCH","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2022-08-01T08:13:20.280753359Z","created_by":"WORKDIR /","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2022-08-01T08:13:20.280753359Z","created_by":"COPY /workspace/kube-webhook-certgen /kube-webhook-certgen # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2022-08-01T08:13:20.280753359Z","created_by":"USER 65532:65532","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2022-08-01T08:13:20.280753359Z","created_by":"ENTRYPOINT [\"/kube-webhook-certgen\"]","comment":"buildkit.dockerfile.v0","empty_layer":true}],"moby.buildkit.buildinfo.v1":"eyJmcm9udGVuZCI6ImRvY2tlcmZpbGUudjAiLCJzb3VyY2VzIjpbeyJ0eXBlIjoiZG9ja2VyLWltYWdlIiwicmVmIjoiZG9ja2VyLmlvL2xpYnJhcnkvZ29sYW5nOjEuMTguMiIsInBpbiI6InNoYTI1NjowNGZhYjVhYWY0ZmMxOGM0MDM3OTkyNDY3NDQ5MWQ5ODhhZjNkOWU5NzQ4NzQ3MmU2NzRkMGI1ZmQ4MzdkZmFjIn0seyJ0eXBlIjoiZG9ja2VyLWltYWdlIiwicmVmIjoiZ2NyLmlvL2Rpc3Ryb2xlc3Mvc3RhdGljOm5vbnJvb3QiLCJwaW4iOiJzaGEyNTY6NTlkOTFhMTdkYmRkOGI3ODVlNjFkYTgxYzkwOTViNzgwOTljYWQ4ZDc3NTdjYzEwOGY0OWU0ZmI1NjRlZjhiMyJ9XX0=","os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:8d7366c22fd8219bfcfb61ed28457854c80e310b0d736b67861b2ea7fcd77843","sha256:2e26ea8c648c9206ff3500470e6bbfe8b1591a09147d90d1450a260a1f11e82d"]}}
2022/10/13 06:40:43 --> GET https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:b9f88661235d25835ef747dab426861d51c4e9923b92623d422d7ac58eb123e9 [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:43 GET /v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:b9f88661235d25835ef747dab426861d51c4e9923b92623d422d7ac58eb123e9 HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Accept-Encoding: gzip


2022/10/13 06:40:43 <-- 307 https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:b9f88661235d25835ef747dab426861d51c4e9923b92623d422d7ac58eb123e9 (60.050456ms) [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:43 HTTP/2.0 307 Temporary Redirect
Content-Length: 211
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 13 Oct 2022 06:40:43 GMT
Location: https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:b9f88661235d25835ef747dab426861d51c4e9923b92623d422d7ac58eb123e9
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 7afdd7a1608c555600bb980b8877ae8b


2022/10/13 06:40:43 --> GET https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:b9f88661235d25835ef747dab426861d51c4e9923b92623d422d7ac58eb123e9 [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:43 GET /v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:b9f88661235d25835ef747dab426861d51c4e9923b92623d422d7ac58eb123e9 HTTP/1.1
Host: asia-east1-docker.pkg.dev
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Referer: https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:b9f88661235d25835ef747dab426861d51c4e9923b92623d422d7ac58eb123e9
Accept-Encoding: gzip


2022/10/13 06:40:43 <-- 302 https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:b9f88661235d25835ef747dab426861d51c4e9923b92623d422d7ac58eb123e9 (59.330986ms) [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:43 HTTP/2.0 302 Found
Content-Length: 828
Content-Type: text/html; charset=utf-8
Date: Thu, 13 Oct 2022 06:40:43 GMT
Docker-Distribution-Api-Version: registry/2.0
Location: /artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JAozZQL7KoCxqf4UbWrbyRWcNChS-NahLcajA3DjXu_D5mVkzEHGIp5JcneGXjZEUoakbrdE-Ccjw3L0Dj8-6D8q3ZTHkNTCLOzTAVXSUM7mmORsbQvQW0sudndsniyFMhwLAxFahMmfGz9OHNbOT2tpRTFu5j1dqfaOwt3ZySveUtEqAt6M_GnGMZFX78Wpt27i2V3K8QTEUBtBo5XCRj4YGKzNq1P2GpPWc5PrGONB_ZAFk2It8SdZg0thAb_Zg9_VBipiwrmLbvOrdare2J10Xq2h_AUsAZQhhz6BMVmUrRAAaNMQpGysXkkpRRmB9VeeGXdTZhAX07UcFgJ40xM4c7j0DKEggFsO2Kto9WX0PzaWvotM3_tz8MboE679ioUibw7jXtUsjTzb35n7hmttYEcuQi52EfTXHvpeYt-MvPAuBg4QqNd_3UPoGVlTEqJXLGm6LsHI_KhGLbeBc3A5-KjKqbGa7ca8BXugzDApTS5tGC-p-jGDTMVqgVsg-HpcO6qiNEmHVzG2eOfcbmE28mbsNOH5-TVnrbqjqwHDbFBMIDiACWcQJHtnUSa0UuArBcVh7yLLYH68FfRaGmMmPN80fDqySknYcNOV-23zyXqS-HRMGMBsG-B2GpCCMq1iqAHEAuRSs1Zg3aBvHBCTdMVLt01IVUGp5ECGZAfzyFSofBtrWq7oekjgOx22bANr3CsgoAb-Q==
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0


2022/10/13 06:40:43 --> GET https://asia-east1-docker.pkg.dev/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JAozZQL7KoCxqf4UbWrbyRWcNChS-NahLcajA3DjXu_D5mVkzEHGIp5JcneGXjZEUoakbrdE-Ccjw3L0Dj8-6D8q3ZTHkNTCLOzTAVXSUM7mmORsbQvQW0sudndsniyFMhwLAxFahMmfGz9OHNbOT2tpRTFu5j1dqfaOwt3ZySveUtEqAt6M_GnGMZFX78Wpt27i2V3K8QTEUBtBo5XCRj4YGKzNq1P2GpPWc5PrGONB_ZAFk2It8SdZg0thAb_Zg9_VBipiwrmLbvOrdare2J10Xq2h_AUsAZQhhz6BMVmUrRAAaNMQpGysXkkpRRmB9VeeGXdTZhAX07UcFgJ40xM4c7j0DKEggFsO2Kto9WX0PzaWvotM3_tz8MboE679ioUibw7jXtUsjTzb35n7hmttYEcuQi52EfTXHvpeYt-MvPAuBg4QqNd_3UPoGVlTEqJXLGm6LsHI_KhGLbeBc3A5-KjKqbGa7ca8BXugzDApTS5tGC-p-jGDTMVqgVsg-HpcO6qiNEmHVzG2eOfcbmE28mbsNOH5-TVnrbqjqwHDbFBMIDiACWcQJHtnUSa0UuArBcVh7yLLYH68FfRaGmMmPN80fDqySknYcNOV-23zyXqS-HRMGMBsG-B2GpCCMq1iqAHEAuRSs1Zg3aBvHBCTdMVLt01IVUGp5ECGZAfzyFSofBtrWq7oekjgOx22bANr3CsgoAb-Q== [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:43 GET /artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JAozZQL7KoCxqf4UbWrbyRWcNChS-NahLcajA3DjXu_D5mVkzEHGIp5JcneGXjZEUoakbrdE-Ccjw3L0Dj8-6D8q3ZTHkNTCLOzTAVXSUM7mmORsbQvQW0sudndsniyFMhwLAxFahMmfGz9OHNbOT2tpRTFu5j1dqfaOwt3ZySveUtEqAt6M_GnGMZFX78Wpt27i2V3K8QTEUBtBo5XCRj4YGKzNq1P2GpPWc5PrGONB_ZAFk2It8SdZg0thAb_Zg9_VBipiwrmLbvOrdare2J10Xq2h_AUsAZQhhz6BMVmUrRAAaNMQpGysXkkpRRmB9VeeGXdTZhAX07UcFgJ40xM4c7j0DKEggFsO2Kto9WX0PzaWvotM3_tz8MboE679ioUibw7jXtUsjTzb35n7hmttYEcuQi52EfTXHvpeYt-MvPAuBg4QqNd_3UPoGVlTEqJXLGm6LsHI_KhGLbeBc3A5-KjKqbGa7ca8BXugzDApTS5tGC-p-jGDTMVqgVsg-HpcO6qiNEmHVzG2eOfcbmE28mbsNOH5-TVnrbqjqwHDbFBMIDiACWcQJHtnUSa0UuArBcVh7yLLYH68FfRaGmMmPN80fDqySknYcNOV-23zyXqS-HRMGMBsG-B2GpCCMq1iqAHEAuRSs1Zg3aBvHBCTdMVLt01IVUGp5ECGZAfzyFSofBtrWq7oekjgOx22bANr3CsgoAb-Q== HTTP/1.1
Host: asia-east1-docker.pkg.dev
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Referer: https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:b9f88661235d25835ef747dab426861d51c4e9923b92623d422d7ac58eb123e9
Accept-Encoding: gzip


2022/10/13 06:40:44 <-- 200 https://asia-east1-docker.pkg.dev/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JAozZQL7KoCxqf4UbWrbyRWcNChS-NahLcajA3DjXu_D5mVkzEHGIp5JcneGXjZEUoakbrdE-Ccjw3L0Dj8-6D8q3ZTHkNTCLOzTAVXSUM7mmORsbQvQW0sudndsniyFMhwLAxFahMmfGz9OHNbOT2tpRTFu5j1dqfaOwt3ZySveUtEqAt6M_GnGMZFX78Wpt27i2V3K8QTEUBtBo5XCRj4YGKzNq1P2GpPWc5PrGONB_ZAFk2It8SdZg0thAb_Zg9_VBipiwrmLbvOrdare2J10Xq2h_AUsAZQhhz6BMVmUrRAAaNMQpGysXkkpRRmB9VeeGXdTZhAX07UcFgJ40xM4c7j0DKEggFsO2Kto9WX0PzaWvotM3_tz8MboE679ioUibw7jXtUsjTzb35n7hmttYEcuQi52EfTXHvpeYt-MvPAuBg4QqNd_3UPoGVlTEqJXLGm6LsHI_KhGLbeBc3A5-KjKqbGa7ca8BXugzDApTS5tGC-p-jGDTMVqgVsg-HpcO6qiNEmHVzG2eOfcbmE28mbsNOH5-TVnrbqjqwHDbFBMIDiACWcQJHtnUSa0UuArBcVh7yLLYH68FfRaGmMmPN80fDqySknYcNOV-23zyXqS-HRMGMBsG-B2GpCCMq1iqAHEAuRSs1Zg3aBvHBCTdMVLt01IVUGp5ECGZAfzyFSofBtrWq7oekjgOx22bANr3CsgoAb-Q== (804.088809ms) [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:44 HTTP/2.0 200 OK
Content-Length: 804101
Accept-Ranges: bytes
Cache-Control: private, max-age=0
Content-Type: application/octet-stream
Date: Thu, 13 Oct 2022 06:40:43 GMT
Expires: Thu, 13 Oct 2022 06:40:43 GMT
Server: UploadServer
X-Goog-Hash: crc32c=OB71IQ==
X-Guploader-Uploadid: ADPycdvcEx9F8BjJUxynltTjcuTIYw0qThNw0C0th2ooh4c04a4ndQejnV8fDUsCL_B3wmNR-GOdfEszkuiSldnd-sWP-A


2022/10/13 06:40:44 --> GET https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:23691ac0df28d0d74653810f3cde139697129a2b46102af61f38691bb4262422 [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:44 GET /v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:23691ac0df28d0d74653810f3cde139697129a2b46102af61f38691bb4262422 HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Accept-Encoding: gzip


2022/10/13 06:40:44 <-- 307 https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:23691ac0df28d0d74653810f3cde139697129a2b46102af61f38691bb4262422 (60.830735ms) [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:44 HTTP/2.0 307 Temporary Redirect
Content-Length: 211
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 13 Oct 2022 06:40:44 GMT
Location: https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:23691ac0df28d0d74653810f3cde139697129a2b46102af61f38691bb4262422
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: eb40c884c0588b95057ca05f9ba2a4f5


2022/10/13 06:40:44 --> GET https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:23691ac0df28d0d74653810f3cde139697129a2b46102af61f38691bb4262422 [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:44 GET /v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:23691ac0df28d0d74653810f3cde139697129a2b46102af61f38691bb4262422 HTTP/1.1
Host: asia-east1-docker.pkg.dev
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Referer: https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/blobs/sha256:23691ac0df28d0d74653810f3cde139697129a2b46102af61f38691bb4262422
Accept-Encoding: gzip


2022/10/13 06:40:44 <-- 302 https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:23691ac0df28d0d74653810f3cde139697129a2b46102af61f38691bb4262422 (68.374742ms) [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:44 HTTP/2.0 302 Found
Content-Length: 832
Content-Type: text/html; charset=utf-8
Date: Thu, 13 Oct 2022 06:40:44 GMT
Docker-Distribution-Api-Version: registry/2.0
Location: /artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JDQWJAzfBapCy0oxvjk-a8yyT-5mdMnGFT4yYG8MDaS5HGhFaF-Du2W61e2SYNMTcHVk7avT_2-2H0q_pYAeLV2gc-h_yt8l64UJyN9iy2vZxJaKXoVPB4US026fuKxz7Yy4mx6AJUFah3wR6qPDOc0VTcSXuL5fBpgteM7kIZA1__mD8le3l0gTI71K6olKbAazie7Jks1Qhe5kURl_fYmaiVLiSqBZ-pitDf5mSyRyJ_iQ4qyPgGfUtiph9WdDBR1xEVKjj13p73i3iYIwX1KGaR6KUtjJ7ZdWcXBYmTwioVu8ylsk3_WppZCBPdmTuFmCNivyjjMlYja-wpOIbJ3k-Ez5G-17WwvUxve61kCSUwZWh3uWYLOOo9h3UOL6f3lPZ8r-xT0Y3yaYXYz0f54RzUo2lOxrYAoLrmI2S4b5XCcR_ynTbv7X44KDgej1tsstZ0Eo0kNyqjQJjcYDbVLkk8mbbup7J77ssqGzLk2NF3lmS65Jt5JH_4L0D5nf56qfqu-wx7VXwVs3a_A8iHOzVY_0h46BOOnEqtgQdcSwsbVKoKe4yThP0wN3OLuJZrORmgNYEg-GuL850LAfeKax3fcNmmyUY7TtKoiYaPEsGWvfnqzPVUaCEp5aOeVPIE_6b0esexE49X5rbgGdRjcQT_Bi3ig4dusmKOXKumoki8ark2Gf9rramw4PwCN6E0u33wW8u62SPGycbI=
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0


2022/10/13 06:40:44 --> GET https://asia-east1-docker.pkg.dev/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JDQWJAzfBapCy0oxvjk-a8yyT-5mdMnGFT4yYG8MDaS5HGhFaF-Du2W61e2SYNMTcHVk7avT_2-2H0q_pYAeLV2gc-h_yt8l64UJyN9iy2vZxJaKXoVPB4US026fuKxz7Yy4mx6AJUFah3wR6qPDOc0VTcSXuL5fBpgteM7kIZA1__mD8le3l0gTI71K6olKbAazie7Jks1Qhe5kURl_fYmaiVLiSqBZ-pitDf5mSyRyJ_iQ4qyPgGfUtiph9WdDBR1xEVKjj13p73i3iYIwX1KGaR6KUtjJ7ZdWcXBYmTwioVu8ylsk3_WppZCBPdmTuFmCNivyjjMlYja-wpOIbJ3k-Ez5G-17WwvUxve61kCSUwZWh3uWYLOOo9h3UOL6f3lPZ8r-xT0Y3yaYXYz0f54RzUo2lOxrYAoLrmI2S4b5XCcR_ynTbv7X44KDgej1tsstZ0Eo0kNyqjQJjcYDbVLkk8mbbup7J77ssqGzLk2NF3lmS65Jt5JH_4L0D5nf56qfqu-wx7VXwVs3a_A8iHOzVY_0h46BOOnEqtgQdcSwsbVKoKe4yThP0wN3OLuJZrORmgNYEg-GuL850LAfeKax3fcNmmyUY7TtKoiYaPEsGWvfnqzPVUaCEp5aOeVPIE_6b0esexE49X5rbgGdRjcQT_Bi3ig4dusmKOXKumoki8ark2Gf9rramw4PwCN6E0u33wW8u62SPGycbI= [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:44 GET /artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JDQWJAzfBapCy0oxvjk-a8yyT-5mdMnGFT4yYG8MDaS5HGhFaF-Du2W61e2SYNMTcHVk7avT_2-2H0q_pYAeLV2gc-h_yt8l64UJyN9iy2vZxJaKXoVPB4US026fuKxz7Yy4mx6AJUFah3wR6qPDOc0VTcSXuL5fBpgteM7kIZA1__mD8le3l0gTI71K6olKbAazie7Jks1Qhe5kURl_fYmaiVLiSqBZ-pitDf5mSyRyJ_iQ4qyPgGfUtiph9WdDBR1xEVKjj13p73i3iYIwX1KGaR6KUtjJ7ZdWcXBYmTwioVu8ylsk3_WppZCBPdmTuFmCNivyjjMlYja-wpOIbJ3k-Ez5G-17WwvUxve61kCSUwZWh3uWYLOOo9h3UOL6f3lPZ8r-xT0Y3yaYXYz0f54RzUo2lOxrYAoLrmI2S4b5XCcR_ynTbv7X44KDgej1tsstZ0Eo0kNyqjQJjcYDbVLkk8mbbup7J77ssqGzLk2NF3lmS65Jt5JH_4L0D5nf56qfqu-wx7VXwVs3a_A8iHOzVY_0h46BOOnEqtgQdcSwsbVKoKe4yThP0wN3OLuJZrORmgNYEg-GuL850LAfeKax3fcNmmyUY7TtKoiYaPEsGWvfnqzPVUaCEp5aOeVPIE_6b0esexE49X5rbgGdRjcQT_Bi3ig4dusmKOXKumoki8ark2Gf9rramw4PwCN6E0u33wW8u62SPGycbI= HTTP/1.1
Host: asia-east1-docker.pkg.dev
User-Agent: crane/7268da01c46ecfb72a3ef6f44d0393f650879663 go-containerregistry/(devel)
Referer: https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/blobs/sha256:23691ac0df28d0d74653810f3cde139697129a2b46102af61f38691bb4262422
Accept-Encoding: gzip


2022/10/13 06:40:45 <-- 200 https://asia-east1-docker.pkg.dev/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AD_f-JDQWJAzfBapCy0oxvjk-a8yyT-5mdMnGFT4yYG8MDaS5HGhFaF-Du2W61e2SYNMTcHVk7avT_2-2H0q_pYAeLV2gc-h_yt8l64UJyN9iy2vZxJaKXoVPB4US026fuKxz7Yy4mx6AJUFah3wR6qPDOc0VTcSXuL5fBpgteM7kIZA1__mD8le3l0gTI71K6olKbAazie7Jks1Qhe5kURl_fYmaiVLiSqBZ-pitDf5mSyRyJ_iQ4qyPgGfUtiph9WdDBR1xEVKjj13p73i3iYIwX1KGaR6KUtjJ7ZdWcXBYmTwioVu8ylsk3_WppZCBPdmTuFmCNivyjjMlYja-wpOIbJ3k-Ez5G-17WwvUxve61kCSUwZWh3uWYLOOo9h3UOL6f3lPZ8r-xT0Y3yaYXYz0f54RzUo2lOxrYAoLrmI2S4b5XCcR_ynTbv7X44KDgej1tsstZ0Eo0kNyqjQJjcYDbVLkk8mbbup7J77ssqGzLk2NF3lmS65Jt5JH_4L0D5nf56qfqu-wx7VXwVs3a_A8iHOzVY_0h46BOOnEqtgQdcSwsbVKoKe4yThP0wN3OLuJZrORmgNYEg-GuL850LAfeKax3fcNmmyUY7TtKoiYaPEsGWvfnqzPVUaCEp5aOeVPIE_6b0esexE49X5rbgGdRjcQT_Bi3ig4dusmKOXKumoki8ark2Gf9rramw4PwCN6E0u33wW8u62SPGycbI= (679.165989ms) [body redacted: omitting binary blobs from logs]
2022/10/13 06:40:45 HTTP/2.0 200 OK
Content-Length: 18581579
Accept-Ranges: bytes
Cache-Control: private, max-age=0
Content-Type: application/octet-stream
Date: Thu, 13 Oct 2022 06:40:44 GMT
Expires: Thu, 13 Oct 2022 06:40:44 GMT
Server: UploadServer
X-Goog-Hash: crc32c=IB+h8w==
X-Guploader-Uploadid: ADPycdtRyf5jgmTVAMwzrDa_OwuWl0FXg0l5nEwviapyOiDvDZrZp-SpTPV_G0agWA_BRLU_DVgMfNxCbkVXt_Ee4S_lew

@BenTheElder
Copy link
Member Author

kubernetes/k8s.io#4337 I've audited all of the regions, AFAICT this was the only affected region.

@ameukam ameukam mentioned this issue Oct 13, 2022
Merged
ameukam added a commit to ameukam/k8s.io that referenced this issue Oct 13, 2022
@ameukam
registry.k8s.io: remove asia-southeast1
55c74df 
Related to: kubernetes/registry.k8s.io#116

The Cloud Run service in this region pointing at the matching AR region, but we have no AR instance there.
Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
@winggundamth
Copy link

Thank you @BenTheElder. Everything is working now

@BenTheElder
Copy link
Member Author

Thanks for the report and for confirming!

@BenTheElder
Copy link
Member Author

fix was merged in kubernetes/k8s.io#4352

@BenTheElder BenTheElder mentioned this issue Oct 13, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BenTheElder BenTheElder

@ameukam ameukam

Labels
kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@BenTheElder @ameukam @winggundamth @k8s-ci-robot