Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Getting 403 Forbidden when trying to pull an image #211

Closed
2 tasks done
hh74 opened this issue Mar 26, 2023 · 4 comments
Closed
2 tasks done

Getting 403 Forbidden when trying to pull an image #211

hh74 opened this issue Mar 26, 2023 · 4 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra.

Comments

@hh74
Copy link

hh74 commented Mar 26, 2023

Is there an existing issue for this?

  • I have searched the existing issues

What did you expect to happen?

The image should be pulled.

Debugging Information

root@eu-f0:~# dig registry.k8s.io

; <<>> DiG 9.18.1-1ubuntu1.3-Ubuntu <<>> registry.k8s.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43511
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;registry.k8s.io.               IN      A

;; ANSWER SECTION:
registry.k8s.io.        2021    IN      A       34.107.244.51

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Sun Mar 26 09:59:03 UTC 2023
;; MSG SIZE  rcvd: 60

root@eu-f0:~# curl https://registry.k8s.io/v2

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>403 Forbidden</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Forbidden</h1>
<h2>Your client does not have permission to get URL <code>/v2</code> from this server.</h2>
<h2></h2>
</body></html>

crictl pull registry.k8s.io/pause:3.9

E0326 09:56:39.930099 3985030 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": pulling from host registry.k8s.io failed with status code [manifests 3.9]: 403 Forbidden" image="registry.k8s.io/pause:3.9"
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/pause:3.9": failed to resolve reference "registry.k8s.io/pause:3.9": pulling from host registry.k8s.io failed with status code [manifests 3.9]: 403 Forbidden

Anything else?

Im using ssdnodes.com vm's from europe/frankfurt.
Using a us based vm from ssdnodes work as expected.
I could provide the public non working ip's if helpful.
Seems to be some load balancing/mirroring issue.

Code of Conduct

  • I agree to follow this project's Code of Conduct
@hh74 hh74 added kind/bug Categorizes issue or PR as related to a bug. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. labels Mar 26, 2023
@dims
Copy link
Member

dims commented Mar 26, 2023

@hh74 this is similar to #138

i think you will need to engage your provider at this point:
#138 (comment)

Note, there are other options like mirroring (see discussion in that same issue as well) what you need into your own registry:
https://github.com/kubernetes/registry.k8s.io/tree/main/docs/mirroring

mirroring helps the community as it reduces the load on our registries and saves money as well. so please consider that.

@TerryHowe
Copy link
Member

Proxy settings maybe? Running curl with a -v might tell you more about who is giving you the 403.

@hh74
Copy link
Author

hh74 commented Mar 26, 2023

@dims Sorry - i have overlooked the other 403 tickets. Reading through them It seems some of my IP's are also blocked. Thats sad, but i will got for the mirroring aproach or contact my provider.

@hh74 hh74 closed this as completed Mar 26, 2023
@dims
Copy link
Member

dims commented Mar 26, 2023

@hh74 no worries, please let us know of the outcome. as there will be many more in your situation in the coming weeks. So all info are useful!

@BenTheElder BenTheElder mentioned this issue Mar 27, 2023
Merged
@deric deric mentioned this issue Oct 5, 2023
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dims @TerryHowe @hh74