Cannot pull from docker.io. error unmarshalling content: invalid character '<' looking for beginning of value #23643
Replies: 1 comment 1 reply
|
The '<' means Harbor got an HTML page where it expected JSON - something in the path answered with an error/block page instead of a registry response. Combined with "quay.io works, docker.io doesn't" in an egress-restricted subnet, I'd look at the firewall rather than Harbor's nginx. The relevant difference: quay.io serves auth and registry API from one host, while docker.io splits them - tokens come from auth.docker.io, the API from registry-1.docker.io, blobs from Cloudflare-backed hosts. If your allow-list only covers registry-1.docker.io, the token request gets intercepted and your proxy answers with an HTML block page. The "artifact not found" your clients see is just the downstream symptom of that failed token fetch, which also explains why the endpoint healthcheck passes - it doesn't exercise the token flow. Quickest check, from the Harbor host: curl -sS "https://auth.docker.io/token?service=registry.docker.io&scope=repository:alpine/ansible:pull" | head -c 200 If that returns HTML instead of a JSON token, you've found it - and the fix is on the firewall allow-list, not in Harbor. |
|
Thank you very much for your reply! Despite all my checks in the firewall itself, it being in the way was still in the back of my mind. However, with the curl, I can confirm that it is not the firewall, as I get a JSON response: Frankly, with how many headaches that firewall has caused, I am somewhat surprised too. With this, and a bit more searching of my own, I am pretty sure it's something in the nginx configuration, as others have experienced the same error in nginx. Never with nginx being the cause of it, sadly. I've attached our nginx.conf below if you want to take a look. I've also tried to find any dedicated nginx logging, but failed; /var/log/nginx doesn't exist and journalctl -xeu nginx returns no entries. Any ideas on what I can try/investigate to get more info? Anything in the config that could be the culprit? Any advice/tips are greatly appreciated! nginx.conf |
Uh oh!
There was an error while loading. Please reload this page.
Hi everyone,
We have a self-hosted harbor instance to function as a proxy cache and as a filter to allow specific containers only. It has worked for quite a while, but unfortunately something is going wrong when pulling from docker.io specifically. I'll list all the steps I've taken and my findings below, as well as some more context, but I want to preface this post with the following: I'm not the one who set the instance up, so I have a few blind spots with regards to configuration details. I will do my best to provide more info upon request as I am certain I will have missed some things. What I want out of this discussion is some pointers, and eventually a solution.
Setup
So, I guess I'll start with our environment setup. We have an internal subnet with heavily restricted access to the outside. Harbor sits within that subnet with slightly more priviliges, in that it is allowed to access container repositories (docker hub, quay.io, etc.). Harbor version 2.15.2 is deployed as Podman containers on a single server, with a persistent datadisk serving as storage. Our machines (both Windows and Linux) pulling containers from harbor all use Podman, which is configured to pull from harbor instead of the native repositories.
Error
Harbor proxy caching works fine when pulling containers from e.g. quay.io, but pulling from docker.io gives the following error:
I checked the journald entries (because /var/log/harbor did not exist) and found the following:
Failed pull from docker.io
For comparison, here's the log for a successful pull from quay.io:
Successful pull from quay.io
I gather from the logs that it is getting an aberrant response somewhere along the line, but I cannot figure out where. Interference from our firewall is unlikely, as I see no blockings anywhere, but I don't know where else to look.
Some other things I've checked
I checked our harbor instance via the GUI, and none of the configured projects in docker.io have any artifacts. This is not the case for other repositories.
docker.io project contents
I double-checked the proxy cache configuration, and it all looks fine to me. The healthchecks all succeed too.
docker.io registry endpoint
I also checked the "Logs" section on the harbor GUI, but it only contains a history of when things were pulled, so no useful info for my issue.
My current theory is that there is some issue with the nginx proxy that gives a wrong response to the harbor client, but I have no clue how to debug the nginx client that's baked into harbor. Can anyone give me some pointers? Alternatively, does anyone have other suggestions for possible explenations, and some tips on where to look into them? Any feedback is greatly appreciated. Also, if further info is necessary, don't hestitate to ask. I will do my best to provide it!
All reactions