Private registry push reports "blob upload unknown" in client even though data gets pushed correctly #2225
Comments
|
Any news on this? |
|
Try adding |
|
I had the same problem with Nginx reverse-proxy behind Amazon ELB (doing ssl termination). Forcing the protocol to |
|
@AndreaGiardini you just saved my life! I spent a whole morning on this issue (with the exact same configuration... ELB -> Nginx -> Registry)! ❤️ |
|
@AndreaGiardini I think I might be running into the same thing, but I'm not a nginx expert, would you mind posting your config? or at least the portion you added? Here's what i've done: |
|
@Nick-Harvey I think you should modify |
|
@AndreaGiardini thank you! This was the fix for me as well going from an F5 which terminates SSL for the client, on to nginx via http and then the docker registry. |
|
I'm seeing a similar issue with a docker registry (nexus3) behind AWS CloudFront. I've set the X-Forwarded-Proto, but the push still comes back with "unknown blob". After some minutes worth of delay (and all components are already pushed), the command completes successfully. |
|
I have the same issue, but I'm running Artifactory behind a Nginx. Nginx (SSL)(https) -> Artifactory (http) |
|
@dignajar if you contact JFrog support, we'll be happy to work with you on this one. That said, this issue feels like a hokey response from the client to a network layer glitch. Seems like if there's an issue other than network layer config, its client-side not server-side (and obviously an edge case, and possibly just bad feedback). The NGINX config you provided looks like the default one we generate (although I haven't checked it line by line) and generally that works (if its not the default NGINX we generate, use the auto-generated one), but there can be additional network-layer issues that can complicate it depending on your specific configuration. |
|
[Opened a new issue #2862 and moved my comments there because it seems like this might be common to several open issues.] |
|
I am also facing the issue behind the nginx ingress. Push works fine on single replica registry but when i increase the number of replica, it gives the unknown blob upload message after waiting and push. My setup is external docker client -> nginx-ingress -> registry ( 2 replica). |
|
I also faced the same issue as @ritarya mentioned. When I run private repo with multiple replicas, docker image push was keep retrying and failing. When I reduced # of replicas to 1, issue immediately went away... |
may be its another issue with REGISTRY_HTTP_SECRET is not same for all replicas |
|
We were experimenting with unauthenticated registry for testing (no secrets). We were using local file system, and I think that is why it was causing the issue. The issue went away after we moved to using Azure Storage for backing store. |
|
In case anyone lands on this with an Apache reverse proxy, here's what fixed the issue on my setup: |
Genio, un crack |
Thanks @jsumners. You saved my life. |
worked! :D |
Have you set service.spec.sessionAffinity to ClientIP? https://kubernetes.io/docs/concepts/services-networking/service/ |
|
If you're using multiple instances behind a load balancer, please make sure to have the same See https://docs.docker.com/registry/configuration/#http for
|
|
Just for posterity, I was using Cloudflare and getting this, moving to use our own traefik router fixed it. |
|
I'm running an haproxy with nginx under docker swarm cluster and I had to define following in the configs : In nginx conf : And following in the haproxy.cfg : And now, no more "unkown blob" issues. Thanks guys! |
|
Just incase anyone else runs into this and they confirmed that their X-Forwarded-Proto is set correctly and you have more than one registry instance/pod running. Check the following two things.
|
|
@ritarya did you find a solution if so could you please put it here? I'm running into the exact same issue. When I have more than two replicas for my private docker registry it gives the unknown blob upload message. When I have one all works just fine. I'm mounting a filestore from GCP to store the docker images |
|
@maartenschalekamp If Im using filestore from GCP which is a managed service. Do you know, How I can enable the flag no_wdelay ? |
|
I have same problem with NFS share when running on multiple instances (ingress controller). Scaling down to 1 replica solved that. I mean having multiple replicas would be awesome but I kind of of made peace with just one. |
|
I had the same problem when I run multiple instances of registry |
|
We recently had this issue with 2.8.3 (most recent registry image at time of writing) and found many issues can trigger this. |
and others
My setup is following:
Previously this set-up was working when I didn't use https, I have the loadbalancer expose the registry as http on port 5000 and I can use it with "localhost:5000" address (on each machine where loadbalancer runs) - in this scenario pushing and using the images works fine.
Now I'm trying to push image to registry through https endpoint:
$ docker push images.bigkuber.inside.datax.pl/kskalski-dataflowsbut I'm getting
However even though push fails, the image is actually uploaded correctly, I can get its status and use it by new containers:
I enabled debug on client docker, here is an exempt from relevant time interval:
As could be expected, registry server is not reporting any problems:
I suspect something is configured incorrectly in loadbalancer doing https termination, since going through localhost:5000 works well. There must be some specific interaction of client and server that happens only during push, but not in any other operation, such that loadbalancer's https serving causes problems.
The text was updated successfully, but these errors were encountered: