Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Push to private registry fails with BLOB_UNKNOWN v2.1.1 #928
I have a private registry sitting behind a pair of nginx proxies. The first terminates the TLS, while the second simply routes any information between the registry (:443) and a separate docker container (:80). The config files for both, version info, and other relevant information is contained in the gist.
I can curl to /v2/ and /v2/_catalog just fine, but any attempt at pushing to the registry ends in a single push then a 'blob upload unknown'. Personally, I'm not sure what exactly is happening, as I had this set up working initially with 2.0 around two weeks ago, but after retesting yesterday, I receive this error, regardless of version (2.0/2.0.1/2.1.0/2.1.1).
Currently, the private registry stores to the local file system.
I'm sure it's likely something I'm glossing over, but if there's any extra info I need to supply I'll get it as quickly as possible.
It's not working from either, even through the 2 layer nginx, GET requests to /v2/ and /v2/_catalog work just fine.
As of now, cf-registry is basically a copy of registry:2.1.1.
@dmp42 It looks like the error
We can see the upload creation operation completing successfully, since it returns a 202. The call to
My theory is that something is dropping the information from the upload url on redirect. Normally, I would suspect this is an issue with the load balancer, but the configuration looks clean. This can possibly be caused by a bug in
@caladbolg7 I'd like to eliminate the configuration environment as a possible cause. Let's take a look at a few things:
@caladbolg7 There may be an issue with having two levels of nginx and correctly resolving the protocol scheme and host. It is possible that the scheme is being incorrectly set by the inner nginx instance. Based on your configuration, you may not even need the second nginx instance. I don't understand you're full environment, but it seems unnecessary.
This was referenced
Aug 27, 2015
If there's a better way to be getting the docker daemon logs that I'm missing, please let me know. Long story short, the upper layer of nginx is necessary use of a separate part of the overarching project.
Thanks again for the help.
@caladbolg7 Thanks for the extra information.
At this point, it is pretty safe to say the upload is actually there. Given the bug we've identified in the client library, I am pretty confident that may be the cause.
Although, I am slightly worried that you may still run into problems with the "double nginx" setup. If you are willing to continue troubleshooting, let's confirm the root cause of this particular issue. Do you happen to have the logs during the failure? These logs look like they only contain daemon startup. You'll have to restart the daemon in debug mode then perform the push.
I'm seeing something similar, but it only seems to happen intermittently. I have 3 servers running registry:2.1.1, all using the same NFS mount for storage. Seemingly randomly, I'll get an error like:
and the corresponding logs from the registry:
Running the 'docker push' command again will usually succeed. I've seen this happen twice so far, and it's been on a different registry instance each time.
It probably doesn't matter, but these registry instances are sitting behind nginx with a simple proxy_pass directive.