Docker push through nginx proxy fails trying to send a 32B layer

[LouisKottmann]

Hello,

I've been all day at this, and I can't seem to get a private registry fully working behind a nginx proxy.

The short version is:
I got a server that I want to use as a docker registry, on port 5004.
It should be protected via SSL and basic auth, and it should store the images in a S3 bucket.
But it is not working, I get this error when trying to push a golang image from the registry host:

[6998] (ubuntu) ~ $ docker push docker.mydomain.com/golang:1.5
The push refers to a repository [docker.mydomain.com/golang] (len: 1)
8a5c8b03ad80: Image push failed 
Error pushing to registry: Server error: 301 trying to push golang blob - sha256:819d5b5871faa444f07b897241f755310b6db9a67cae489677fb96396f414298

From a remote host, I see that it starts sending the first layer, but it's only 32 byte big and fails after writing "EOF" at the prompt.

What I got working is:

• docker login (it is not asking my password since I had already logged in)

~ master $ docker login docker.mydomain.com
Username (): myusername
WARNING: login credentials saved in /home/me/.docker/config.json
Login Succeeded

• Pushing locally and storing on S3

I checked by ssh'ing to my server, retagging an image to localhost:5004/busybox:latest, adding the --insecure-registry switch to DOCKER_OPTS and then pushing the newly tagged image.
That works just fine, and on the AWS console I can see that the S3 bucket has new files.
So the credentials in place are working, and the registry can access the bucket.

• Hitting the API (http is redirected to https by nginx)

~/... $ curl -L --user username:password http://docker.mydomain.com/v2/_catalog
{"repositories":["busybox"]}

• Pulling with everything on (nginx, SSL, basic auth, S3..)

If I try to pull the image I sent using --insecure-registry from the remote host, it works!!:

~/... $ docker pull docker.mydomain.com/busybox:latest
latest: Pulling from busybox
cf2616975b4a: Already exists 
6ce2e90b0bc7: Already exists 
8c2e06607696: Already exists 
Digest: sha256:3cc6b183efb34ff773f81ce230362ef67288375fdeb9cc8d50c221820fbe5e3b
Status: Image is up to date for docker.praditus.com/busybox:latest

What is not working is:

• Pushing from a remote host with everything on (nginx, SSL, basic auth, S3..)

~/Work/Praditus/alloy xolo $ docker push docker.mydomain.com:443/ubuntu:15.04 
The push refers to a repository [docker.praditus.com:443/ubuntu] (len: 1)
013f3d01d247: Pushing [==================================================>]     32 B/32 B
EOF

Note how even though no errors are shown here, the return code is not 0.
Also note that ths output of this command is different if I type it on the registry's host (see at the top of this text).

If I remove the basic auth in nginx and delete the file /home/me/.docker/config.json, I get the same error. I think basic auth works fine, I use SSHA hashed passwords as recommended.

The nginx logs show (not necessarily in correct order since there are 2 nginxes load-balanced):

10.0.0.144 - - [10/Sep/2015:15:40:35 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.19.0-26-generic os/linux arch/amd64"
10.0.0.103 - - [10/Sep/2015:15:40:35 +0000] "HEAD /v2/ubuntu/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 404 0 "-" "docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.19.0-26-generic os/linux arch/amd64"
10.0.0.144 - - [10/Sep/2015:15:40:48 +0000] "POST /v2/ubuntu/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.19.0-26-generic os/linux arch/amd64"

The registry log shows:

time="2015-09-10T15:38:13Z" level=info msg="response completed" http.request.host=docker.mudomain.com http.request.id=c9c829f2-afe1-459b-b080-f2a1733ed5b7 http.request.method=GET http.request.remoteaddr=<my ip> http.request.uri="/v2/" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.19.0-26-generic os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.647111ms http.response.status=200 http.response.written=2 instance.id=06a59d1b-0761-4cb7-985c-e02cad4b284f version=v2.1.1 
10.0.1.231 - - [10/Sep/2015:15:38:13 +0000] "GET /v2/ HTTP/1.0" 200 2 "" "docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.19.0-26-generic os/linux arch/amd64"
time="2015-09-10T15:38:13Z" level=error msg="response completed with error" err.code="BLOB_UNKNOWN" err.detail=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 err.message="blob unknown to registry" http.request.host=docker.praditus.com http.request.id=4dc0b2a3-b777-4320-a405-521ce5253138 http.request.method=HEAD http.request.remoteaddr=<my ip> http.request.uri="/v2/ubuntu/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.19.0-26-generic os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=62.800451ms http.response.status=404 http.response.written=157 instance.id=06a59d1b-0761-4cb7-985c-e02cad4b284f vars.digest="sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" vars.name=ubuntu version=v2.1.1 
10.0.1.72 - - [10/Sep/2015:15:38:13 +0000] "HEAD /v2/ubuntu/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.0" 404 157 "" "docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.19.0-26-generic os/linux arch/amd64"
time="2015-09-10T15:38:14Z" level=info msg="response completed" http.request.host=docker.mydomain.com http.request.id=6abc3b5a-ae3b-4e30-b34e-14a09c1ab6d2 http.request.method=POST http.request.remoteaddr=<my ip> http.request.uri="/v2/ubuntu/blobs/uploads/" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.19.0-26-generic os/linux arch/amd64" http.response.duration=195.187871ms http.response.status=202 http.response.written=0 instance.id=06a59d1b-0761-4cb7-985c-e02cad4b284f version=v2.1.1 
10.0.1.231 - - [10/Sep/2015:15:38:13 +0000] "POST /v2/ubuntu/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.19.0-26-generic os/linux arch/amd64"

The docker daemon on the remote host shows:

DEBU[0018] Calling POST /images/{name:.*}/push          
INFO[0018] POST /v1.20/images/docker.mydomain.com/ubuntu/push?tag=15.04 
DEBU[0018] hostDir: /etc/docker/certs.d/docker.mydomain.com 
DEBU[0018] Trying to push docker.mydomain.com/ubuntu to https://docker.mydomain.com v2 
DEBU[0018] Checking "15.04" against graph.Repository{"15.04":"013f3d01d24738964bb7101fa83a926181d600ebecca7206dced59669e6e6778"} 
DEBU[0018] Pushing repository: ubuntu:15.04             
DEBU[0018] Pushing layer: 013f3d01d24738964bb7101fa83a926181d600ebecca7206dced59669e6e6778 
DEBU[0019] [graph] TarLayer with reassembly: 013f3d01d24738964bb7101fa83a926181d600ebecca7206dced59669e6e6778 
DEBU[0019] [graph] 013f3d01d24738964bb7101fa83a926181d600ebecca7206dced59669e6e6778 is at "/var/lib/docker/aufs/mnt/013f3d01d24738964bb7101fa83a926181d600ebecca7206dced59669e6e6778" 
DEBU[0019] rendered layer for 013f3d01d24738964bb7101fa83a926181d600ebecca7206dced59669e6e6778 of [32] size 
DEBU[0019] Not continuing with error: EOF

So at this point it seems to me that the problem could come from nginx, configured as such (SSL is handled at ELB level):

    # Serve the private docker registry
    upstream docker {
        server <registry's host ip>:5004;
    }

    server {
        listen <proxied ssl port>;
        server_name docker.mydomain.com;

        access_log /home/...;

        # disable any limits to avoid HTTP 413 for large image uploads
        client_max_body_size 0;

        # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
        chunked_transfer_encoding on;

        location /v2/ {
            # Do not allow connections from docker 1.5 and earlier
            # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
            if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
                return 404;
            }

            # To add basic authentication to v2 use auth_basic setting plus add_header
            auth_basic "Registry realm";
            auth_basic_user_file /path/to/htpasswd;
            more_set_headers 'Docker-Distribution-Api-Version: registry/2.0';

            proxy_pass                          http://docker;
            proxy_set_header  Host              $http_host;   # required for docker client's sake
            proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
            proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header  X-Forwarded-Proto $scheme;
            proxy_read_timeout                  900;
        }
    }

I meticulously followed the instructions in the official documentation, switching add_header ... always to more_set_headers, as the discussions in the github issues indicate the former may cause issues.

I really don't see what I'm doing wrong in there.
The registry version is 2.1.1, but I tried the 2.0 as well with no better results:

... ~/... $ docker exec my_registry_v2 registry -version
registry github.com/docker/distribution v2.1.1

I started the registry by using this config.yml (verbatim):

version: 0.1
log:
    fields:
        service: registry
http:
    addr: :5004
storage:
    cache:
        layerinfo: inmemory
    s3:
        accesskey: <your awsaccesskey>
        secretkey: <your awssecretkey>
        region: <your bucket region>
        bucket: <your bucketname>
        encrypt: true
        secure: true
        v4auth: true
        chunksize: 5242880
        rootdirectory: /

With this Dockerfile:

FROM registry:2.1.1

CMD ["/etc/docker/registry/config.yml"]
COPY config.yml /etc/docker/registry/config.yml

Building it:

docker build --no-cache --rm --tag my-registry:2.1.1 .

And then using this command:

docker run -d --name my_registry_v2 \
           -p 5004:5004 \
           -e "REGISTRY_STORAGE_S3_ACCESSKEY=MY_KEY_ID" \
           -e "REGISTRY_STORAGE_S3_SECRETKEY=MY_KEY" \
           -e "REGISTRY_STORAGE_S3_BUCKET=MY_BUCKET" \
           -e "REGISTRY_STORAGE_S3_REGION=MY_REGION" \
           my-registry:2.1.1

The docker clients used are in version 1.7.1 and 1.8.1.

Am I doing something wrong? Is this a bug?

Regards,
Louis Kottmann

[stevvooe]

@LouisKottmann Sorry about the frustration. Multiple layers of http proxies can be complex to configure correctly.

The issue is that the registry cannot resolve the correct host based on the forwarding headers. This is a common issue when using ELB with nginx. Basically, when the registry redirects the client to the upload location, it incorrectly issues the redirect to http, when it should be https. We can confirm this since we don't see the following PATCH and PUT calls after upload creation in the logs.

This is happening because the value of the X-Forwarded-XXX is being reset by nginx. The fix for this involves correctly resolving the hostname for the redirects by ensuring the X-Forwarded-XXX headers are set correctly.

First, confirm that ELB is running in "https" mode. In this mode, ELB will set these header for you. Once that is confirmed, remove the following lines from the nginx configuration:

            proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
            proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header  X-Forwarded-Proto $scheme;

Please give this a try and we'll go from there.

[LouisKottmann]

Holy mother of configuration gotchas, this solved it!

Thank you so much @stevvooe !
This should definitely be in the documentation I'll make a PR asap

What a nice way to start the day :)

[stevvooe]

@LouisKottmann Glad to have helped!

We'll consider this a documentation issue, so we'll leave it open until your PR arrives or we address the confusion elsewhere.

[hellertime]

Just ran into this same issue with HAProxy and the fix by @stevvooe worked there too:

option forwardfor
option forwardfor header X-Real-IP
http-request set-header X-Forwarded-Proto https if { ssl_fc }

[RichardScothern]

thanks @LouisKottmann . Closing this for now.

[manvalls]

For those of you using a TLS <--> TCP proxy, this can be solved too by using:

REGISTRY_HTTP_RELATIVEURLS=true

[dbaba]

@manvalls Very helpful info. Thanks.

Same as

http:
  relativeurls: true

in registry config.yml.

[kernelpig]

@dbaba 十分感谢，完美解决了push失败问题，🙏，

• 环境：nginx(https)-> harbor(nginx/http)，

• 错误：docker login成功，docker push报错：unauthorized: authentication required

[stevvooe]

relativeurls: true should probably be the default.

[juanluisbaptiste]

@stevvooe I was having this same issue but with traefik and your suggestion worked too, thanks !!

[Statemood]

It's worked ! Thanks all!

[blechalupe]

Set REGISTRY_HTTP_RELATIVEURLS registry environment variable to "true" worked for me with Fabio.

[karimbzu]

The following error appears when trying to push images to harbor (using helm) which is installed on top of Openshift OKD3.11.

[root@master ~]# docker push core.harbor.domain/library/alpine:v1.0
The push refers to a repository [core.harbor.domain/library/alpine]
f1b5933fe4b5: Pushing [==================================================>] 5.533 MB/5.533 MB
unknown blob

Thanks in advance

[nickolashkraus]

@stevvooe Thank you for your solution! However, there is a small, but significant, error in the documentation.

So if you have an Nginx instance sitting behind it, remove these lines from the example config below:

proxy_set_header  Host              $http_host;   # required for docker client’s sake
proxy_set_header  X-Real-IP         $remote_addr; # pass on real client’s IP
proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
proxy_set_header  X-Forwarded-Proto $scheme;

The above is incorrect. The following line should not be removed:

proxy_set_header  Host              $http_host;   # required for docker client’s sake

When this line is removed, I receive the following:

nginx

nginx_1     | 172.31.28.26 - - [05/Jun/2019:19:33:25 +0000] "HEAD /v2/alpine/blobs/sha256:055936d3920576da37aa9bc460d70c5f212028bda1c08c0879aedf03d7a66ea1 HTTP/1.1" 404 0 "-" "docker/18.09.2
 go/go1.10.6 git-commit/6247962 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.2 \x5C(darwin\x5C))"

registry

registry_1  | 172.17.0.3 - - [05/Jun/2019:19:33:25 +0000] "HEAD /v2/alpine/blobs/sha256:055936d3920576da37aa9bc460d70c5f212028bda1c08c0879aedf03d7a66ea1 HTTP/1.1" 404 157 "" "docker/18.09.2
go/go1.10.6 git-commit/6247962 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.2 \\(darwin\\))"

When only X-Real-IP, X-Forwarded-For, and X-Forwarded-Proto are removed, the push succeeds as expected.

[itsecforu]

@nickolashkraus from every location or only /v2 ?

[nickolashkraus]

@itsecforu I left the company for which this was relevant, however this was only /v2 IIRC. An NGINX server sat in front or our Docker Registry, which was used exclusively for proxying requests.