Nginx-Ingress: Docker Registry behind Ingress - docker push don't work #2444

muffin87 · 2017-03-03T12:58:31Z

Hi everyone,

im running K8s version 1.5.2 on CentOS.
I wanted to deploy a vanilla docker registry and expose the service with Nginx Ingress controller.

Issue

docker push does not work with the nginx-ingress-controller in front of the docker-registry.

It seems to me that something in the nginx-ingress-controller config is blocking the docker push payload upload (tar file).

Any idea whats happening here and how I can solve this?

Error Message:

docker-client:
Error: Status 404 trying to push repository nginx: "404 page not found\n"

dockerd
Upload failed: EOF

Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.177922934Z" level=debug msg="Pushing layer: sha256:809c8c0dd73c9081a8ab9795b5fb9beab60c7f602a0886dbbdf48e939d608163"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.401819584Z" level=debug msg="Assembling tar data for d388be1b57d4bbb37c16836b3fafba89e433a413f7448680d6992ecfe4f07089"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.416382683Z" level=debug msg="Assembling tar data for 6780c17ba8c81862656e46d0812c0eaf7fc2eb55447cf1f71b1e3e64e548256b"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.416555479Z" level=debug msg="Assembling tar data for 6c84bf0c3c99e5454f5536068db727174c4dea0add6c7c54052604dbbba4a46a"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.519050420Z" level=error msg="Upload failed, retrying: EOF"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.537034379Z" level=error msg="Upload failed, retrying: EOF"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.541124148Z" level=error msg="Upload failed, retrying: EOF"
Mar  3 11:29:53 moby root: time="2017-03-03T11:29:53.983465337Z" level=debug msg="Pushing layer: sha256:e8fa134cb7b8a5e2943c9ea6e48a90c9f7fc80fd26780ce9d74bbab927a6cfc5"
Mar  3 11:29:54 moby root: time="2017-03-03T11:29:54.227004087Z" level=debug msg="Assembling tar data for b3c45395c4b867925ed531458101d8c5cb8c5462f06a678c8132f06e2db8c588"
Mar  3 11:29:54 moby root: time="2017-03-03T11:29:54.345157833Z" level=error msg="Upload failed, retrying: EOF"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.520650325Z" level=debug msg="Pushing layer: sha256:7cbcbac42c44c6c38559e5df3a494f44987333c8023a40fec48df2fce1fc146b"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.537773800Z" level=debug msg="Pushing layer: sha256:e8d45b8ab3ca8a83ba4e340f4b901c593da00f7f2bb068937a8b32b3beb606af"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.541707653Z" level=debug msg="Pushing layer: sha256:809c8c0dd73c9081a8ab9795b5fb9beab60c7f602a0886dbbdf48e939d608163"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.744771310Z" level=debug msg="Assembling tar data for 6c84bf0c3c99e5454f5536068db727174c4dea0add6c7c54052604dbbba4a46a"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.756433036Z" level=debug msg="Assembling tar data for 6780c17ba8c81862656e46d0812c0eaf7fc2eb55447cf1f71b1e3e64e548256b"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.760178212Z" level=debug msg="Assembling tar data for d388be1b57d4bbb37c16836b3fafba89e433a413f7448680d6992ecfe4f07089"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.851262730Z" level=error msg="Upload failed: EOF"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.856225668Z" level=error msg="Upload failed: EOF"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.863372727Z" level=error msg="Upload failed: EOF"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.863480799Z" level=error msg="Attempting next endpoint for push after error: EOF"

nginx-ingress log:

<censored> - ucpapp [03/Mar/2017:11:29:20 +0000] "HEAD /v2/nginx/blobs/sha256:ba1ed986f6a0df1b3fa02fe8cf76e45e7f7ae248c2a2d9f6ce901585c38d5a79 HTTP/1.1" 404 0 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" "-"
<censored> - ucpapp [03/Mar/2017:11:29:20 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" "-"
<censored> - ucpapp [03/Mar/2017:11:29:20 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" "-"
<censored> - ucpapp [03/Mar/2017:11:29:21 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" "-"
<censored> - ucpapp [03/Mar/2017:11:29:21 +0000] "HEAD /v2/nginx/blobs/sha256:a362ec94dcabb258895c0cde3905dafd6c40df5ffa4a668fdf6b175655774192 HTTP/1.1" 404 0 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" "-"

Setup

dockerd -> nginx-ingress -> k8s-service-object -> nginx-for-auth -> registry

docker push through the nginx results in error.
Skipping the nginx-ingress and pushing to the k8s-service-object via NodePort works fine
docker login works fine
docker push manifest files end up in the registry fine

Configs

rendered nginx-config

upstream tools-docker-registry-registry.<censored>-docker-registry {

  server 10.254.4.9:80;
}

server {
  listen 80;

  server_name registry.<censored>;

  location / {
    proxy_http_version 1.1;

    proxy_connect_timeout 60s;
    proxy_read_timeout 60s;
    client_max_body_size 1m;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_buffering on;

    proxy_pass http://tools-docker-registry-registry.<censored>-docker-registry;

  }
}

registry ingress conf

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: docker-registry
  namespace: tools
spec:
  rules:
    - host: registry.<censored>
      http:
        paths:
        - backend:
            serviceName: docker-registry
            servicePort: 80

The text was updated successfully, but these errors were encountered:

aledbf · 2017-03-03T13:13:54Z

@muffin87 what ingress controller and version are you using?

muffin87 · 2017-03-03T13:18:10Z

@aledbf: I tried several versions:

nginx-ingress-controller:0.9.0-beta.2
nginx-ingress-controller:0.9.0-beta.1
nginx-ingress-controller:0.8.0

I also tried the other controller from https://github.com/nginxinc/kubernetes-ingress

aledbf · 2017-03-03T13:21:58Z

@muffin87 the nginx-config is from your nginx right?

Add the annotation ingress.kubernetes.io/proxy-body-size: 0 to the ingress rule. By default this value is 1m.

aledbf · 2017-03-03T13:22:46Z

@muffin87 in the ingress rule the service docker-registry is the real docker registry or your nginx?

andiMenge · 2017-03-03T13:31:47Z

@aledbf docker-registry in the ingress rule points to the nginx used for authentication, not the registry itself.

ingress.kubernetes.io/proxy-body-size: 0 has no effect on the issue

aledbf · 2017-03-03T13:34:26Z

@andiMenge please switch to the real docker registry to verify you can push an image (test the annotation is working)

andiMenge · 2017-03-03T13:37:05Z

@aledbf We tried this, with the same results. Actually we just implemented the nginx-for-auth just to see if it makes a difference.

If I bypass the ingress-controller and push directly to the k8s-service-object via NodePort everthing works fine

aledbf · 2017-03-03T14:05:57Z

@andiMenge please post the output of curl -v http://registry.<censored>/v2/, the nginx.conf from the ingress controller using kubectl exec <ingress pod> cat /etc/nginx/nginx.conf and the logskubectl logs <ingress pod>.

andiMenge · 2017-03-03T14:56:35Z

@aledbf I switched back to our old and simpler config. The setup looks now like this:

dockerd -> nignx-ingress-controller -> k8s-registry-service-object -> registry

nginx-ingress-controller:0.9.0-beta.1

ingress-controller log

::ffff:<censored> - [::ffff:<censored>] - ucpapp [03/Mar/2017:14:36:30 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "curl/7.51.0" 133 0.003 [tools-docker-registry-5000] 10.254.31.12:5000 2 0.003 200

curl -vkL -u user:pw http://registry.censored/v2/

*   Trying <censored>...
* TCP_NODELAY set
* Connected to registry.<censored> (<censored>) port 80 (#0)
* Server auth using Basic with user '<censored>'
> GET /v2/ HTTP/1.1
> Host: registry.<censored>
> Authorization: Basic dWNwYXBwOiwuYXBwLHVjcA==
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.11.10
< Date: Fri, 03 Mar 2017 14:36:30 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 2
< Connection: keep-alive
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
<
* Curl_http_done: called premature == 0
* Connection #0 to host registry.<censored> left intact
{}%

ingress-controller config

daemon off;

worker_processes 64;
pid /run/nginx.pid;
events {
    multi_accept        on;
    worker_connections  16384;
    use                 epoll;
}

http {
    real_ip_header      X-Forwarded-For;
    set_real_ip_from    0.0.0.0/0;
    real_ip_recursive   on;

    geoip_country       /etc/nginx/GeoIP.dat;
    geoip_city          /etc/nginx/GeoLiteCity.dat;
    geoip_proxy_recursive on;
    # lua section to return proper error codes when custom pages are used
    lua_package_path '.?.lua;./etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/lua-resty-http/lib/?.lua;';
    init_by_lua_block {
        require("error_page")
    }

    sendfile            on;
    aio                 threads;
    tcp_nopush          on;
    tcp_nodelay         on;
    log_subrequest      on;

    reset_timedout_connection on;

    keepalive_timeout 75s;

    client_header_buffer_size       1k;
    large_client_header_buffers     4 8k;

    types_hash_max_size             2048;
    server_names_hash_max_size      512;
    server_names_hash_bucket_size   64;
    map_hash_bucket_size            64;

    include /etc/nginx/mime.types;
    default_type text/html;
    gzip on;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_min_length 256;
    gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
    gzip_proxied any;

    server_tokens on;

    log_format upstreaminfo '$remote_addr - '
        '[$proxy_add_x_forwarded_for] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" '
        '$request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';

    map $request_uri $loggable {
        default 1;
    }

    access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
    error_log  /var/log/nginx/error.log notice;

    resolver 10.253.0.8 valid=30s;

    # Retain the default nginx handling of requests without a "Connection" header
    map $http_upgrade $connection_upgrade {
        default          upgrade;
        ''               close;
    }

    # trust http_x_forwarded_proto headers correctly indicate ssl offloading
    map $http_x_forwarded_proto $pass_access_scheme {
        default          $http_x_forwarded_proto;
        ''               $scheme;
    }

    map $http_x_forwarded_port $pass_server_port {
       default           $http_x_forwarded_port;
       ''                $server_port;
    }

    # map port 442 to 443 for header X-Forwarded-Port
    map $pass_server_port $pass_port {
        442              443;
        default          $pass_server_port;
    }

    # Map a response error watching the header Content-Type
    map $http_accept $httpAccept {
        default          html;
        application/json json;
        application/xml  xml;
        text/plain       text;
    }

    map $httpAccept $httpReturnType {
        default          text/html;
        json             application/json;
        xml              application/xml;
        text             text/plain;
    }

    server_name_in_redirect off;
    port_in_redirect        off;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    # turn on session caching to drastically improve performance
    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_session_timeout 10m;

    # allow configuring ssl session tickets
    ssl_session_tickets on;

    # slightly reduce the time-to-first-byte
    ssl_buffer_size 4k;

    # allow configuring custom ssl ciphers
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;

    # In case of errors try the next upstream server before returning an error
    proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

    upstream tools-docker-registry-5000 {
        least_conn;
        server 10.254.31.12:5000 max_fails=0 fail_timeout=0;
    }

    upstream upstream-default-backend {
        least_conn;
        server 10.254.31.11:8080 max_fails=0 fail_timeout=0;
    }
    server {
        server_name _;
        listen [::]:80 ipv6only=off default_server reuseport backlog=511;
        listen 442  default_server reuseport backlog=511 ssl http2;
        # PEM sha: 146d6a1a176900b4af00faf1b177c82d6c7b39ae
        ssl_certificate                         /ingress-controller/ssl/system-snake-oil-certificate.pem;
        ssl_certificate_key                     /ingress-controller/ssl/system-snake-oil-certificate.pem;

        more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains; preload";
        location / {
            set $proxy_upstream_name "upstream-default-backend";

            port_in_redirect off;
            client_max_body_size                    "1m";

            proxy_set_header Host                   $host;

            # Pass Real IP
            proxy_set_header X-Real-IP              $remote_addr;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";

            proxy_http_version                      1.1;
            proxy_pass http://upstream-default-backend;
        }
        # health checks in cloud providers require the use of port 80
        location /healthz {
            access_log off;
            return 200;
        }

        # this is required to avoid error if nginx is being monitored
        # with an external software (like sysdig)
        location /nginx_status {
            allow 127.0.0.1;
            allow ::1;
            deny all;

            access_log off;
            stub_status on;
        }
    }

    server {
        server_name registry.<censored>;
        listen [::]:80;

        location / {
            set $proxy_upstream_name "tools-docker-registry-5000";

            port_in_redirect off;

            auth_basic "Authentication Required - Docker-Registry";
            auth_basic_user_file /etc/ingress-controller/auth/tools-docker-registry.passwd;
            proxy_set_header Authorization "";

            client_max_body_size                    "0";

            proxy_set_header Host                   $host;

            # Pass Real IP
            proxy_set_header X-Real-IP              $remote_addr;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";

            proxy_http_version                      1.1;
            proxy_pass http://tools-docker-registry-5000;
        }
    }

    # default server, used for NGINX healthcheck and access to nginx stats
    server {
        # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
        # Changing this value requires a change in:
        # https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/nginx/command.go#L104
        listen [::]:18080 ipv6only=off default_server reuseport backlog=511;

        location /healthz {
            access_log off;
            return 200;
        }
        location /nginx_status {
            access_log off;
            stub_status on;
        }

        # this location is used to extract nginx metrics
        # using prometheus.
        # TODO: enable extraction for vts module.
        location /internal_nginx_status {
            allow 127.0.0.1;
            allow ::1;
            deny all;

            access_log off;
            stub_status on;
        }

        location / {
            set $proxy_upstream_name "upstream-default-backend";
            proxy_pass             http://upstream-default-backend;
        }

    }

    # default server for services without endpoints
    server {
        listen 8181;
        set $proxy_upstream_name "-";

        location / {
            return 503;
        }
    }
}

stream {
    # map FQDN that requires SSL passthrough
    map $ssl_preread_server_name $stream_upstream {
        # send SSL traffic to this nginx in a different port
        default                         nginx-ssl-backend;
    }

    log_format log_stream '$remote_addr [$time_local] $protocol [$ssl_preread_server_name] [$stream_upstream] $status $bytes_sent $bytes_received $session_time';

    access_log /var/log/nginx/access.log log_stream;

    error_log  /var/log/nginx/error.log;

    # configure default backend for SSL
    upstream nginx-ssl-backend {
        server 127.0.0.1:442;
    }
    server {
        listen                  [::]:443 ipv6only=off;
        proxy_pass              $stream_upstream;
        ssl_preread             on;
    }

    # TCP services

    # UDP services
}

ingress.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: docker-registry
  namespace: tools
  annotations:
    # type of authentication
    ingress.kubernetes.io/auth-type: basic
    # name of the secret that contains the user/password definitions
    ingress.kubernetes.io/auth-secret: docker-registry-basic-auth
    # message to display with an appropiate context why the authentication is required
    ingress.kubernetes.io/auth-realm: "Authentication Required - Docker-Registry"
    # request body maximum allowed size unlimited
    ingress.kubernetes.io/proxy-body-size: "0"
spec:
  rules:
    - host: registry.<censored>
      http:
        paths:
        - backend:
            serviceName: docker-registry
            servicePort: 5000

service.yml

apiVersion: v1
kind: Service
metadata:
  name: docker-registry
  namespace: tools
spec:
  selector:
    app: docker-registry
  type: NodePort
  ports:
  - port: 5000
    protocol: TCP

deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: docker-registry
  namespace: tools
  labels:
    app: docker-registry
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
        - name: docker-registry
          image: registry:2.6
          ports:
            - containerPort: 5000
        #   volumeMounts:
        #     - mountPath: /var/lib/registry
        #       name: image-data-volume
        # volumes:
        #   - name: image-data-volume
        #     hostPath:
        #       path: /data/docker-registry

aledbf · 2017-03-03T15:08:58Z

@muffin87 from the log and the curl output this is working. Did you configured your docker client to use registry.<censored> as an insecure registry?

andiMenge · 2017-03-03T15:18:06Z

@aledbf in this case: yes, registry.<censored> is configured insecure.
Normally we have a F5 Loadbalancer in front of the ingress-controller that handles TSL termination.

For debugging reasons I am bypassing the F5 so I have to use the insecure registry.
Connecting through the F5 results in the same error.

andiMenge · 2017-03-03T15:31:42Z

@aledbf this is what we can see in the ingress-controller logs while doing a docker push

::ffff:<censored> - [::ffff:<censored>] - ucpapp [03/Mar/2017:15:27:54 +0000] "PUT /v1/repositories/openjdk/ HTTP/1.1" 404 19 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" 623 0.041 [tools-docker-registry-5000] 10.254.31.12:5000 19 0.000 404

I am not sure how a docker pushworks under the hood and why dockerd is switching to /v1/.

aledbf · 2017-03-03T15:48:53Z

@andiMenge please check the docker-registry logs. I think you will find the same 404 error. If that is the case then the registry is returning the error, not the ingress controller.
Can you pull from the registry?

andiMenge · 2017-03-03T16:11:22Z

@aledbf you are right, the 404 is from the registry. Strange thing is, if I bypass the ingress, push and pulls work.

This is what I get if I try to pull (currently there is no valid image inside the registry)

Mar  3 15:59:50 moby root: time="2017-03-03T15:59:50.085761251Z" level=debug msg="[registry] Calling GET https://registry.<censored>/v1/repositories/openjdk/images"
Mar  3 15:59:50 moby root: time="2017-03-03T15:59:50.304449080Z" level=error msg="Not continuing with pull after error: Error: image openjdk:latest not found"
Mar  3 16:02:08 moby root: time="2017-03-03T16:02:08.656002530Z" level=debug msg="Calling GET /_ping"
Mar  3 16:02:08 moby root: time="2017-03-03T16:02:08.698394305Z" level=debug msg="Calling POST /v1.26/images/create?fromImage=registry.<censored>%2Fopenjdk&tag=latest"
Mar  3 16:02:08 moby root: time="2017-03-03T16:02:08.779542000Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.<censored>"
Mar  3 16:02:08 moby root: time="2017-03-03T16:02:08.819091547Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.<censored>"
Mar  3 16:02:08 moby root: time="2017-03-03T16:02:08.819317223Z" level=debug msg="Trying to pull registry.<censored>/openjdk from https://registry.<censored> v2"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.312269446Z" level=error msg="Attempting next endpoint for pull after error: manifest unknown: manifest unknown"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.312431155Z" level=debug msg="Trying to pull registry.<censored>/openjdk from https://registry.<censored> v1"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.352921913Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.<censored>"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.353219839Z" level=debug msg="attempting v1 ping for registry endpoint https://registry.<censored>/v1/"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582216052Z" level=debug msg="Error unmarshalling the _ping PingResult: invalid character '<' looking for beginning of value"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582360892Z" level=debug msg="PingResult.Version: \"\""
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582427266Z" level=debug msg="Registry standalone header: ''"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582488624Z" level=debug msg="PingResult.Standalone: true"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582552560Z" level=debug msg="Endpoint https://registry.<censored>/v1/ is eligible for private registry. Enabling decorator."
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582660693Z" level=debug msg="[registry] Calling GET https://registry.<censored>/v1/repositories/openjdk/images"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.817444263Z" level=error msg="Not continuing with pull after error: Error: image openjdk:latest not found"

I will do some tests with valid images inside the registry later...

andiMenge · 2017-03-03T16:16:51Z

This is what it looks like from inside the registry:

my earlier push of the openjdk image was recognised...

/var/lib/registry/docker/registry/v2/repositories/openjdk/_uploads # ls
25da551f-cd87-45bc-8489-edd1c5523a1f  87b2ece7-84e7-4c9e-84d0-7409b133dab2
32e6fdec-31e8-47ba-a3c5-fb029485dcf8  8b35825c-7514-4f29-9c68-103d5913bd37
336c4928-5060-49d5-8baf-26407965e8ed  8dfb236f-3bd5-4261-9980-66a6aad1dbaa
39a4d3a2-32d6-42ef-9239-2a55243854b5  a9105502-e429-406c-8a6b-d598e0150845
3da96d01-a6f3-4141-a3bc-dcb5c4b2ef39  b08bcd17-19ce-424d-aa50-7202c647c501
3f647174-3a9b-4942-b12e-a88bd47127cc  b155623f-e2ac-4812-8e7c-db22c5b447c3
4585d21d-2414-4d57-afda-999dc56b212d  b17045ae-e498-47c0-9b69-9041b636ec62
48632729-5f29-4deb-9975-3e157a38ad70  b66e2ac1-f86e-48dc-8b82-0540f9cbcb5f
491cf7c4-d9ad-40f6-87d3-aaf7c509b7a8  bb6dc397-f344-43fb-999e-a66c572a80eb
4c40f4de-0b8b-4168-8ae1-9103bffdc898  c26de260-d9d8-447b-9693-ef0864af258a
5c666d9f-d074-48d4-9f5f-47f7efbcea9e  c52b329a-c573-4b5d-a2b5-3b727da3a9a1
611794f5-0e09-411f-8fa9-8a00118d55c1  c8dc6bea-98ed-4903-ad39-7cf220b04f29
673572fd-4082-43fd-9b5d-8a5418e1adf8  cb4676b1-69f5-478f-a205-bb250d803272
6caada73-951c-4403-91e9-9f0fa5c8eb8e  fc317e83-4971-4003-8813-771b72a73c86
6cefc76f-f979-4d79-b41e-2ca335de916a  fc3df036-3c9e-4395-b8c9-dcad0c1bd339
/var/lib/registry/docker/registry/v2/repositories/openjdk/_uploads # du -sch
244.0K	.
244.0K	total

andiMenge · 2017-03-04T11:54:47Z

the issue was a misconfiguration in the registry that caused redirecting issues in the ingress

if set properly, the ingress works fine
host: https://myregistryaddress.org:5000

mtrofimm · 2017-05-25T21:44:17Z

@andiMenge I have the same issue. Can you give some more information how you solved this problem?

muffin87 · 2017-06-05T11:07:44Z

@mtrofimm:

You have to set the right URL in your registry config (config.yml).

Example: Your registry should be reachable under https://my-awesome-registry.myproject.de

Registry config:

    http:
      addr: :5000
      debug:
        addr: :5001
      host: https://my-awesome-registry.myproject.de
      headers:
        X-Content-Type-Options: [nosniff]

And you also have to set this URL in your ingress manifest.

K8s ingress file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-registry
  namespace: mynamespace
  annotations:
    # type of authentication
    ingress.kubernetes.io/auth-type: basic
    # name of the secret that contains the user/password definitions
    ingress.kubernetes.io/auth-secret: docker-registry-basic-auth
    # message to display with an appropiate context why the authentication is required
    ingress.kubernetes.io/auth-realm: "Authentication Required - Docker-Registry"
    # request body maximum allowed size unlimited
    ingress.kubernetes.io/proxy-body-size: "0"
spec:
  rules:
    - host: my-awesome-registry.myproject.de
      http:
        paths:
        - backend:
            serviceName: my-registry
            servicePort: 5000

shenshouer · 2017-11-07T08:04:53Z

I push docker image to registry behind nginx ingress and meet the error: blob unknown to registry.

aledbf · 2017-11-17T14:30:56Z

@muffin87 @shenshouer I will add an example to show how is possible to expose a docker registry running in the cluster using kube-lego without limit in the uploads.

shenshouer · 2017-11-27T01:02:11Z

I think the ingress.kubernetes.io/proxy-body-size: "0" int annotations of nginx ingress doesn`t work fine!

angapov · 2018-05-23T13:30:22Z

It's nginx.ingress.kubernetes.io/proxy-body-size: "0" now

…on for docker-registry, see kubernetes-retired/contrib#2444

andiMenge mentioned this issue Mar 3, 2017

Can't docker push through nginx-ingress nginxinc/kubernetes-ingress#122

Closed

muffin87 closed this as completed Mar 4, 2017

jpds mentioned this issue Oct 29, 2017

Docker Registry ingress going to default backend with POST kubernetes/ingress-nginx#1623

Closed

mfoco added a commit to mfoco/deepops that referenced this issue Mar 4, 2019

added proxy-body-size parameter set to 0 in nginx ingress configurati…

af9cbea

…on for docker-registry, see kubernetes-retired/contrib#2444

mfoco mentioned this issue Mar 4, 2019

added proxy-body-size parameter set to 0 NVIDIA/deepops#126

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nginx-Ingress: Docker Registry behind Ingress - docker push don't work #2444

Nginx-Ingress: Docker Registry behind Ingress - docker push don't work #2444

muffin87 commented Mar 3, 2017

aledbf commented Mar 3, 2017

muffin87 commented Mar 3, 2017

aledbf commented Mar 3, 2017

aledbf commented Mar 3, 2017

andiMenge commented Mar 3, 2017

aledbf commented Mar 3, 2017

andiMenge commented Mar 3, 2017

aledbf commented Mar 3, 2017

andiMenge commented Mar 3, 2017

aledbf commented Mar 3, 2017

andiMenge commented Mar 3, 2017

andiMenge commented Mar 3, 2017 •

edited

aledbf commented Mar 3, 2017

andiMenge commented Mar 3, 2017

andiMenge commented Mar 3, 2017

andiMenge commented Mar 4, 2017

mtrofimm commented May 25, 2017

muffin87 commented Jun 5, 2017 •

edited

shenshouer commented Nov 7, 2017

aledbf commented Nov 17, 2017

shenshouer commented Nov 27, 2017

angapov commented May 23, 2018

Nginx-Ingress: Docker Registry behind Ingress - docker push don't work #2444

Nginx-Ingress: Docker Registry behind Ingress - docker push don't work #2444

Comments

muffin87 commented Mar 3, 2017

Issue

Error Message:

Setup

Configs

aledbf commented Mar 3, 2017

muffin87 commented Mar 3, 2017

aledbf commented Mar 3, 2017

aledbf commented Mar 3, 2017

andiMenge commented Mar 3, 2017

aledbf commented Mar 3, 2017

andiMenge commented Mar 3, 2017

aledbf commented Mar 3, 2017

andiMenge commented Mar 3, 2017

aledbf commented Mar 3, 2017

andiMenge commented Mar 3, 2017

andiMenge commented Mar 3, 2017 • edited

aledbf commented Mar 3, 2017

andiMenge commented Mar 3, 2017

andiMenge commented Mar 3, 2017

andiMenge commented Mar 4, 2017

mtrofimm commented May 25, 2017

muffin87 commented Jun 5, 2017 • edited

shenshouer commented Nov 7, 2017

aledbf commented Nov 17, 2017

shenshouer commented Nov 27, 2017

angapov commented May 23, 2018

andiMenge commented Mar 3, 2017 •

edited

muffin87 commented Jun 5, 2017 •

edited