Gitlab registry - error parsing HTTP 404 response body: invalid character #1025

weaktyper · 2016-12-17T13:52:54Z

I'm using letsencrypt, nginx-proxy and gitlab with gitlab-registry.
I can't open the gitlab-registry page of a project without getting a 500 error. Gitlab is
working fine and the url https://$REGISTR_DOMAIN/v2/ seems to work when sent a curl request except that it is not authorised. ie https://$REGISTR_DOMAIN/v2/ doesn't have a tcp connection refused which is why I can't understand why gitlab is having that problem.

This is the error in /home/git/gitlab/log/production.log of the gitlab container.

Faraday::ConnectionFailed (Failed to open TCP connection to $REGISTR_DOMAIN:443 (Connection refused - connect(2) for "$REGISTR_DOMAIN" port 443)):
  lib/container_registry/client.rb:19:in `repository_tags'
  lib/container_registry/repository.rb:22:in `manifest'
  lib/container_registry/repository.rb:31:in `tags'
  app/controllers/projects/container_registry_controller.rb:8:in `index'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'

This is the relevant parts of the docker-compose.yml

version: '2'
services:
  redis:
    ...

  postgresql:
    ...

  gitlab:
    image: sameersbn/gitlab:$GITLAB_VERSION
    container_name: gitlab
    network_mode: "bridge"
    links:
      - redis
      - postgresql
    depends_on:
      - redis
      - postgresql
    expose:
      - 80
    volumes:
      - ~/data/gitlab:/home/git/data
      - ~/data/gitlab/logs:/var/log/gitlab
      - ~/certificates:/certs/
    environment:
      - VIRTUAL_HOST=$GIT_DOMAIN
      - VIRTUAL_PORT=80
      - HTTPS_METHOD=nohttp
      - LETSENCRYPT_HOST=$GIT_DOMAIN
      - LETSENCRYPT_EMAIL=$LETSENCRYPT_EMAIL
      - SSL_SELF_SIGNED=false

      - GITLAB_REGISTRY_ENABLED=true
      - GITLAB_PROJECTS_CONTAINER_REGISTRY=true
      - GITLAB_REGISTRY_ISSUER=gitlab-issuer
      - GITLAB_REGISTRY_HOST=$REGISTRY_DOMAIN
      - GITLAB_REGISTRY_API_URL=https://$REGISTRY_DOMAIN
      - GITLAB_REGISTRY_KEY_PATH=/certs/$REGISTRY_DOMAIN.key
      - GITLAB_REGISTRY_PORT=443
      - SSL_REGISTRY_KEY_PATH=/certs/$REGISTRY_DOMAIN.key
      - SSL_REGISTRY_CERT_PATH=/certs/$REGISTRY_DOMAIN.crt

  registry:
    restart: always
    image: registry:$REGISTRY_VERSION
    container_name: gitlab-registry
    network_mode: "bridge"
    volumes:
      - ~/data/gitlab/shared/registry:/registry
      - ~/certificates:/certs
    environment:
      - VIRTUAL_HOST=$REGISTRY_DOMAIN
      - LETSENCRYPT_HOST=$REGISTRY_DOMAIN
      - LETSENCRYPT_EMAIL=$LETSENCRYPT_EMAIL

      - REGISTRY_LOG_LEVEL=info
      - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
      - REGISTRY_AUTH_TOKEN_REALM=https://$REGISTRY_DOMAIN/jwt/auth
      - REGISTRY_AUTH_TOKEN_SERVICE=container_registry
      - REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
      - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/$REGISTRY_DOMAIN.crt
      - REGISTRY_STORAGE_DELETE_ENABLED=true

networks:
  default:
    external:
      name: bridge

Anyone have any idea or have a working configuration?

Perhaps it's some problem with the certificates but they do exist.

The text was updated successfully, but these errors were encountered:

solidnerd · 2016-12-17T15:31:42Z

Hey,

for me it seems that you use different approaches to get your network working. This could lead to your problem. Why do you use network_mode: bridge and not networks ? If you are working with networks you don't need any links: anymore.

oozone · 2016-12-17T18:35:03Z

Exact same problem here

$ docker pull alpine:latest
latest: Pulling from library/alpine
Digest: sha256:1354db23ff5478120c980eca1611a51c9f2b88b61f24283ee8200bf9a54f2e5c
Status: Image is up to date for alpine:latest
$ docker tag alpine:latest [ME]:5501/group/mailserver:latest
$ docker push [ME]:5501/group/mailserver
The push refers to a repository [[ME]:5501/group/mailserver]
Get https://[ME]:5501/v1/_ping: read tcp 172.17.0.12:55632->xxxxxxxx:5501: read: connection reset by peer

ERROR: Build failed: exit code 1

weaktyper · 2016-12-17T21:40:48Z

Hi @solidnerd, I am using the network_mode: bridge to overcome a bug in nginx dockergen, issue here.
Why the links are needed, I am not quite sure yet... I will try removing them to see if it continues to work.

This is the error I get when I change the network to an external network such as nginx-proxy and use network: nginx-proxy in each container instead:
no host in upstream ":80" in /etc/nginx/conf.d/default.conf:35 -- this matches up with the issue documented in nginx-proxy repo. I can't find a solution to that.

Update:
The reason I added the link is that without it the database isn't setup successfully:
I keep getting this error and then the gitlab runtime startup crashes.

+ /usr/lib/postgresql/9.6/bin/pg_isready -h host -p 5432 -U user -d gitlabhq_production -t 1
++ expr 1 - 1
+ timeout=0

If I add a link in gitlab to postgresql,

    links:
      - redis # This should not be necessary but for some reason db setup fails without it.
      - postgresql # This should not be necessary but for some reason db setup fails without it.

then this goes away. I agree with you though that with networks, it should not be like this but I can't explain it. The passwords and settings are correct, same as for the postgresql container setup.

To be honest, I'm more concerned about why my TCP connection is refused though.

Do you happen to use letsencrypt-companion & nginx-proxy with gitlab-registry working?

UPDATE 2:
After removing all volumes, data, and reboot the VPS, it seems I have got the container registry page working now. Other than that, not sure if I did anything else.

Started GET "/docker/docker-compose-gitlab/container_registry" for ip_address at 2016-12-17 23:02:40 +0000
Processing by Projects::ContainerRegistryController#index as HTML
  Parameters: {"namespace_id"=>"docker", "project_id"=>"docker-compose-gitlab"}
Completed 200 OK in 188ms (Views: 43.9ms | ActiveRecord: 14.7ms)

The only big thing is I still cannot login with docker.
docker login $REGISTRY_DOMAIN returns

Error response from daemon: Get https://$REGISTRY_DOMAIN/v2/: error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"

nginx log:

$REGISTRY_DOMAIN my_ip_address - username [17/Dec/2016:23:07:49 +0000] "GET /jwt/auth?account=username&client_id=docker&offline_token=true&service=container_registry HTTP/1.1" 404 19 "-" "docker/1.12.4 go/go1.6.4 git-commit/1564f02 kernel/4.8.0-30-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.4 \x5C(linux\x5C))"

registry log:

172.17.0.3 - - [17/Dec/2016:23:05:53 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/4.4.0-47-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \\(linux\\))"
172.17.0.3 - - [17/Dec/2016:23:05:53 +0000] "GET /jwt/auth?account=username&client_id=docker&offline_token=true&service=container_registry HTTP/1.1" 404 19 "" "docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/4.4.0-47-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \\(linux\\))"
time="2016-12-17T23:06:11.104502604Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.6.2 http.request.host=$REGISTRY_DOMAIN http.request.id=fc55661e-99c3-44f7-bffe-87a5ae2c1e94 http.request.method=GET http.request.remoteaddr=SERVER_IP_ADDRESS http.request.uri="/v2/" http.request.useragent="docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/4.4.0-47-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \\(linux\\))" instance.id=566e754d-25a4-49b9-8317-508ec5b4ec26 service=registry version=v2.4.1

maybe something wrong with the nginx configuration? (similar to #876 and somewhat like this issue) I'll try change some of the settings and see what happens. Similar issues: docker forum 404, gitlab forum, similar issue gitlab forum, similar issue on gitlab forum, gitlab issue 983

Wonder if someone has a working configuration with docker-letsencrypt-companion and nginx-proxy?

weaktyper · 2016-12-18T13:16:26Z

This is my expanded nginx configuration for $REGISTRY_DOMAIN IN the nginx-proxy server:

upstream $REGISTRY_DOMAIN {
			# gitlab-registry
			server 172.17.0.5:5000;
}
server {
	server_name $REGISTRY_DOMAIN;
	listen 80 ;
	access_log /var/log/nginx/access.log vhost;
	return 301 https://$host$request_uri;
}
server {
	server_name $REGISTRY_DOMAIN;
	listen 443 ssl http2 ;
	access_log /var/log/nginx/access.log vhost;
	ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
	ssl_ciphers <<SSL ciphers here>>;
	ssl_prefer_server_ciphers on;
	ssl_session_timeout 5m;
	ssl_session_cache shared:SSL:50m;
	ssl_certificate /etc/nginx/certs/$REGISTRY_DOMAIN.crt;
	ssl_certificate_key /etc/nginx/certs/$REGISTRY_DOMAIN.key;
	ssl_dhparam /etc/nginx/certs/$REGISTRY_DOMAIN.dhparam.pem;
	add_header Strict-Transport-Security "max-age=31536000";
	## Start of configuration add by letsencrypt container

	location /.well-known/acme-challenge/ {
	    allow all;
	    root /usr/share/nginx/html;
	    try_files $uri =404;
	    break;
	}
	## End of configuration add by letsencrypt container
	client_max_body_size 0;
	chunked_transfer_encoding on;
        add_header Docker-Distribution-API-Version 'registry/2.0' always;

	location / {
		proxy_pass http://$REGISTRY_DOMAIN;
                proxy_set_header Authorization "";
                proxy_set_header  Host              $http_host;
                proxy_set_header  X-Real-IP         $remote_addr;
                proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
                proxy_set_header  X-Forwarded-Proto $scheme;
                proxy_read_timeout                  900;
	}
}

This is what I get when I send a curl request curl -vu **:** $REGISTRY_DOMAIN/v2

*   Trying $SERVER_IP...
* Connected to $REGISTRY_DOMAIN ($SERVER_IP) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 697 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: $REGISTRY_DOMAIN (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=$REGISTRY_DOMAIN
*        start date: Sat, 17 Dec 2016 12:39:00 GMT
*        expire date: Fri, 17 Mar 2017 12:39:00 GMT
*        issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
*        compression: NULL
* ALPN, server did not agree to a protocol
* Server auth using Basic with user 'username'
> GET /v2/ HTTP/1.1
> Host: $REGISTRY_DOMAIN
> Authorization: Basic ************************
> User-Agent: curl/7.50.1
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Server: nginx/1.10.2
< Date: Sun, 18 Dec 2016 13:33:33 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 87
< Connection: keep-alive
< Docker-Distribution-Api-Version: registry/2.0
< Www-Authenticate: Bearer realm="https://$REGISTRY_DOMAIN/jwt/auth",service="container_registry"
< X-Content-Type-Options: nosniff
< 
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
* Connection #0 to host $REGISTRY_DOMAIN left intact

weaktyper · 2017-01-20T15:33:43Z

I gave up on Gitlab-registry in the end and just using gitlab for now. Will come back and see if this is magically fixed in some future release.

mat1010 · 2018-01-16T22:30:07Z

Almost one year later:

I encountered the exact same issue and found the reason for it.

  - REGISTRY_AUTH_TOKEN_REALM=https://$REGISTRY_DOMAIN/jwt/auth

This has to point to the GITLAB domain itself - not the domain of the registry ... After changing it and restarting the registry daemon it works fine.

Anyway I still encounter the issue stated here, since the domain itself is not properly written after a reconfigure ...: https://gitlab.com/gitlab-org/gitlab-ce/issues/26265

curunoir · 2019-03-03T13:37:13Z

Hello, I'm facing the same issue using traefik in front of gitlab and registry.

version: '2'

services:

  gitlab:
    restart: always
    image: sameersbn/gitlab:11.7.5
    depends_on:
    - redis
    - postgresql
    ports:
    - "10022:22"
    volumes:
    - /srv/docker/gitlab/gitlab:/home/git/data:Z
    - ./certs:/certs
    networks:
      - gitlab
      - traefik
    labels:
      - "traefik.frontend.rule=Host:gitlab.devtools.kpax-manage.com"
      - "traefik.port=80"
      - "traefik.backend=gitlab"
      - "traefik.frontend.entryPoints=http,https"
      - "traefik.docker.network=traefik"
    environment:
    - DEBUG=false

    - TZ=Europe/Brussels
    - GITLAB_TIMEZONE=Brussels

    - GITLAB_HTTPS=false
    - SSL_SELF_SIGNED=false

    - GITLAB_HOST=gitlab.mydomain.com
    - GITLAB_PORT=80
    - GITLAB_SSH_PORT=10022


    # Registry
    - GITLAB_REGISTRY_ENABLED=true
    - GITLAB_REGISTRY_HOST=registry.mydomain.com
    - GITLAB_REGISTRY_PORT=443
    - GITLAB_REGISTRY_PATH=/registry
    - GITLAB_REGISTRY_API_URL=http://registry:5000
    - GITLAB_REGISTRY_KEY_PATH=/certs/registry.key
    - GITLAB_REGISTRY_ISSUER=gitlab-issuer

  registry:
    image: registry:2.6.2
    restart: always
    expose:
     - "5000"
    ports:
     - "5000:5000"
    volumes:
      - ./gitlab/shared/registry:/registry
      - ./certs:/certs
    environment:
      - REGISTRY_LOG_LEVEL=info
      - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
      - REGISTRY_AUTH_TOKEN_REALM=https://registry.mydomain.com/jwt/auth
      - REGISTRY_AUTH_TOKEN_SERVICE=container_registry
      - REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
      - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry.crt
      - REGISTRY_STORAGE_DELETE_ENABLED=true
      - GITLAB_REGISTRY_ISSUER=gitlab-issuer
    networks:
      - gitlab
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:registry.mydomain.com"
      - "traefik.port=5000"
      - "traefik.backend=registry"
      - "traefik.frontend.entryPoints=http,https"
      - "traefik.docker.network=traefik"

networks:
  gitlab:
    driver: bridge
  traefik:
    external:
      name: traefik

weaktyper changed the title ~~Gitlab registry - TCP connection refused~~ Gitlab registry - error parsing HTTP 404 response body: invalid character Dec 18, 2016

weaktyper closed this as completed Jan 20, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gitlab registry - error parsing HTTP 404 response body: invalid character #1025

Gitlab registry - error parsing HTTP 404 response body: invalid character #1025

weaktyper commented Dec 17, 2016 •

edited

Loading

solidnerd commented Dec 17, 2016

oozone commented Dec 17, 2016 •

edited

Loading

weaktyper commented Dec 17, 2016 •

edited

Loading

weaktyper commented Dec 18, 2016 •

edited

Loading

weaktyper commented Jan 20, 2017

mat1010 commented Jan 16, 2018 •

edited

Loading

curunoir commented Mar 3, 2019 •

edited

Loading

Gitlab registry - error parsing HTTP 404 response body: invalid character #1025

Gitlab registry - error parsing HTTP 404 response body: invalid character #1025

Comments

weaktyper commented Dec 17, 2016 • edited Loading

solidnerd commented Dec 17, 2016

oozone commented Dec 17, 2016 • edited Loading

weaktyper commented Dec 17, 2016 • edited Loading

weaktyper commented Dec 18, 2016 • edited Loading

weaktyper commented Jan 20, 2017

mat1010 commented Jan 16, 2018 • edited Loading

curunoir commented Mar 3, 2019 • edited Loading

weaktyper commented Dec 17, 2016 •

edited

Loading

oozone commented Dec 17, 2016 •

edited

Loading

weaktyper commented Dec 17, 2016 •

edited

Loading

weaktyper commented Dec 18, 2016 •

edited

Loading

mat1010 commented Jan 16, 2018 •

edited

Loading

curunoir commented Mar 3, 2019 •

edited

Loading