Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running Harbor with HTTP behind a HTTPS Reverse Proxy (nginx) #3114

Open
Flipez opened this Issue Aug 24, 2017 · 16 comments

Comments

Projects
None yet
@Flipez
Copy link

Flipez commented Aug 24, 2017

I've tried to get harbor running behind (another) nginx reverse proxy.

So the first setup looked like this:

nginx (host,ssl) -> harbor-nginx(non-ssl) -> harbor

If I setup the first nginx with ssl redirect it basically works but the docker client send the first try in plain which I do not want.

I tried different settings but none of them works because something inside harbor seems to redirect the requests to port 80. So if I stop exposing port 80 on the host the login fails with a connection refused even if I submit domain:443 or https://domain.

Are there any guidelines for this scenario?

The simple reason is that I do not want to maintain neither the SSL settings nor the cert within the container. This is globally handled by ansible.

Please let me know if any config of the current or previous setup would be useful.

# docker version
Client:
 Version:      17.06.1-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 22:51:12 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.1-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 22:50:04 2017
 OS/Arch:      linux/amd64
 Experimental: false
@totojack

This comment has been minimized.

Copy link

totojack commented Sep 7, 2017

+1

1 similar comment
@mdgreenwald

This comment has been minimized.

Copy link

mdgreenwald commented Sep 29, 2017

+1

@jsumners

This comment has been minimized.

Copy link

jsumners commented Oct 5, 2017

This is just the issue I was looking for prior to installing for the first time. In my organization we use HAProxy for SSL termination. I currently have docker-distribution running while I research VMWare's solution. With docker-distribution this sort of configuration can be tricky (docker/distribution#2225), so I expected to find such an issue for Harbor.

This is a configuration that needs to be supported.

@spark8103

This comment has been minimized.

Copy link

spark8103 commented Jan 4, 2018

Jan 4 17:26:35 172.19.0.1 registry[22654]: time="2018-01-04T09:26:35.950809267Z" level=error msg="response completed with error" auth.user.name=admin err.code="blob unknown" err.detail=sha256:9ec141d7a6d669ded82f1d0977abf28eb71ae48f767babaa4651dcb9474dcb4f err.message="blob unknown to registry" go.version=go1.7.3 http.request.host=reg.sparkknow.com http.request.id=4761d537-1500-4009-9ddb-96ca993422cf http.request.method=HEAD http.request.remoteaddr=172.31.128.101 http.request.uri="/v2/library/alpine/blobs/sha256:9ec141d7a6d669ded82f1d0977abf28eb71ae48f767babaa4651dcb9474dcb4f" http.request.useragent="docker/17.12.0-ce go/go1.9.2 git-commit/c97c6d6 kernel/3.10.0-693.11.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.12.0-ce \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.729592ms http.response.status=404 http.response.written=157 instance.id=6a311010-3363-437c-b3e4-8c88eb62dfdf service=registry vars.digest="sha256:9ec141d7a6d669ded82f1d0977abf28eb71ae48f767babaa4651dcb9474dcb4f" vars.name="library/alpine" version=v2.6.2

I have error too.

@rolep

This comment has been minimized.

Copy link

rolep commented Jan 22, 2018

Probably it can be used as https behind https proxy.

Somehow I got it running with following:
traefik (https, letsencrypt) <-> harbor (nginx, https, self-signed certs) <-> harbor

Actually, thought it will be complete ssl-passthrough to harbor-nginx, but the result is more convenient, as users see letsencrypt cert.
Traefik auto-generates letsencrypt cert (for a bundle of names), InsecureSkipVerify = true to disable cert check for backends. Also have traefik.frontend.passTLSCert=true (because of it thought it will pass-through) and traefik.protocol=https.

Certs are self-signed by local CA. In harbor.cfg selected https, path to cert.key and bundle-cert.pem.
Also using them for token service (off-topic: how to actually use/generate tokens for docker login?).

@wangycc

This comment has been minimized.

Copy link

wangycc commented May 22, 2018

+1
traefik (https, letsencrypt) <-> harbor (nginx, https, self-signed certs) <-> harbor

traefik(443) <-> harbor(nginx,443,self-signed certs),insecureSkipVerify = true

harbor self-signed:
https://github.com/vmware/harbor/blob/master/docs/configure_https.md

@ensean

This comment has been minimized.

Copy link

ensean commented Jun 3, 2018

An extra section for using external nginx in the harbor.cfg file would be great.

After a few days's trying, I got the following combination work finally...

nginx1 (https, letsencrypt) <-> harbor proxy (nginx, http) <-> harbor ui/registry etc.

The most import is to follow this troubleshooting remove the following line from common/templates/nginx/nginx.http.conf

proxy_set_header X-Forwarded-Proto $scheme;
@lxkaka

This comment has been minimized.

Copy link

lxkaka commented Aug 22, 2018

@ensean could you show your nginx1 configuration? thanks in advance

@ensean

This comment has been minimized.

Copy link

ensean commented Aug 22, 2018

@lxkaka here goes my nginx1 conf


upstream harbor {
  server harbor_proxy_ip:8080;
}

server {
    listen 443 ssl;
    server_name harbor.mycomp.com;

    ssl_certificate      /etc/nginx/conf.d/mycomp.com.crt;
    ssl_certificate_key  /etc/nginx/conf.d/mycomp.com.key;

    client_max_body_size 0;
    chunked_transfer_encoding on;

    location / {
        proxy_pass http://harbor/;
        proxy_set_header  Host              $http_host;
        proxy_set_header  X-Real-IP         $remote_addr;
        proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;

        proxy_buffering off;
        proxy_request_buffering off;
    }
}
@k7

This comment has been minimized.

Copy link

k7 commented Sep 27, 2018

After a struggle, I have fixed it.

I have the service like this:

nginx (host,ssl) -> harbor-nginx(non-ssl) -> harbor

First, I emulate login via curl, it was good.

curl -u admin:passwd123 "https://my.registry.company.com/service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry"

{
  "token": "xxxxxx",
  "expires_in": 1800,
  "issued_at": "2018-09-27T03:42:27Z"
}

It means there was other things error.

Via tcpdump, I found docker login will make a http request https://my.registry.company.com/v2/ first.

curl -X GET -I "https://my.registry.company.com/v2/"

HTTP/1.1 401 Unauthorized
Server: nginx/1.12.2
Date: Thu, 27 Sep 2018 07:20:58 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 87
Connection: keep-alive
Docker-Distribution-Api-Version: registry/2.0
Set-Cookie: beegosessionID=75df8e3e78eacf5848a83877908ca96; Path=/; HttpOnly
Www-Authenticate: Bearer realm="http://my.registry.company.com/v2//service/token",service="harbor-registry"

Now, I found the problem. realm="http://my.registry.company.com/v2//service/token". it was http, not https.

So I edit common/config/registry/config.yml,and fixed it.

auth:
  token:
    issuer: harbor-token-issuer
    realm: https://my.registry.company.com/service/token  # http -> https
    rootcertbundle: /some/root.crt
    service: harbor-registry

Great, It works.

docker login private-registry.tupu.ai -u admin
Login Succeeded
@rruu

This comment has been minimized.

Copy link

rruu commented Oct 5, 2018

and you can changes prepare script with and reinstall

  • public_url = protocol + "://" + hostname
  • public_url = "https" + "://" + hostname
@shinebayar-g

This comment has been minimized.

Copy link

shinebayar-g commented Oct 16, 2018

Hey @k7 Thank you very much for the explanation. Without changing that particular config, I was getting

Error response from daemon: Get https://goharbor.mydomain.com/v2/: unauthorized: authentication required

error when trying to login. After changing that it works! However if I run install.sh script it overrides that changed settings. So I just restarted all docker process instead of re-install.

Oh just noticed @rruu mentioned that we can change template's public_url option to change it.

@jakirpatel

This comment has been minimized.

Copy link

jakirpatel commented Oct 25, 2018

I faced a similar issue while running Harbor with HTTP behind an HTTPS Reverse Proxy.
I did the following changes:
In harbor.cfg:

  1. Make sure the hostname points to the VIP. (In my case its reverse proxy https load balancer nginx)
  2. customize_crt = off( As external source that means https nginx will do this)

  1. In config of nginx common
    /common/templates/nginx/nginx.http.conf
    Comment this line: proxy_set_header X-Forwarded-Proto $$scheme;
    Make sure to comment it / , /v2/ and /service/ endpoints.
  2. Prepare your harbor with ./prepare.
    After preparation in common/config/registry/config.yml make realm https.
auth:
  token:
    issuer: harbor-token-issuer
    realm: http://vip-address/service/token
    rootcertbundle: /etc/registry/root.crt
    service: harbor-registry

Change to:

auth:
  token:
    issuer: harbor-token-issuer
    realm: https://vip-address/service/token
    rootcertbundle: /etc/registry/root.crt
    service: harbor-registry
  1. Start the harbor.

This solved my problem.

@CrystalMethod

This comment has been minimized.

Copy link

CrystalMethod commented Dec 31, 2018

I can confirm that Harbor works behind an HTTPS reverse proxy (Synology NAS/Nginx) with SSL termination. In addition there is Traefik as another proxy in between - however, this only has the task of routing to different Docker based services. All communication uses port 80 of Harbor Proxy.

@nicholasamorim

This comment has been minimized.

Copy link

nicholasamorim commented Mar 19, 2019

I can confirm it runs with the modifications that are shown in here but docker login always returns me 400 Bad Request. Everything else works fine and emulating login via curl works fine too.

The funny thing is that if I enter the incorrect login/password, the behaviour works as expected, telling me I'm unauthorized.

@mxplusb

This comment has been minimized.

Copy link

mxplusb commented Mar 24, 2019

I had the same problem here as most folks, I wanted to run Harbor behind a reverse proxy (I'm using Nginx), but I also wanted Clair, Chart Museum, and Notary, which means HTTPS all the way. Here is what I did, thanks to @jakirpatel for the initial work.

  1. If Harbor with Notary, Clair, and Chart Museum is running, run docker-compose -f ./docker-compose.yml -f ./docker-compose.notary.yml -f ./docker-compose.clair.yml -f ./docker-compose.chartmuseum.yml down -v
  2. Generate your self-signed HTTPS certificates from the Harbor instructions.
  3. In harbor.cfg, set your hostname to the external VIP, so vip.domain.com.
  4. In harbor.cfg, set customize_cert = on
  5. In the nginx HTTPS config, /common/templates/nginx/nginx.https.conf, comment out all instances of proxy_set_header X-Forwarded-Proto $$scheme;
  6. Run ./prepare --with-notary --with-clair --with-chartmuseum
  7. Start everything back up with docker-compose -f ./docker-compose.yml -f ./docker-compose.notary.yml -f ./docker-compose.clair.yml -f ./docker-compose.chartmuseum.yml up -d
  8. On your reverse proxy server (I'm using Nginx as an example since that's what I use), create the harbor.conf file.
  9. Populate it with this:
# this block is to redirect 80 to 443, i.e. HTTP to HTTPS
server {
        listen 80;
        server_name vip.domain.com;
        return 301 https://$server_name$request_uri;
}

server {
        listen 443;
        server_name     vip.domain.com;
        add_header      Strict-Transport-Security "max-age=31536000" always;
        # your certs, I'm using let's encrypt domain wildcards.
        ssl_certificate /etc/letsencrypt/live/domain.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/domain.com/privkey.pem;
        ssl_stapling on;
        ssl_stapling_verify on;

        location / {
                client_max_body_size 2000m;
                proxy_buffering off;
                proxy_ssl_verify off;
                proxy_set_header Host $http_host;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "Upgrade";
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_pass https://harbor.internal.domain.com;
    }
}

Edit: I'm still working through the Notary configs. I've never used Notary before so I only partially understand it. If I get it working, I'll update here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.