Can not access to Rancher Management Server behind Nginx as https reverse proxy

[winggundamth]

Hi. I'm using Nginx container to listen on port 80 and 443 to act as reverse proxy for every containers both http and https. After I put Rancher Management Server behind Nginx it always showing an error on only https

Error
[Exception... "<no message>" nsresult: "0x805e0006 (<unknown>)" location: "JS frame :: https://releases.rancher.com/ui/0.24.0/assets/vendor.js :: .send :: line 3" data: no]
Reload to try again or log out

But http access and direct access to Rancher Management Server is working fine. This is how I run Rancher Management Server

docker run --name=rancher-server -d --restart=always -p 8080:8080 rancher/server

This is my Nginx configuration

server {
  listen 80;
  server_name rancher.example.com ;

  location / {
    include proxy_params;
    proxy_pass http://172.17.42.1:8080;
  }

}

server {

  listen 443;

  ssl on;
  ssl_certificate_key /etc/ssl/private/rancher.example.com.key;
  ssl_certificate /etc/ssl/private/rancher.example.com.crt;

  ## Strong SSL Security
  ## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
  ssl_ciphers "ECDHE-RSA-AES128-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA128:DHE-RSA-AES128-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA128:ECDHE-RSA-AES128-SHA384:ECDHE-RSA-AES128-SHA128:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA128:DHE-RSA-AES128-SHA128:DHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA384:AES128-GCM-SHA128:AES128-SHA128:AES128-SHA128:AES128-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;
  add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
  add_header X-Frame-Options DENY;
  add_header X-Content-Type-Options nosniff;
  ssl_stapling on; # Requires nginx >= 1.3.7
  ssl_stapling_verify on; # Requires nginx >= 1.3.7
  resolver 8.8.8.8 8.8.4.4 valid=300s;
  resolver_timeout 5s;

  server_name rancher.example.com ;
  server_tokens off; ## Don't show the nginx version number, a security best practice

  ## Increase this if you want to upload large attachments
  client_max_body_size 0;

  location / {
    include proxy_params;
    proxy_pass http://172.17.42.1:8080;
  }

}

[flaccid]

Good to know I'm not alone. I ran into this too and with a simply nginx conf.

[hjianhao]

This is my config, it works fine.

server {
listen 8002;
server_name xxx.asuscomm.com;(replace with your server name)

location / {
    proxy_set_header Host $host:$server_port;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://192.168.1.140:8080; (replace with your intranet ip address)
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

}

[winggundamth]

@hjianhao Please see that problem happen with https not http. I already try with your configuration in https context but still not working.

[vincent99]

Please see #140 (comment) ; x-api-request-url until we add support for the more standard x-forwarded-proto (#1309).

[lenovouser]

@vincent99 x-api-request-url doesn't work.

I installed Rancher and got 2 (kind of) different issues. I am getting this error in the rancher-server console when I select the standard "rancher.mydomain.tld" as Host Registration:

time="2015-07-18T14:50:58Z" level=error msg="Unable to start EventRouter" Err="websocket: bad handshake" 
time="2015-07-18T14:50:58Z" level=info msg="Exiting go-machine-service..." 
time="2015-07-18T14:50:59Z" level=info msg="Setting log level" logLevel=info 
time="2015-07-18T14:50:59Z" level=info msg="Starting go-machine-service..." gitcommit=102d311 
time="2015-07-18T14:50:59Z" level=info msg="Initializing event router" workerCount=10 
time="2015-07-18T14:51:00Z" level=error msg="Failed to subscribe to events." error="websocket: bad handshake" subscribeUrl="ws://rancher.mydomain.tld/v1/subscribe?eventNames=physicalhost.create%3Bhandler%3DgoMachineService&eventNames=physicalhost.bootstrap%3Bhandler%3DgoMachineService&eventNames=physicalhost.remove%3Bhandler%3DgoMachineService&eventNames=ping%3Bhandler%3DgoMachineService"

This happens about 2 times every second. The reason here is that he is not using wss.

This error doesn't come up when I don't select the standard rancher.mydomain.tld and instead manually put https://rancher.mydomain.tld into the Host Registration to force secure connections. Then, instead this one is showing up in the rancher-agent console:

time="2015-07-18T14:59:43Z" level="info" msg="Starting event router." 
time="2015-07-18T14:59:43Z" level="info" msg="Watching state directory: /var/lib/rancher/state/containers" 
time="2015-07-18T14:59:43Z" level="info" msg="Processing event: &docker.APIEvents{Status:\"start\", ID:\"2a6baa8257e30382bb21022e0904cb1dbb4dabe512b6ba0d464a344351c3e92e\", From:\"-simulated-\", Time:0}" 
time="2015-07-18T14:59:43Z" level="info" msg="Connecting to proxy." url="wss://https://rancher.mydomain.tld/v1/connectbackend?token=eyJhbGciOiJSUzI1NiJ9.eyJleHAiOjE0MzcyMzE4ODQsInN1YiI6ImNhdHRsZSIsImlzcyI6Imh0dHA6XC9cL2NhdHRsZS5pbyIsInJlcG9ydGVkVXVpZCI6IjI1MzA0ZGNmLWMzOWEtNDcwNS1iMDU5LTUxMTIyYjJhZTJhZSIsImlhdCI6MTQzNzIzMTU4NCwia2lkIjoiZGVmYXVsdCJ9.cnr734y-o5youdnFr9xYd2L7vLjxp6T8WuA9YlEsY-oU9dQzyC0BYsCY2k9ln5LX--hSf79nxH4qPPBhkuHekQidH2TBuIwYBFW8Izm396HfMlGm3qE2tT-sYyJ2ifIsqSN6KhlrO8vtrRpqGTbI3XwtvEiXgu9DxjbiB0QKtkkOfmNtZCW2FhQ7vhIg83hckjAYrH4AkflkE0rTgDrZC_mYu7iIljdQZMLmoAZPoyc4hPtQe7Z1HVQBcIfZdxNOgLu3E-eJPeOsYliXG4UJcGo1Wjp_P_mprkdNVt2f4dMTLAXeADonruc_RAfWURBuk4wE--kHGrJ9_nTu5wwKnw" 
time="2015-07-18T14:59:43Z" level="fatal" msg="Failed to connect to proxy." error="dial tcp 46.101.xxx.xxx:0: connection refused"

The problem here is that he is trying to connect to wss://https://, which is obviously doomed to fail 😄

That is why I am kind of in a dilemma because no matter what I do, there will occur an error on one of the both sides rancher-server and rancher-agent

Related Issue
Related Forum Post

[cjellick]

@Apfeluser I'll be fixing #1580 today and it will be in this week's release.

[cjellick]

Fixed. The host and x-forwarded-{port,proto} headers are now supported. Will be in release v0.31.0, assuming it passes QA.

[cjellick]

@sangeethah Here is an exact set of steps to follow to setup Rancher with SSL.

On your desktop

scp rancher.io.bundled.crt <rancher-server host ip>:.
scp rancher.io.key <rancher-server host ip>:.

On your rancher-server host

sudo su -
docker run -d --restart=always --name=rancher-server -e "CATTLE_API_ALLOW_CLIENT_OVERRIDE=true" rancher/server:v0.31.0-rc1
mkdir /root/nginx
vim /root/nginx/rancher.io.conf
# - Copy and pasted config from my documentation PR*
# - Changed all instances of <server> to sang.rancher.io. It must match your domain name.
# - Changed <cert_file> to `/etc/nginx/ssl/rancher.io.bundled.crt`
# - Chagne <key_file> to `/etc/nginx/ssl/rancher.io.key`

mkdir /root/nginx/ssl
cp /home/<your user name>/rancher.io.* /root/nginx/ssl/

docker run -d -p 80:80 -p 443:443 -v /root/nginx/ssl:/etc/nginx/ssl/:ro -v /root/nginx/rancher.io.conf:/etc/nginx/conf.d/rancher.io.conf:ro --link='rancher-server' nginx

• My documentation PR: Updte documentation for SSL changes rancher.github.io#16

[sangeethah]

Tested with rancher/server:v0.31.0-rc1

Able to bring up a rancher-server with SSL behind Nginx as https reverse proxy.
Following are the steps followed:

docker run -d --restart=always --name=rancher-server -e "CATTLE_API_ALLOW_CLIENT_OVERRIDE=true" rancher/server:v0.31.0-rc1
mkdir /root/nginx

docker run -d -p 80:80 -p 443:443 -v /root/nginx/ssl:/etc/nginx/ssl/:ro -v /root/nginx/rancher.io.conf:/etc/nginx/conf.d/rancher.io.conf:ro --link='rancher-server' nginx

nginx confifuration used:

upstream rancher {
    server rancher-server:8080;
}

server {
    listen 443 ssl;
    server_name <servername>;
    ssl_certificate /etc/nginx/ssl/rancher.io.bundled.crt;
    ssl_certificate_key /etc/nginx/ssl/rancher.io.key;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://rancher;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

server {
    listen 80;
    server_name <servername>;
    return 301 https://$server_name$request_uri;
}

Able to add custom hosts , Digital Ocean hosts to rancher server successfully.
Able to view host stats.
Able to view container logs and also exec shell.
Basic sanity checks to make sure that the connectivity between containers from different hosts work.

[sarslans]

@sangeethah Can we use same step in Racnher v1.6.17 ? is it tested for somebody?

[TonyEight]

@sarslans Have you been able to test @sangeethah's Nginx conf ?
I'm also trying to setup Rancher behind a legacy Nginx reverse proxy.

[kevinsingapore]

@hjianhao Please see that problem happen with https not http. I already try with your configuration in https context but still not working.

i had also the same problem.

[kevinsingapore]

@sarslans Have you been able to test @sangeethah's Nginx conf ?
I'm also trying to setup Rancher behind a legacy Nginx reverse proxy.

do u have solve the problem?