New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Websocket connection can't be made #412

Closed
ROODAY opened this Issue Apr 6, 2016 · 28 comments

Comments

Projects
None yet
2 participants
@ROODAY
Copy link

ROODAY commented Apr 6, 2016

I'm trying to run an instance of Jupyter on my Digital Ocean server using this container: https://hub.docker.com/r/jupyter/notebook/. I have nginx-proxy configured to connect to that container with the correct hostname, but despite editing the notes.rooday.com_location file in /etc/nginx/vhost.d/, websocket connections can't be made.

This is my notes.rooday.com_location file:

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400;

This is the error Chrome gives me when I open a jupyter iPython notebook and attempt to make a websocket connection:

WebSocket connection to 'ws://notes.rooday.com/api/kernels/677b0d97-20cd-4b1a-b499-c9b513b859d9/channels?session_id=6E236819286E46D38B03D34047252F48' failed: Error during WebSocket handshake: Unexpected response code: 400
@wader

This comment has been minimized.

Copy link

wader commented Apr 6, 2016

I think nginx-proxy already have most of that configuration https://github.com/jwilder/nginx-proxy/blob/master/nginx.tmpl#L46

Can you see if the request get to the container? interesting to know who returns 400 (bad request)

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 6, 2016

The logs for both nginx-proxy and jupyter seem to show 400 errors.

nginx-proxy

nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:16 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:17 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:19 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"

jupyter

[I 21:20:08.864 NotebookApp] 302 GET / (172.17.0.2) 3.53ms
[I 21:20:08.893 NotebookApp] 302 GET /tree (172.17.0.2) 4.38ms
[W 21:20:15.372 NotebookApp] 401 POST /login?next=%2Ftree (172.17.0.2) 4.28ms referer=http://notes.rooday.com/login?next=%2Ftree
[I 21:20:17.962 NotebookApp] 302 POST /login?next=%2Ftree (172.17.0.2) 4.61ms
[I 21:24:22.371 NotebookApp] 302 GET / (172.17.0.2) 2.79ms
[I 21:37:44.561 NotebookApp] 302 GET /notebooks/Untitled.ipynb (172.17.0.2) 1.37ms
[I 21:37:46.107 NotebookApp] 302 GET /tree (172.17.0.2) 1.54ms
[I 21:42:06.768 NotebookApp] 302 GET / (172.17.0.2) 4.08ms
[W 21:42:15.099 NotebookApp] 404 GET /nbextensions/widgets/extension.js?v=20160406210551 (172.17.0.2) 2.85ms referer=http://notes.rooday.com/notebooks/Untitled.ipynb
[I 21:42:15.199 NotebookApp] Kernel started: 007c6099-febe-40b3-847c-5d90bcc4cf4b
[W 21:42:15.271 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20160406210551 (172.17.0.2) 9.50ms referer=http://notes.rooday.com/notebooks/Untitled.ipynb
[W 21:42:16.172 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 878.60ms referer=None
[W 21:42:17.221 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 7.58ms referer=None
[W 21:42:19.288 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 20.13ms referer=None
[W 21:42:23.340 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 7.98ms referer=None
[W 21:42:31.394 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 6.49ms referer=None
[W 21:42:47.459 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 9.92ms referer=None
@wader

This comment has been minimized.

Copy link

wader commented Apr 6, 2016

Seems like the 400 comes jupyter. Try to run the jupyter container and expose the web port so that you can browse to it directly and not thru nginx-proxy, might help debugging. I would guess it's something with jupyter or that your extra config makes nginx-proxy forward requests that it does not like.

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 7, 2016

@wader I made an issue on the Jupyter/notebook repo and their response lead to this example nginx configuration file:

## Based on: https://github.com/calpolydatascience/jupyterhub-deploy-data301/blob/master/roles/nginx/templates/nginx.conf.j2
user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {

  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

  upstream jupyter {
      server jupyter:8888 fail_timeout=0;
  }


    server {
        listen 80;
        server_name xsede.carlboettiger.info;
        rewrite        ^ https://$host$request_uri? permanent;
    }

    server {
        listen 443;

        client_max_body_size 50M;
        server_name xsede.carlboettiger.info;

        ssl on;
        ssl_certificate /data/cert.crt;
        ssl_certificate_key /data/key.key;

        ssl_ciphers "AES128+EECDH:AES128+EDH";
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        add_header X-Content-Type-Options nosniff;
        ssl_stapling on; # Requires nginx >= 1.3.7
        ssl_stapling_verify on; # Requires nginx => 1.3.7
        resolver_timeout 5s;


        # Expose logs to "docker logs".
        # See https://github.com/nginxinc/docker-nginx/blob/master/Dockerfile#L12-L14
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        location / {
            proxy_pass http://jupyter;

            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
     location ~* /(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
            proxy_pass http://jupyter;

            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # WebSocket support
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;

        }
    }
}

I added everything that it had except for the SSL related bit, resulting in these two files:

/etc/nginx/vhost.d/notes.rooday.com

client_max_body_size 50M;
error_log /var/log/nginx/error.log;

location ~* /(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
        proxy_pass http://notes.rooday.com;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
}

and /etc/nginx/vhost.d/notes.rooday.com_location

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400;

Despite all this, the error persists. One of the Project Jupyter members commented that it might be a regex or priority bug preventing the Upgrade block from being applied correctly.

@wader

This comment has been minimized.

Copy link

wader commented Apr 7, 2016

Ok hmm in /etc/nginx/vhost.d/notes.rooday.com do you use notes.rooday.com as hostname? not really sure how that will work... seems bad if that will proxy the request thru nginx-proxy again.
The error is still that 400 is returned? can you paste the nginx-proxy log?

You could try a quick and dirty way to see the response/reply in the container using tcpdump, but be sure to disable gzip for the request. I.e. use the "Copy as curl" in chrome developer tools and remove gzip headers.
For the container do something like docker exec -ti <container-id> bash, install tcpdump and run tcpdump -A. You could also dump to a pcap and inspect in wireshark.

@wader

This comment has been minimized.

Copy link

wader commented Apr 7, 2016

But i guess the best would be to run jupyter in some kind of debug mode. Is that possible?

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 8, 2016

Yes, I was able to run the jupyter/notebook container in debug mode by creating a new image based off of it that had the --debug flag in the commands. After running it in debug, connecting to it, and opening an IPython 3 notebook and getting the connection error, I checked the container's logs and saw this:

[W 22:31:20.357 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 1269.18ms referer=None
[D 22:31:20.358 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:21.388 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:21.403 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 15.53ms referer=None
[D 22:31:21.403 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:23.437 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:23.450 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 12.59ms referer=None
[D 22:31:23.450 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:25.284 NotebookApp] 200 GET /custom/custom.css (172.17.0.2) 6.46ms
[D 22:31:25.433 NotebookApp] 200 GET /static/notebook/js/main.min.js.map (172.17.0.2) 71.94ms
[D 22:31:25.451 NotebookApp] 304 GET /static/style/style.min.css.map (172.17.0.2) 8.74ms
[D 22:31:27.480 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:27.490 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 10.67ms referer=None
[D 22:31:27.491 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:35.538 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:35.549 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 12.31ms referer=None
[D 22:31:35.550 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:51.590 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:51.616 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 30.62ms referer=None
[D 22:31:51.617 NotebookApp] Can "Upgrade" only to "WebSocket".

Here are the full logs: http://pastebin.com/vTqc1Xug

The difference I noticed running it in debug is that now after every 400 error there is this line:

Can "Upgrade" only to "WebSocket".

Would this be a configuration issue with nginx or Jupyter Notebook? One of the Project Jupyter members did mention that it could be a regex issue in the nginx config file that wasn't applying the upgrade code properly.

@wader

This comment has been minimized.

Copy link

wader commented Apr 9, 2016

Yeah looking at the code https://github.com/jupyter/notebook/blob/41d6da235cbf3bcf6d7f818e11a066e0fd12ff8b/notebook/allow76.py#L38 it looks like the upgrade header is the problem.

Maybe this can give some clue what happens to the upgrade header, run a netcat on port 80 with VIRTUAL_HOST=notes.rooday.com:

docker run -ti --rm --expose 80 -e VIRTUAL_HOST=notes.rooday.com debian:jessie sh -c "apt-get update && apt-get -y install netcat && while true ; do nc -v -l -p 80 ; done"

Then do websocket like request:

curl http://notes.rooday.com/api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED -H "Connection: Upgrade" -H "Upgrade: websocket"

How do the headers look? for me i get:

GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1
Host: test
Upgrade: websocket
Connection: upgrade
X-Real-IP: 213.114.234.13
X-Forwarded-For: 213.114.234.13
X-Forwarded-Proto: http
User-Agent: curl/7.43.0
Accept: */*

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 9, 2016

Here's what I got from the curl:

172.17.0.2: inverse host lookup failed: Unknown host
connect to [172.17.0.6] from (UNKNOWN) [172.17.0.2] 44656
GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1
X-Real-IP: 172.17.0.1
Host: notes.rooday.com
X-Forwarded-For: 104.131.93.98, 172.17.0.1
Connection: upgrade
Accept-Encoding: gzip
CF-IPCountry: US
CF-RAY: 29108b61a5a40efd-EWR
X-Forwarded-Proto: http
CF-Visitor: {"scheme":"http"}
User-Agent: curl/7.35.0
Accept: */*
CF-Connecting-IP: 104.131.93.98
CF-Respect-Strong-Etag: 0

Does the file you linked from the notebook project say that the Connection has to be "Upgrade" instead of "upgrade"?

@wader

This comment has been minimized.

Copy link

wader commented Apr 10, 2016

Nearly, it says that the "Connection" header needs to contain "upgrade" and the "Upgrade" header needs to be "websocket", both compared incasesensitive. So looking at the headers received by the container from the curl requests i looks like the problem is the missing Upgrade: websocket headers. Any idea how could be stripping it?
One thing that looks a bit suspicious is the X-Forwarded-For: 104.131.93.98, 172.17.0.1. Seems like the request goes thru two proxies hmm. I would guess 104.131.93.98 is the proxy listening externally and 172.17.0.1 is a proxy listening in a container somewhere (both could be the same i guess). Could this be because of the proxy_pass http://notes.rooday.com? could you try removing it? im not really follow what the custom config is suppose to do, most of it looks like thing nginx-proxy already does.
Hope that helps!

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 10, 2016

I commented out the proxy pass for the api paths so that the config file looks like this:

client_max_body_size 50M;
error_log /var/log/nginx/error.log;

location ~* /(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
        #proxy_pass http://notes.rooday.com;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
}

Restarting nginx and connecting to the notebook gave the connection error from before, but checking the nginx logs I see this:

nginx.1    | 2016/04/10 14:47:10 [error] 21#21: *17 open() "/etc/nginx/html/api/kernels/673aad9e-9042-4867-bf10-59dba3f408d4/channels" failed (2: No such file or directory), client: 172.17.0.1, server: notes.rooday.com, request: "GET /api/kernels/673aad9e-9042-4867-bf10-59dba3f408d4/channels?session_id=0E354A9F626E44698F2BD484DA09F317 HTTP/1.1", host: "notes.rooday.com"

I'm not sure, but I'm guessing that now without the proxy pass its trying to find an api folder from the server instead of from the container?

I also tried leaving the proxy pass in there and changing the Upgrade header, so that it looked like this:

client_max_body_size 50M;
error_log /var/log/nginx/error.log;

location ~* /(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
        proxy_pass http://notes.rooday.com;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        #proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Upgrade "websocket";
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
}

With that config, I actually was able to connect to the kernel when in the notebook. However, when I tried running some code from the notebook, I got this error from the jupyter logs:

[D 14:52:53.058 NotebookApp] 200 GET /custom/custom.css (172.17.0.2) 2.12ms
[D 14:52:53.122 NotebookApp] 304 GET /static/style/style.min.css.map (172.17.0.2) 2.50ms

The nginx logs also show a 304 (I couldnt find one for the custom.css but I'm sure it was there somewhere)

nginx.1    | notes.rooday.com 172.17.0.1 - - [10/Apr/2016:14:52:53 +0000] "GET /static/style/style.min.css.map HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"

I'm new to nginx so I'm just guessing here but since I changed the upgrade header to "websocket" normal http requests for assets are failing?

Also, I'm not sure if this is responsible, but in my chrome console I see these errors and warnings:

actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later...
Failed to load resource: the server responded with a status of 404 (Not Found) http://notes.rooday.com/nbextensions/widgets/extension.js?v=20160408223056
Failed to load resource: the server responded with a status of 404 (Not Found) http://notes.rooday.com/nbextensions/widgets/notebook/js/extension.js?v=20160408223056
Widgets are not available.  Please install widgetsnbextension or ipywidgets 4.0

Would that be an error, or is the real problem here that when $http_upgrade is used it's not switching properly?

@wader

This comment has been minimized.

Copy link

wader commented Apr 10, 2016

I tried to run jupyter/notebook behind nginx-proxy myself now and it seems to work, i can create interpreters and evaluate python code etc.
I did this, first run nginx-proxy (you already have one running i guess?):

docker run -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy

Run jupyter/notebook:

 docker run --rm -e VIRTUAL_HOST=docker-machine jupyter/notebook

Surf to http://docker-machine. In my case docker-machine is a hostname resolving to a IP of a virtualbox machine running docker-machine.
To me https://github.com/calpolydatascience/jupyterhub-deploy-data301/blob/master/roles/nginx/templates/nginx.conf.j2 looks like a config example if you run a manually configured nginx in front of jupyter. But in case of nginx-proxy all or most of that will be done automatically for you. So my guess is it will work fine once you remove /etc/nginx/vhost.d/notes.rooday.com* and run the container and just add a VIRTUAL_HOST.

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 10, 2016

That didn't seem to work for me. I had nginx running so I removed and re ran a jupyter container with this command:

docker run --name jupyter -v "/home/jupyter/notebooks:/notebooks" -d -e VIRTUAL_HOST=notes.rooday.com jupyter/notebook:stable

I mounted a directory so the files would be separate from my home directory, and I added -d so I wouldn't have to leave it running in the console. The :stable at the end fixes the error I get in the chrome console about not having the widgets/extensions.

For me, notes.rooday.com resolves to my digital ocean server's ip, on which there is docker-engine serving the docker containers. Commenting out everything in the notes.rooday.com conf file in the vhost.d folder and then connecting to the jupyter notebook gave the connection error as before.

Leaving the code in the notes.rooday.com file and setting the Upgrade header to "websocket" lets me connect to the kernel like it was before, but then I can't get the assets when I try to run code.

@wader

This comment has been minimized.

Copy link

wader commented Apr 10, 2016

That is strange. Are you sure you deleted all files for notes.rooday.com in /etc/nginx/vhost.d?
Can you dump the generated nginx config with docker exec <nginx-proxy-container-id> cat /etc/nginx/conf.d/default.conf?

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 10, 2016

Here's the dump:

# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
server {
        server_name _; # This is just an invalid value which will never trigger on a real hostname.
        listen 80;
        access_log /var/log/nginx/access.log vhost;
        return 503;
}
upstream blog.rooday.com {
                        # ghost
                        server 172.17.0.3:2368;
}
server {
        server_name blog.rooday.com;
        listen 80 ;
        access_log /var/log/nginx/access.log vhost;
        location / {
                proxy_pass http://blog.rooday.com;
        }
}
upstream docker.rooday.com {
                        # webui
                        server 172.17.0.5:9000;
}
server {
        server_name docker.rooday.com;
        listen 80 ;
        access_log /var/log/nginx/access.log vhost;
        location / {
                proxy_pass http://docker.rooday.com;
                auth_basic      "Restricted docker.rooday.com";
                auth_basic_user_file    /etc/nginx/htpasswd/docker.rooday.com;
        }
}
upstream notes.rooday.com {
                        # jupyter
                        server 172.17.0.4:8888;
}
server {
        server_name notes.rooday.com;
        listen 80 ;
        access_log /var/log/nginx/access.log vhost;
        location / {
                proxy_pass http://notes.rooday.com;
        }
}
@wader

This comment has been minimized.

Copy link

wader commented Apr 10, 2016

Very strange, this my dump when it works:

$ docker run -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy
$ docker run --rm -e VIRTUAL_HOST=docker-machine jupyter/notebook
$ docker exec desperate_torvalds cat /etc/nginx/conf.d/default.conf
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    listen 80;
    access_log /var/log/nginx/access.log vhost;
    return 503;
}
upstream docker-machine {
            # boring_sinoussi
            server 172.17.0.9:8888;
}
server {
    server_name docker-machine;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    location / {
        proxy_pass http://docker-machine;
    }
}
@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 10, 2016

Could it be a difference in the way that docker-machine and docker engine handle connections?

@wader

This comment has been minimized.

Copy link

wader commented Apr 10, 2016

Should be the same, docker-machine is more or less "just" a tool for installing docker engine. In my case docker-machine runs docker-engine in a linux vm running in virtualbox. I just noticed you run jupyter/notebook:stable not jupyter/notebook but made no difference, works but logs an exception at start up ImportError: No module named 'mockextension'.

So you still see the Can "Upgrade" only to "WebSocket". error and see no Upgrade: websocket if you do the netcat trick?

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 10, 2016

I started a jupyter notebook container in debug by building one from this dockerfile:

FROM jupyter/notebook
CMD ["jupyter", "notebook", "--debug"]

Then running:

docker run --name jupyter-debug -v "/home/jupyter/notebooks:/notebooks" -d -e VIRTUAL_HOST=notes.rooday.com jupyter/notebook:debug

I ran netcat and the curl, giving this output:

listening on [any] 80 ...
172.17.0.2: inverse host lookup failed: Unknown host
connect to [172.17.0.6] from (UNKNOWN) [172.17.0.2] 48735
GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1
Host: notes.rooday.com
Connection: close
X-Real-IP: 172.17.0.1
X-Forwarded-For: 104.131.93.98, 172.17.0.1
X-Forwarded-Proto: http
Accept-Encoding: gzip
CF-IPCountry: US
CF-RAY: 291905aa4d1c46fe-EWR
CF-Visitor: {"scheme":"http"}
User-Agent: curl/7.35.0
Accept: */*
CF-Connecting-IP: 104.131.93.98
CF-Respect-Strong-Etag: 0

I notice that now the Connection header is "close" instead of "upgrade".

Here are the Jupyter logs:

[W 20:42:47.195 NotebookApp] 400 GET /api/kernels/bbb92b03-dca7-49fb-925d-a05bfc5529a5/channels?session_id=604C181624B4498983BB65A7240D59A9 (172.17.0.2) 1017.02ms referer=None
[D 20:42:47.195 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 20:42:48.243 NotebookApp] Initializing websocket connection /api/kernels/bbb92b03-dca7-49fb-925d-a05bfc5529a5/channels
[W 20:42:48.259 NotebookApp] 400 GET /api/kernels/bbb92b03-dca7-49fb-925d-a05bfc5529a5/channels?session_id=604C181624B4498983BB65A7240D59A9 (172.17.0.2) 16.37ms referer=None
[D 20:42:48.259 NotebookApp] Can "Upgrade" only to "WebSocket".

And here are the nginx logs:

nginx.1    | notes.rooday.com 172.17.0.1 - - [10/Apr/2016:20:42:47 +0000] "GET /api/kernels/bbb92b03-dca7-49fb-925d-a05bfc5529a5/channels?session_id=604C181624B4498983BB65A7240D59A9 HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [10/Apr/2016:20:42:48 +0000] "GET /api/kernels/bbb92b03-dca7-49fb-925d-a05bfc5529a5/channels?session_id=604C181624B4498983BB65A7240D59A9 HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [10/Apr/2016:20:42:50 +0000] "GET /api/kernels/bbb92b03-dca7-49fb-925d-a05bfc5529a5/channels?session_id=604C181624B4498983BB65A7240D59A9 HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [10/Apr/2016:20:42:54 +0000] "GET /api/kernels/bbb92b03-dca7-49fb-925d-a05bfc5529a5/channels?session_id=604C181624B4498983BB65A7240D59A9 HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [10/Apr/2016:20:43:02 +0000] "GET /api/kernels/bbb92b03-dca7-49fb-925d-a05bfc5529a5/channels?session_id=604C181624B4498983BB65A7240D59A9 HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
@wader

This comment has been minimized.

Copy link

wader commented Apr 10, 2016

Hmm what about the CF-* headers? is that cloudflare? could they strip for some reason?

@wader

This comment has been minimized.

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 10, 2016

Holy ****. Yeah I googled that immediately when you suggested cloudflare. I didn't realize that the CF meant that. Welp, grey-clouding notes.rooday.com in the cloudflare panel made it work instantly.

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 10, 2016

Anyone trying to use websockets behind an nginx-proxy and also is using cloudflare, grey-cloud the A record in the DNS panel.

Update:

Cloudflare now allows websockets for everyone, read more here.

@ROODAY ROODAY closed this Apr 10, 2016

@wader

This comment has been minimized.

Copy link

wader commented Apr 10, 2016

Oh 👍 😄

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 10, 2016

All that effort for such a simple solution. :p

@wader

This comment has been minimized.

Copy link

wader commented Apr 10, 2016

grey-cloud means cloudflare just tunnels traffic without any smartness?

@ROODAY

This comment has been minimized.

Copy link

ROODAY commented Apr 10, 2016

Yup, there's a little cloud icon in the DNS panel, when it's on traffic is sent through Cloudflare, when it's grey Cloudflare just sends it along.

@wader

This comment has been minimized.

Copy link

wader commented Apr 10, 2016

Ok very good to know.
I find it weirdly fun to debug strange stuff! 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment