Unable to establish websocket connection behind nginx reverse proxy

[ROODAY]

I have a Digital Ocean server running Ubuntu 14.04, with Jupyter Notebook (https://hub.docker.com/r/jupyter/notebook/) running inside a Docker container. I also have the nginx-proxy (https://github.com/jwilder/nginx-proxy) Docker container to serve as my reverse proxy.

I have it configured such that notes.rooday.com would be pointed to the Jupyter container, and that works as intended. I'm able to view the application, authenticate, make and edit files, etc. However, attempting to run an ipython notebook file gives a Connection Failed error:

A connection to the notebook server could not be established. The notebook will continue trying to reconnect, but until it does, you will NOT be able to run code. Check your network connection or notebook server configuration.

Looking inside the Chrome console, I see this error:

WebSocket connection to 'ws://notes.rooday.com/api/kernels/251adc99-3c1b-4d97-a9ae-9ecbfba3bc70/channels?session_id=28DC42BB11FD413A86D75BAE570CA379' failed: Error during WebSocket handshake: Unexpected response code: 400

The kernel attempts to reconnect, but to no avail. Checking the docker logs for the nginx proxy yields the following relevant lines:

nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:16 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:17 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:19 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"

and the logs for the Jupyter container show

[I 21:20:08.864 NotebookApp] 302 GET / (172.17.0.2) 3.53ms
[I 21:20:08.893 NotebookApp] 302 GET /tree (172.17.0.2) 4.38ms
[W 21:20:15.372 NotebookApp] 401 POST /login?next=%2Ftree (172.17.0.2) 4.28ms referer=http://notes.rooday.com/login?next=%2Ftree
[I 21:20:17.962 NotebookApp] 302 POST /login?next=%2Ftree (172.17.0.2) 4.61ms
[I 21:24:22.371 NotebookApp] 302 GET / (172.17.0.2) 2.79ms
[I 21:37:44.561 NotebookApp] 302 GET /notebooks/Untitled.ipynb (172.17.0.2) 1.37ms
[I 21:37:46.107 NotebookApp] 302 GET /tree (172.17.0.2) 1.54ms
[I 21:42:06.768 NotebookApp] 302 GET / (172.17.0.2) 4.08ms
[W 21:42:15.099 NotebookApp] 404 GET /nbextensions/widgets/extension.js?v=20160406210551 (172.17.0.2) 2.85ms referer=http://notes.rooday.com/notebooks/Untitled.ipynb
[I 21:42:15.199 NotebookApp] Kernel started: 007c6099-febe-40b3-847c-5d90bcc4cf4b
[W 21:42:15.271 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20160406210551 (172.17.0.2) 9.50ms referer=http://notes.rooday.com/notebooks/Untitled.ipynb
[W 21:42:16.172 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 878.60ms referer=None
[W 21:42:17.221 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 7.58ms referer=None
[W 21:42:19.288 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 20.13ms referer=None
[W 21:42:23.340 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 7.98ms referer=None
[W 21:42:31.394 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 6.49ms referer=None
[W 21:42:47.459 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 9.92ms referer=None

Based on this, and asking in the nginx-proxy issues page, it would seem that Jupyter is the cause of the problem. The nginx proxy is configured to allow websocket connections, but they're still not going through. Has anyone else attempted to run Jupyter behind a reverse proxy, or know enough to help me find a workaround?

[willingc]

@ROODAY Here's one approach that is documented in the jupyter/docker-stacks repo: https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes#running-behind-a-nginx-proxy

cc/ @parente

[ROODAY]

@willingc I tried following the conf but I still am not able to make a web socket connection. nginx-proxy allows mounting of a folder of conf files for specific domains that get included into the main conf, so I was able to add this to the server block for jupyter:

client_max_body_size 50M;
error_log /var/log/nginx/error.log;

location / {
        proxy_pass http://notes.rooday.com;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
}

location ~* /(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
        proxy_pass http://notes.rooday.com;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
}

The only parts from the suggested conf file that I did not include were these:

        ssl on;
        ssl_certificate /data/cert.crt;
        ssl_certificate_key /data/key.key;

        ssl_ciphers "AES128+EECDH:AES128+EDH";
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        add_header X-Content-Type-Options nosniff;
        ssl_stapling on; # Requires nginx >= 1.3.7
        ssl_stapling_verify on; # Requires nginx => 1.3.7
        resolver_timeout 5s;

Is SSL required to get websockets working?

Edit:

Here is my jupyter_notebook_config.py file

c.NotebookApp.ip = '172.17.0.4'
c.NotebookApp.password = u'whoathatsnotsupposedtobehere'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 8888
c.NotebookApp.trust_xheaders = True
c.NotebookApp.allow_origin = '*'

[minrk]

That looks right, but I'll ping @rgbkrk for any ideas about what might not be right about the nginx config for websocket forwarding. My guess is it's a tiny regex or priority bug preventing the Upgrade block from being applied correctly.

[ROODAY]

Is there a way to run the Jupyter Notebook container in a debug mode to see more information on what happens when I load an IPython notebook?

[willingc]

@ROODAY Jupyter Notebook can be run in debug mode with the command jupyter notebook --debug

[ROODAY]

I tried debugging it within the Docker container since I'm sure the issue is related to everything being in a Docker container. However, when I tried to build a Docker image of Jupyter Notebook with a modified version of the Docker file that had the --debug flag (here: http://pastebin.com/AMtbq7bX), I received an error about setup.py not being found.

This is the command I ran to build the image (the docker file was at /home/rooday/dockerfile):

sudo docker build -t jupyter-debug:test --rm=true /home/rooday/

These were the last few lines of the build:

Directory '/usr/src/jupyter-notebook' is not installable. File 'setup.py' not found.
The command '/bin/sh -c BUILD_DEPS="nodejs-legacy npm" &&     apt-get update -qq &&     DEBIAN_FRONTEND=noninteractive apt-get install -yq $BUILD_DEPS &&         pip3 install --no-cache-dir /usr/src/jupyter-notebook &&     pip2 install --no-cache-dir widgetsnbextension &&     pip3 install --no-cache-dir widgetsnbextension &&         npm cache clean &&     apt-get clean &&     rm -rf /root/.npm &&     rm -rf /root/.cache &&     rm -rf /root/.config &&     rm -rf /root/.local &&     rm -rf /root/tmp &&     rm -rf /var/lib/apt/lists/* &&     apt-get purge -y --auto-remove         -o APT::AutoRemove::RecommendsImportant=false -o APT::AutoRemove::SuggestsImportant=false $BUILD_DEPS' returned a non-zero code: 1

Edit: If necessary, here's the full console output when the build command was ran: http://pastebin.com/ybJPnwC0

[willingc]

What happens if you run without --rm=true

sudo docker build -t jupyter-debug:test /home/rooday/

[ROODAY]

That gives the same error of

Directory '/usr/src/jupyter-notebook' is not installable. File 'setup.py' not found.
The command '/bin/sh -c BUILD_DEPS="nodejs-legacy npm" &&     apt-get update -qq &&     DEBIAN_FRONTEND=noninteractive apt-get install -yq $BUILD_DEPS &&         pip3 install --no-cache-dir /usr/src/jupyter-notebook &&     pip2 install --no-cache-dir widgetsnbextension &&     pip3 install --no-cache-dir widgetsnbextension &&         npm cache clean &&     apt-get clean &&     rm -rf /root/.npm &&     rm -rf /root/.cache &&     rm -rf /root/.config &&     rm -rf /root/.local &&     rm -rf /root/tmp &&     rm -rf /var/lib/apt/lists/* &&     apt-get purge -y --auto-remove         -o APT::AutoRemove::RecommendsImportant=false -o APT::AutoRemove::SuggestsImportant=false $BUILD_DEPS' returned a non-zero code: 1

[ROODAY]

A friend pointed out to me that the build error was likely due to the fact that the environment I was building it in did not have all the same folders and resources that the original was built in, and that I should instead use a dockerfile to extend the already made jupyter/notebook container.

Using this Dockerfile:

FROM jupyter/notebook
CMD ["jupyter", "notebook", "--debug"]

and this build command:
sudo docker build -t jupyter/notebook:debug /home/rooday/

I was able to build a Docker container that ran Jupyter Notebook in debug. Upon connecting to the notebook, creating a new IPython 3 Notebook and editing it, I received the same connection as before. This is the last few lines of output from the Jupyter Notebook container's logs (full logs here: http://pastebin.com/vTqc1Xug):

[W 22:31:20.357 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 1269.18ms referer=None
[D 22:31:20.358 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:21.388 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:21.403 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 15.53ms referer=None
[D 22:31:21.403 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:23.437 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:23.450 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 12.59ms referer=None
[D 22:31:23.450 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:25.284 NotebookApp] 200 GET /custom/custom.css (172.17.0.2) 6.46ms
[D 22:31:25.433 NotebookApp] 200 GET /static/notebook/js/main.min.js.map (172.17.0.2) 71.94ms
[D 22:31:25.451 NotebookApp] 304 GET /static/style/style.min.css.map (172.17.0.2) 8.74ms
[D 22:31:27.480 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:27.490 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 10.67ms referer=None
[D 22:31:27.491 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:35.538 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:35.549 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 12.31ms referer=None
[D 22:31:35.550 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:51.590 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:51.616 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 30.62ms referer=None
[D 22:31:51.617 NotebookApp] Can "Upgrade" only to "WebSocket".

The difference I notice now from when it wasn't in debug is there are now lines saying

Can "Upgrade" only to "WebSocket".

Is there a setting that is preventing the connection from being upgraded to a websocket connection?

[ROODAY]

@willingc
I've been talking to a person from the nginx-proxy repo to help figure out the issues on the nginx side, but I made some progress in that one config allowed me to make a kernel websocket connection. However, when I did that, I saw these errors in the chrome console:

actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later...
Failed to load resource: the server responded with a status of 404 (Not Found) 
http://notes.rooday.com/nbextensions/widgets/extension.js?v=20160408223056
Failed to load resource: the server responded with a status of 404 (Not Found) http://notes.rooday.com/nbextensions/widgets/notebook/js/extension.js?v=20160408223056
Widgets are not available.  Please install widgetsnbextension or ipywidgets 4.0

Is this something that should be ignored, or could it be a reason I'm having trouble?

[willingc]

@ROODAY In the jupyter/notebook's Dockerfile, widgetsnbextension is installed for Python 2 and 3. I would recommend installing it.

[ROODAY]

@willingc Could this be the issue? The dockerfile for Jupyter/Notebook on DockerHub does not match the docker file from the Jupyter/Notebook repository.

https://hub.docker.com/r/jupyter/notebook/~/dockerfile/

Edit:

I run a new jupyter/notebook container with the stable tag with this command:

docker run --name jupyter -d -p 8888:8888 -v "/home/jupyter/notebooks:/notebooks" jupyter/notebook:stable

And from doing that it seemed to install the extensions as I saw this in the chrome console:

Session: kernel_created (8631fdc3-87a9-405f-bdf5-cc486238fdcc)
Starting WebSockets: ws://notes.rooday.com/api/kernels/fc593fbd-f32e-4ecb-a8fb-1a665ca916ed
Loading extension: widgets/notebook/js/extension
loaded widgets
Kernel: kernel_connected (fc593fbd-f32e-4ecb-a8fb-1a665ca916ed)

However, I still get 304 errors when trying to run code as I have to force Upgrade "websocket" to make a connection, therefore interfering with normal requests for assets like Javascript and CSS.

[willingc]

@minrk @parente Which Dockerfile should @ROODAY use? The one on DockerHub, the on in the jupyter/notebook repo, or one of the ones in jupyter/docker-stacks? Thanks.

[ROODAY]

@willingc Nevermind, I found the issue. As it turns out, it was not related to either nginx-proxy or Jupyter, and rather Cloudflare. At the moment, Websocket connections through Cloudflare is in beta, and by default is disabled. Cloudflare recommends as a workaround to simply grey-cloud A records that make use of websocket connections, so in my case that was notes.rooday.com

https://support.cloudflare.com/hc/en-us/articles/200169466-Can-I-use-CloudFlare-with-WebSockets-

For anyone in the future attempting to use Cloudflare, Jupyter/notebook, and nginx-proxy, refer to this:
nginx-proxy/nginx-proxy#412

Update:

Cloudflare now allows websockets for everyone, read more here.

[willingc]

@ROODAY I'm glad you got it resolved. Thanks for passing along the content to help others too. 🍰

[lucaswxp]

+1 I had similar issue and the problem was solved by disabling (grey-cloud) http-proxy in cloudflare.

[jakocoo]

If you are not on CloudFlare and still have the issue. Here's the fix:
https://stackoverflow.com/questions/22665809/how-to-configure-ipython-behind-nginx-in-a-subpath/23912400#23912400

proxy_set_header Origin ""; is the key.

[takluyver]

Does anyone know if that has security implications? IIRC, if the Origin header is unset, the server skips the usual cross-origin checks, assuming that it's a non-browser client connecting. Will the proxy server do cross-origin checks itself before passing the request on? Or will it just remove the origin header and pass it on regardless?

[rgbkrk]

It has security implications for the websockets on the server side (where we put protections in place), however the browser will continue to do CORS checks.

[takluyver]

Are the browser checks sufficient to protect users? Will it allow making requests, even if the user can't see the results?

[rgbkrk]

The browser will make the requests (though our XSRF stuff should prevent that at the server side) still, then check the headers from the response.

[takluyver]

So a malicious site could e.g. start lots of kernels by making POST requests, even though it couldn't then use those kernels?

[takluyver]

Or will the XSRF prevent that? I have trouble remembering how all these different bits work?

[rgbkrk]

The XSRF protection will prevent that, assuming they don't have the token.

[takluyver]

OK, thanks Kyle :-)

[disrupted]

@jakocoo

If you are not on CloudFlare and still have the issue. Here's the fix:
https://stackoverflow.com/questions/22665809/how-to-configure-ipython-behind-nginx-in-a-subpath/23912400#23912400

proxy_set_header Origin ""; is the key.

I tried that but I am still having issues with Safari saying "Kernel starting, please wait..." (which I learned happens when it can't establish a secure websocket).

my config looks like this:

nginx jupyter.conf

location /jupyter/ {
        proxy_pass http://localhost:8888;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_set_header Origin "";
}

location ~* /jupyter/(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
        proxy_pass http://localhost:8888;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Origin "";
}

jupyter_notebook_config.py

c.NotebookApp.allow_origin = '*'
c.NotebookApp.base_url = '/jupyter'
c.NotebookApp.open_browser = False
c.NotebookApp.password = 'sha1:REDACTED'
c.NotebookApp.port = 8888
c.NotebookApp.trust_xheaders = True