New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to establish websocket connection behind nginx reverse proxy #1311

Closed
ROODAY opened this Issue Apr 6, 2016 · 26 comments

Comments

Projects
None yet
8 participants
@ROODAY

ROODAY commented Apr 6, 2016

I have a Digital Ocean server running Ubuntu 14.04, with Jupyter Notebook (https://hub.docker.com/r/jupyter/notebook/) running inside a Docker container. I also have the nginx-proxy (https://github.com/jwilder/nginx-proxy) Docker container to serve as my reverse proxy.

I have it configured such that notes.rooday.com would be pointed to the Jupyter container, and that works as intended. I'm able to view the application, authenticate, make and edit files, etc. However, attempting to run an ipython notebook file gives a Connection Failed error:

A connection to the notebook server could not be established. The notebook will continue trying to reconnect, but until it does, you will NOT be able to run code. Check your network connection or notebook server configuration.

Looking inside the Chrome console, I see this error:

WebSocket connection to 'ws://notes.rooday.com/api/kernels/251adc99-3c1b-4d97-a9ae-9ecbfba3bc70/channels?session_id=28DC42BB11FD413A86D75BAE570CA379' failed: Error during WebSocket handshake: Unexpected response code: 400

The kernel attempts to reconnect, but to no avail. Checking the docker logs for the nginx proxy yields the following relevant lines:

nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:16 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:17 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:19 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"

and the logs for the Jupyter container show

[I 21:20:08.864 NotebookApp] 302 GET / (172.17.0.2) 3.53ms
[I 21:20:08.893 NotebookApp] 302 GET /tree (172.17.0.2) 4.38ms
[W 21:20:15.372 NotebookApp] 401 POST /login?next=%2Ftree (172.17.0.2) 4.28ms referer=http://notes.rooday.com/login?next=%2Ftree
[I 21:20:17.962 NotebookApp] 302 POST /login?next=%2Ftree (172.17.0.2) 4.61ms
[I 21:24:22.371 NotebookApp] 302 GET / (172.17.0.2) 2.79ms
[I 21:37:44.561 NotebookApp] 302 GET /notebooks/Untitled.ipynb (172.17.0.2) 1.37ms
[I 21:37:46.107 NotebookApp] 302 GET /tree (172.17.0.2) 1.54ms
[I 21:42:06.768 NotebookApp] 302 GET / (172.17.0.2) 4.08ms
[W 21:42:15.099 NotebookApp] 404 GET /nbextensions/widgets/extension.js?v=20160406210551 (172.17.0.2) 2.85ms referer=http://notes.rooday.com/notebooks/Untitled.ipynb
[I 21:42:15.199 NotebookApp] Kernel started: 007c6099-febe-40b3-847c-5d90bcc4cf4b
[W 21:42:15.271 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20160406210551 (172.17.0.2) 9.50ms referer=http://notes.rooday.com/notebooks/Untitled.ipynb
[W 21:42:16.172 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 878.60ms referer=None
[W 21:42:17.221 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 7.58ms referer=None
[W 21:42:19.288 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 20.13ms referer=None
[W 21:42:23.340 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 7.98ms referer=None
[W 21:42:31.394 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 6.49ms referer=None
[W 21:42:47.459 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 9.92ms referer=None

Based on this, and asking in the nginx-proxy issues page, it would seem that Jupyter is the cause of the problem. The nginx proxy is configured to allow websocket connections, but they're still not going through. Has anyone else attempted to run Jupyter behind a reverse proxy, or know enough to help me find a workaround?

@willingc

This comment has been minimized.

Show comment
Hide comment
@willingc

willingc Apr 6, 2016

Member

@ROODAY Here's one approach that is documented in the jupyter/docker-stacks repo: https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes#running-behind-a-nginx-proxy

cc/ @parente

Member

willingc commented Apr 6, 2016

@ROODAY Here's one approach that is documented in the jupyter/docker-stacks repo: https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes#running-behind-a-nginx-proxy

cc/ @parente

@ROODAY

This comment has been minimized.

Show comment
Hide comment
@ROODAY

ROODAY Apr 7, 2016

@willingc I tried following the conf but I still am not able to make a web socket connection. nginx-proxy allows mounting of a folder of conf files for specific domains that get included into the main conf, so I was able to add this to the server block for jupyter:

client_max_body_size 50M;
error_log /var/log/nginx/error.log;

location / {
        proxy_pass http://notes.rooday.com;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
}

location ~* /(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
        proxy_pass http://notes.rooday.com;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
}

The only parts from the suggested conf file that I did not include were these:

        ssl on;
        ssl_certificate /data/cert.crt;
        ssl_certificate_key /data/key.key;

        ssl_ciphers "AES128+EECDH:AES128+EDH";
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        add_header X-Content-Type-Options nosniff;
        ssl_stapling on; # Requires nginx >= 1.3.7
        ssl_stapling_verify on; # Requires nginx => 1.3.7
        resolver_timeout 5s;

Is SSL required to get websockets working?

Edit:

Here is my jupyter_notebook_config.py file

c.NotebookApp.ip = '172.17.0.4'
c.NotebookApp.password = u'whoathatsnotsupposedtobehere'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 8888
c.NotebookApp.trust_xheaders = True
c.NotebookApp.allow_origin = '*'

ROODAY commented Apr 7, 2016

@willingc I tried following the conf but I still am not able to make a web socket connection. nginx-proxy allows mounting of a folder of conf files for specific domains that get included into the main conf, so I was able to add this to the server block for jupyter:

client_max_body_size 50M;
error_log /var/log/nginx/error.log;

location / {
        proxy_pass http://notes.rooday.com;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
}

location ~* /(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
        proxy_pass http://notes.rooday.com;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
}

The only parts from the suggested conf file that I did not include were these:

        ssl on;
        ssl_certificate /data/cert.crt;
        ssl_certificate_key /data/key.key;

        ssl_ciphers "AES128+EECDH:AES128+EDH";
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        add_header X-Content-Type-Options nosniff;
        ssl_stapling on; # Requires nginx >= 1.3.7
        ssl_stapling_verify on; # Requires nginx => 1.3.7
        resolver_timeout 5s;

Is SSL required to get websockets working?

Edit:

Here is my jupyter_notebook_config.py file

c.NotebookApp.ip = '172.17.0.4'
c.NotebookApp.password = u'whoathatsnotsupposedtobehere'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 8888
c.NotebookApp.trust_xheaders = True
c.NotebookApp.allow_origin = '*'
@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Apr 7, 2016

Member

That looks right, but I'll ping @rgbkrk for any ideas about what might not be right about the nginx config for websocket forwarding. My guess is it's a tiny regex or priority bug preventing the Upgrade block from being applied correctly.

Member

minrk commented Apr 7, 2016

That looks right, but I'll ping @rgbkrk for any ideas about what might not be right about the nginx config for websocket forwarding. My guess is it's a tiny regex or priority bug preventing the Upgrade block from being applied correctly.

@ROODAY

This comment has been minimized.

Show comment
Hide comment
@ROODAY

ROODAY Apr 8, 2016

Is there a way to run the Jupyter Notebook container in a debug mode to see more information on what happens when I load an IPython notebook?

ROODAY commented Apr 8, 2016

Is there a way to run the Jupyter Notebook container in a debug mode to see more information on what happens when I load an IPython notebook?

@willingc

This comment has been minimized.

Show comment
Hide comment
@willingc

willingc Apr 8, 2016

Member

@ROODAY Jupyter Notebook can be run in debug mode with the command jupyter notebook --debug

Member

willingc commented Apr 8, 2016

@ROODAY Jupyter Notebook can be run in debug mode with the command jupyter notebook --debug

@ROODAY

This comment has been minimized.

Show comment
Hide comment
@ROODAY

ROODAY Apr 8, 2016

I tried debugging it within the Docker container since I'm sure the issue is related to everything being in a Docker container. However, when I tried to build a Docker image of Jupyter Notebook with a modified version of the Docker file that had the --debug flag (here: http://pastebin.com/AMtbq7bX), I received an error about setup.py not being found.

This is the command I ran to build the image (the docker file was at /home/rooday/dockerfile):

sudo docker build -t jupyter-debug:test --rm=true /home/rooday/

These were the last few lines of the build:

Directory '/usr/src/jupyter-notebook' is not installable. File 'setup.py' not found.
The command '/bin/sh -c BUILD_DEPS="nodejs-legacy npm" &&     apt-get update -qq &&     DEBIAN_FRONTEND=noninteractive apt-get install -yq $BUILD_DEPS &&         pip3 install --no-cache-dir /usr/src/jupyter-notebook &&     pip2 install --no-cache-dir widgetsnbextension &&     pip3 install --no-cache-dir widgetsnbextension &&         npm cache clean &&     apt-get clean &&     rm -rf /root/.npm &&     rm -rf /root/.cache &&     rm -rf /root/.config &&     rm -rf /root/.local &&     rm -rf /root/tmp &&     rm -rf /var/lib/apt/lists/* &&     apt-get purge -y --auto-remove         -o APT::AutoRemove::RecommendsImportant=false -o APT::AutoRemove::SuggestsImportant=false $BUILD_DEPS' returned a non-zero code: 1

Edit: If necessary, here's the full console output when the build command was ran: http://pastebin.com/ybJPnwC0

ROODAY commented Apr 8, 2016

I tried debugging it within the Docker container since I'm sure the issue is related to everything being in a Docker container. However, when I tried to build a Docker image of Jupyter Notebook with a modified version of the Docker file that had the --debug flag (here: http://pastebin.com/AMtbq7bX), I received an error about setup.py not being found.

This is the command I ran to build the image (the docker file was at /home/rooday/dockerfile):

sudo docker build -t jupyter-debug:test --rm=true /home/rooday/

These were the last few lines of the build:

Directory '/usr/src/jupyter-notebook' is not installable. File 'setup.py' not found.
The command '/bin/sh -c BUILD_DEPS="nodejs-legacy npm" &&     apt-get update -qq &&     DEBIAN_FRONTEND=noninteractive apt-get install -yq $BUILD_DEPS &&         pip3 install --no-cache-dir /usr/src/jupyter-notebook &&     pip2 install --no-cache-dir widgetsnbextension &&     pip3 install --no-cache-dir widgetsnbextension &&         npm cache clean &&     apt-get clean &&     rm -rf /root/.npm &&     rm -rf /root/.cache &&     rm -rf /root/.config &&     rm -rf /root/.local &&     rm -rf /root/tmp &&     rm -rf /var/lib/apt/lists/* &&     apt-get purge -y --auto-remove         -o APT::AutoRemove::RecommendsImportant=false -o APT::AutoRemove::SuggestsImportant=false $BUILD_DEPS' returned a non-zero code: 1

Edit: If necessary, here's the full console output when the build command was ran: http://pastebin.com/ybJPnwC0

@willingc

This comment has been minimized.

Show comment
Hide comment
@willingc

willingc Apr 8, 2016

Member

What happens if you run without --rm=true

sudo docker build -t jupyter-debug:test /home/rooday/

Member

willingc commented Apr 8, 2016

What happens if you run without --rm=true

sudo docker build -t jupyter-debug:test /home/rooday/

@ROODAY

This comment has been minimized.

Show comment
Hide comment
@ROODAY

ROODAY Apr 8, 2016

That gives the same error of

Directory '/usr/src/jupyter-notebook' is not installable. File 'setup.py' not found.
The command '/bin/sh -c BUILD_DEPS="nodejs-legacy npm" &&     apt-get update -qq &&     DEBIAN_FRONTEND=noninteractive apt-get install -yq $BUILD_DEPS &&         pip3 install --no-cache-dir /usr/src/jupyter-notebook &&     pip2 install --no-cache-dir widgetsnbextension &&     pip3 install --no-cache-dir widgetsnbextension &&         npm cache clean &&     apt-get clean &&     rm -rf /root/.npm &&     rm -rf /root/.cache &&     rm -rf /root/.config &&     rm -rf /root/.local &&     rm -rf /root/tmp &&     rm -rf /var/lib/apt/lists/* &&     apt-get purge -y --auto-remove         -o APT::AutoRemove::RecommendsImportant=false -o APT::AutoRemove::SuggestsImportant=false $BUILD_DEPS' returned a non-zero code: 1

ROODAY commented Apr 8, 2016

That gives the same error of

Directory '/usr/src/jupyter-notebook' is not installable. File 'setup.py' not found.
The command '/bin/sh -c BUILD_DEPS="nodejs-legacy npm" &&     apt-get update -qq &&     DEBIAN_FRONTEND=noninteractive apt-get install -yq $BUILD_DEPS &&         pip3 install --no-cache-dir /usr/src/jupyter-notebook &&     pip2 install --no-cache-dir widgetsnbextension &&     pip3 install --no-cache-dir widgetsnbextension &&         npm cache clean &&     apt-get clean &&     rm -rf /root/.npm &&     rm -rf /root/.cache &&     rm -rf /root/.config &&     rm -rf /root/.local &&     rm -rf /root/tmp &&     rm -rf /var/lib/apt/lists/* &&     apt-get purge -y --auto-remove         -o APT::AutoRemove::RecommendsImportant=false -o APT::AutoRemove::SuggestsImportant=false $BUILD_DEPS' returned a non-zero code: 1
@ROODAY

This comment has been minimized.

Show comment
Hide comment
@ROODAY

ROODAY Apr 8, 2016

A friend pointed out to me that the build error was likely due to the fact that the environment I was building it in did not have all the same folders and resources that the original was built in, and that I should instead use a dockerfile to extend the already made jupyter/notebook container.

Using this Dockerfile:

FROM jupyter/notebook
CMD ["jupyter", "notebook", "--debug"]

and this build command:
sudo docker build -t jupyter/notebook:debug /home/rooday/

I was able to build a Docker container that ran Jupyter Notebook in debug. Upon connecting to the notebook, creating a new IPython 3 Notebook and editing it, I received the same connection as before. This is the last few lines of output from the Jupyter Notebook container's logs (full logs here: http://pastebin.com/vTqc1Xug):

[W 22:31:20.357 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 1269.18ms referer=None
[D 22:31:20.358 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:21.388 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:21.403 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 15.53ms referer=None
[D 22:31:21.403 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:23.437 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:23.450 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 12.59ms referer=None
[D 22:31:23.450 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:25.284 NotebookApp] 200 GET /custom/custom.css (172.17.0.2) 6.46ms
[D 22:31:25.433 NotebookApp] 200 GET /static/notebook/js/main.min.js.map (172.17.0.2) 71.94ms
[D 22:31:25.451 NotebookApp] 304 GET /static/style/style.min.css.map (172.17.0.2) 8.74ms
[D 22:31:27.480 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:27.490 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 10.67ms referer=None
[D 22:31:27.491 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:35.538 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:35.549 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 12.31ms referer=None
[D 22:31:35.550 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:51.590 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:51.616 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 30.62ms referer=None
[D 22:31:51.617 NotebookApp] Can "Upgrade" only to "WebSocket".

The difference I notice now from when it wasn't in debug is there are now lines saying

Can "Upgrade" only to "WebSocket".

Is there a setting that is preventing the connection from being upgraded to a websocket connection?

ROODAY commented Apr 8, 2016

A friend pointed out to me that the build error was likely due to the fact that the environment I was building it in did not have all the same folders and resources that the original was built in, and that I should instead use a dockerfile to extend the already made jupyter/notebook container.

Using this Dockerfile:

FROM jupyter/notebook
CMD ["jupyter", "notebook", "--debug"]

and this build command:
sudo docker build -t jupyter/notebook:debug /home/rooday/

I was able to build a Docker container that ran Jupyter Notebook in debug. Upon connecting to the notebook, creating a new IPython 3 Notebook and editing it, I received the same connection as before. This is the last few lines of output from the Jupyter Notebook container's logs (full logs here: http://pastebin.com/vTqc1Xug):

[W 22:31:20.357 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 1269.18ms referer=None
[D 22:31:20.358 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:21.388 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:21.403 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 15.53ms referer=None
[D 22:31:21.403 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:23.437 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:23.450 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 12.59ms referer=None
[D 22:31:23.450 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:25.284 NotebookApp] 200 GET /custom/custom.css (172.17.0.2) 6.46ms
[D 22:31:25.433 NotebookApp] 200 GET /static/notebook/js/main.min.js.map (172.17.0.2) 71.94ms
[D 22:31:25.451 NotebookApp] 304 GET /static/style/style.min.css.map (172.17.0.2) 8.74ms
[D 22:31:27.480 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:27.490 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 10.67ms referer=None
[D 22:31:27.491 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:35.538 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:35.549 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 12.31ms referer=None
[D 22:31:35.550 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:51.590 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:51.616 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 30.62ms referer=None
[D 22:31:51.617 NotebookApp] Can "Upgrade" only to "WebSocket".

The difference I notice now from when it wasn't in debug is there are now lines saying

Can "Upgrade" only to "WebSocket".

Is there a setting that is preventing the connection from being upgraded to a websocket connection?

@ROODAY

This comment has been minimized.

Show comment
Hide comment
@ROODAY

ROODAY Apr 10, 2016

@willingc
I've been talking to a person from the nginx-proxy repo to help figure out the issues on the nginx side, but I made some progress in that one config allowed me to make a kernel websocket connection. However, when I did that, I saw these errors in the chrome console:

actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later...
Failed to load resource: the server responded with a status of 404 (Not Found) 
http://notes.rooday.com/nbextensions/widgets/extension.js?v=20160408223056
Failed to load resource: the server responded with a status of 404 (Not Found) http://notes.rooday.com/nbextensions/widgets/notebook/js/extension.js?v=20160408223056
Widgets are not available.  Please install widgetsnbextension or ipywidgets 4.0

Is this something that should be ignored, or could it be a reason I'm having trouble?

ROODAY commented Apr 10, 2016

@willingc
I've been talking to a person from the nginx-proxy repo to help figure out the issues on the nginx side, but I made some progress in that one config allowed me to make a kernel websocket connection. However, when I did that, I saw these errors in the chrome console:

actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later...
Failed to load resource: the server responded with a status of 404 (Not Found) 
http://notes.rooday.com/nbextensions/widgets/extension.js?v=20160408223056
Failed to load resource: the server responded with a status of 404 (Not Found) http://notes.rooday.com/nbextensions/widgets/notebook/js/extension.js?v=20160408223056
Widgets are not available.  Please install widgetsnbextension or ipywidgets 4.0

Is this something that should be ignored, or could it be a reason I'm having trouble?

@willingc

This comment has been minimized.

Show comment
Hide comment
@willingc

willingc Apr 10, 2016

Member

@ROODAY In the jupyter/notebook's Dockerfile, widgetsnbextension is installed for Python 2 and 3. I would recommend installing it.

Member

willingc commented Apr 10, 2016

@ROODAY In the jupyter/notebook's Dockerfile, widgetsnbextension is installed for Python 2 and 3. I would recommend installing it.

@ROODAY

This comment has been minimized.

Show comment
Hide comment
@ROODAY

ROODAY Apr 10, 2016

@willingc Could this be the issue? The dockerfile for Jupyter/Notebook on DockerHub does not match the docker file from the Jupyter/Notebook repository.

https://hub.docker.com/r/jupyter/notebook/~/dockerfile/

Edit:

I run a new jupyter/notebook container with the stable tag with this command:

docker run --name jupyter -d -p 8888:8888 -v "/home/jupyter/notebooks:/notebooks" jupyter/notebook:stable

And from doing that it seemed to install the extensions as I saw this in the chrome console:

Session: kernel_created (8631fdc3-87a9-405f-bdf5-cc486238fdcc)
Starting WebSockets: ws://notes.rooday.com/api/kernels/fc593fbd-f32e-4ecb-a8fb-1a665ca916ed
Loading extension: widgets/notebook/js/extension
loaded widgets
Kernel: kernel_connected (fc593fbd-f32e-4ecb-a8fb-1a665ca916ed)

However, I still get 304 errors when trying to run code as I have to force Upgrade "websocket" to make a connection, therefore interfering with normal requests for assets like Javascript and CSS.

ROODAY commented Apr 10, 2016

@willingc Could this be the issue? The dockerfile for Jupyter/Notebook on DockerHub does not match the docker file from the Jupyter/Notebook repository.

https://hub.docker.com/r/jupyter/notebook/~/dockerfile/

Edit:

I run a new jupyter/notebook container with the stable tag with this command:

docker run --name jupyter -d -p 8888:8888 -v "/home/jupyter/notebooks:/notebooks" jupyter/notebook:stable

And from doing that it seemed to install the extensions as I saw this in the chrome console:

Session: kernel_created (8631fdc3-87a9-405f-bdf5-cc486238fdcc)
Starting WebSockets: ws://notes.rooday.com/api/kernels/fc593fbd-f32e-4ecb-a8fb-1a665ca916ed
Loading extension: widgets/notebook/js/extension
loaded widgets
Kernel: kernel_connected (fc593fbd-f32e-4ecb-a8fb-1a665ca916ed)

However, I still get 304 errors when trying to run code as I have to force Upgrade "websocket" to make a connection, therefore interfering with normal requests for assets like Javascript and CSS.

@willingc

This comment has been minimized.

Show comment
Hide comment
@willingc

willingc Apr 10, 2016

Member

@minrk @parente Which Dockerfile should @ROODAY use? The one on DockerHub, the on in the jupyter/notebook repo, or one of the ones in jupyter/docker-stacks? Thanks.

Member

willingc commented Apr 10, 2016

@minrk @parente Which Dockerfile should @ROODAY use? The one on DockerHub, the on in the jupyter/notebook repo, or one of the ones in jupyter/docker-stacks? Thanks.

@ROODAY

This comment has been minimized.

Show comment
Hide comment
@ROODAY

ROODAY Apr 10, 2016

@willingc Nevermind, I found the issue. As it turns out, it was not related to either nginx-proxy or Jupyter, and rather Cloudflare. At the moment, Websocket connections through Cloudflare is in beta, and by default is disabled. Cloudflare recommends as a workaround to simply grey-cloud A records that make use of websocket connections, so in my case that was notes.rooday.com

https://support.cloudflare.com/hc/en-us/articles/200169466-Can-I-use-CloudFlare-with-WebSockets-

For anyone in the future attempting to use Cloudflare, Jupyter/notebook, and nginx-proxy, refer to this:
jwilder/nginx-proxy#412

Update:

Cloudflare now allows websockets for everyone, read more here.

ROODAY commented Apr 10, 2016

@willingc Nevermind, I found the issue. As it turns out, it was not related to either nginx-proxy or Jupyter, and rather Cloudflare. At the moment, Websocket connections through Cloudflare is in beta, and by default is disabled. Cloudflare recommends as a workaround to simply grey-cloud A records that make use of websocket connections, so in my case that was notes.rooday.com

https://support.cloudflare.com/hc/en-us/articles/200169466-Can-I-use-CloudFlare-with-WebSockets-

For anyone in the future attempting to use Cloudflare, Jupyter/notebook, and nginx-proxy, refer to this:
jwilder/nginx-proxy#412

Update:

Cloudflare now allows websockets for everyone, read more here.

@ROODAY ROODAY closed this Apr 10, 2016

@willingc

This comment has been minimized.

Show comment
Hide comment
@willingc

willingc Apr 10, 2016

Member

@ROODAY I'm glad you got it resolved. Thanks for passing along the content to help others too. 🍰

Member

willingc commented Apr 10, 2016

@ROODAY I'm glad you got it resolved. Thanks for passing along the content to help others too. 🍰

@minrk minrk added this to the no action milestone May 30, 2016

@lucaswxp

This comment has been minimized.

Show comment
Hide comment
@lucaswxp

lucaswxp Oct 11, 2016

+1 I had similar issue and the problem was solved by disabling (grey-cloud) http-proxy in cloudflare.

lucaswxp commented Oct 11, 2016

+1 I had similar issue and the problem was solved by disabling (grey-cloud) http-proxy in cloudflare.

@jakocoo

This comment has been minimized.

Show comment
Hide comment
@jakocoo

jakocoo May 25, 2017

If you are not on CloudFlare and still have the issue. Here's the fix:
https://stackoverflow.com/questions/22665809/how-to-configure-ipython-behind-nginx-in-a-subpath/23912400#23912400

proxy_set_header Origin ""; is the key.

jakocoo commented May 25, 2017

If you are not on CloudFlare and still have the issue. Here's the fix:
https://stackoverflow.com/questions/22665809/how-to-configure-ipython-behind-nginx-in-a-subpath/23912400#23912400

proxy_set_header Origin ""; is the key.

@takluyver

This comment has been minimized.

Show comment
Hide comment
@takluyver

takluyver May 25, 2017

Member

Does anyone know if that has security implications? IIRC, if the Origin header is unset, the server skips the usual cross-origin checks, assuming that it's a non-browser client connecting. Will the proxy server do cross-origin checks itself before passing the request on? Or will it just remove the origin header and pass it on regardless?

Member

takluyver commented May 25, 2017

Does anyone know if that has security implications? IIRC, if the Origin header is unset, the server skips the usual cross-origin checks, assuming that it's a non-browser client connecting. Will the proxy server do cross-origin checks itself before passing the request on? Or will it just remove the origin header and pass it on regardless?

@rgbkrk

This comment has been minimized.

Show comment
Hide comment
@rgbkrk

rgbkrk May 25, 2017

Member

It has security implications for the websockets on the server side (where we put protections in place), however the browser will continue to do CORS checks.

Member

rgbkrk commented May 25, 2017

It has security implications for the websockets on the server side (where we put protections in place), however the browser will continue to do CORS checks.

@takluyver

This comment has been minimized.

Show comment
Hide comment
@takluyver

takluyver May 25, 2017

Member

Are the browser checks sufficient to protect users? Will it allow making requests, even if the user can't see the results?

Member

takluyver commented May 25, 2017

Are the browser checks sufficient to protect users? Will it allow making requests, even if the user can't see the results?

@rgbkrk

This comment has been minimized.

Show comment
Hide comment
@rgbkrk

rgbkrk May 25, 2017

Member

The browser will make the requests (though our XSRF stuff should prevent that at the server side) still, then check the headers from the response.

Member

rgbkrk commented May 25, 2017

The browser will make the requests (though our XSRF stuff should prevent that at the server side) still, then check the headers from the response.

@takluyver

This comment has been minimized.

Show comment
Hide comment
@takluyver

takluyver May 26, 2017

Member

So a malicious site could e.g. start lots of kernels by making POST requests, even though it couldn't then use those kernels?

Member

takluyver commented May 26, 2017

So a malicious site could e.g. start lots of kernels by making POST requests, even though it couldn't then use those kernels?

@takluyver

This comment has been minimized.

Show comment
Hide comment
@takluyver

takluyver May 26, 2017

Member

Or will the XSRF prevent that? I have trouble remembering how all these different bits work?

Member

takluyver commented May 26, 2017

Or will the XSRF prevent that? I have trouble remembering how all these different bits work?

@rgbkrk

This comment has been minimized.

Show comment
Hide comment
@rgbkrk

rgbkrk May 29, 2017

Member

The XSRF protection will prevent that, assuming they don't have the token.

Member

rgbkrk commented May 29, 2017

The XSRF protection will prevent that, assuming they don't have the token.

@takluyver

This comment has been minimized.

Show comment
Hide comment
@takluyver

takluyver May 30, 2017

Member

OK, thanks Kyle :-)

Member

takluyver commented May 30, 2017

OK, thanks Kyle :-)

@disrupted

This comment has been minimized.

Show comment
Hide comment
@disrupted

disrupted Jul 8, 2018

@jakocoo

If you are not on CloudFlare and still have the issue. Here's the fix:
https://stackoverflow.com/questions/22665809/how-to-configure-ipython-behind-nginx-in-a-subpath/23912400#23912400

proxy_set_header Origin ""; is the key.

I tried that but I am still having issues with Safari saying "Kernel starting, please wait..." (which I learned happens when it can't establish a secure websocket).

my config looks like this:

nginx jupyter.conf

location /jupyter/ {
        proxy_pass http://localhost:8888;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_set_header Origin "";
}

location ~* /jupyter/(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
        proxy_pass http://localhost:8888;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Origin "";
}

jupyter_notebook_config.py

c.NotebookApp.allow_origin = '*'
c.NotebookApp.base_url = '/jupyter'
c.NotebookApp.open_browser = False
c.NotebookApp.password = 'sha1:REDACTED'
c.NotebookApp.port = 8888
c.NotebookApp.trust_xheaders = True

disrupted commented Jul 8, 2018

@jakocoo

If you are not on CloudFlare and still have the issue. Here's the fix:
https://stackoverflow.com/questions/22665809/how-to-configure-ipython-behind-nginx-in-a-subpath/23912400#23912400

proxy_set_header Origin ""; is the key.

I tried that but I am still having issues with Safari saying "Kernel starting, please wait..." (which I learned happens when it can't establish a secure websocket).

my config looks like this:

nginx jupyter.conf

location /jupyter/ {
        proxy_pass http://localhost:8888;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_set_header Origin "";
}

location ~* /jupyter/(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
        proxy_pass http://localhost:8888;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Origin "";
}

jupyter_notebook_config.py

c.NotebookApp.allow_origin = '*'
c.NotebookApp.base_url = '/jupyter'
c.NotebookApp.open_browser = False
c.NotebookApp.password = 'sha1:REDACTED'
c.NotebookApp.port = 8888
c.NotebookApp.trust_xheaders = True
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment