Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upUnable to establish websocket connection behind nginx reverse proxy #1311
Comments
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
willingc
Apr 6, 2016
Member
@ROODAY Here's one approach that is documented in the jupyter/docker-stacks repo: https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes#running-behind-a-nginx-proxy
cc/ @parente
|
@ROODAY Here's one approach that is documented in the jupyter/docker-stacks repo: https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes#running-behind-a-nginx-proxy cc/ @parente |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ROODAY
Apr 7, 2016
@willingc I tried following the conf but I still am not able to make a web socket connection. nginx-proxy allows mounting of a folder of conf files for specific domains that get included into the main conf, so I was able to add this to the server block for jupyter:
client_max_body_size 50M;
error_log /var/log/nginx/error.log;
location / {
proxy_pass http://notes.rooday.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400;
}
location ~* /(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
proxy_pass http://notes.rooday.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
The only parts from the suggested conf file that I did not include were these:
ssl on;
ssl_certificate /data/cert.crt;
ssl_certificate_key /data/key.key;
ssl_ciphers "AES128+EECDH:AES128+EDH";
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
add_header X-Content-Type-Options nosniff;
ssl_stapling on; # Requires nginx >= 1.3.7
ssl_stapling_verify on; # Requires nginx => 1.3.7
resolver_timeout 5s;
Is SSL required to get websockets working?
Edit:
Here is my jupyter_notebook_config.py file
c.NotebookApp.ip = '172.17.0.4'
c.NotebookApp.password = u'whoathatsnotsupposedtobehere'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 8888
c.NotebookApp.trust_xheaders = True
c.NotebookApp.allow_origin = '*'
ROODAY
commented
Apr 7, 2016
|
@willingc I tried following the conf but I still am not able to make a web socket connection. nginx-proxy allows mounting of a folder of conf files for specific domains that get included into the main conf, so I was able to add this to the server block for jupyter: The only parts from the suggested conf file that I did not include were these: Is SSL required to get websockets working? Edit: Here is my jupyter_notebook_config.py file |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
minrk
Apr 7, 2016
Member
That looks right, but I'll ping @rgbkrk for any ideas about what might not be right about the nginx config for websocket forwarding. My guess is it's a tiny regex or priority bug preventing the Upgrade block from being applied correctly.
|
That looks right, but I'll ping @rgbkrk for any ideas about what might not be right about the nginx config for websocket forwarding. My guess is it's a tiny regex or priority bug preventing the Upgrade block from being applied correctly. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ROODAY
Apr 8, 2016
Is there a way to run the Jupyter Notebook container in a debug mode to see more information on what happens when I load an IPython notebook?
ROODAY
commented
Apr 8, 2016
|
Is there a way to run the Jupyter Notebook container in a debug mode to see more information on what happens when I load an IPython notebook? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
willingc
Apr 8, 2016
Member
@ROODAY Jupyter Notebook can be run in debug mode with the command jupyter notebook --debug
|
@ROODAY Jupyter Notebook can be run in debug mode with the command |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ROODAY
Apr 8, 2016
I tried debugging it within the Docker container since I'm sure the issue is related to everything being in a Docker container. However, when I tried to build a Docker image of Jupyter Notebook with a modified version of the Docker file that had the --debug flag (here: http://pastebin.com/AMtbq7bX), I received an error about setup.py not being found.
This is the command I ran to build the image (the docker file was at /home/rooday/dockerfile):
sudo docker build -t jupyter-debug:test --rm=true /home/rooday/
These were the last few lines of the build:
Directory '/usr/src/jupyter-notebook' is not installable. File 'setup.py' not found.
The command '/bin/sh -c BUILD_DEPS="nodejs-legacy npm" && apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -yq $BUILD_DEPS && pip3 install --no-cache-dir /usr/src/jupyter-notebook && pip2 install --no-cache-dir widgetsnbextension && pip3 install --no-cache-dir widgetsnbextension && npm cache clean && apt-get clean && rm -rf /root/.npm && rm -rf /root/.cache && rm -rf /root/.config && rm -rf /root/.local && rm -rf /root/tmp && rm -rf /var/lib/apt/lists/* && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false -o APT::AutoRemove::SuggestsImportant=false $BUILD_DEPS' returned a non-zero code: 1
Edit: If necessary, here's the full console output when the build command was ran: http://pastebin.com/ybJPnwC0
ROODAY
commented
Apr 8, 2016
|
I tried debugging it within the Docker container since I'm sure the issue is related to everything being in a Docker container. However, when I tried to build a Docker image of Jupyter Notebook with a modified version of the Docker file that had the This is the command I ran to build the image (the docker file was at /home/rooday/dockerfile): These were the last few lines of the build: Edit: If necessary, here's the full console output when the build command was ran: http://pastebin.com/ybJPnwC0 |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
willingc
Apr 8, 2016
Member
What happens if you run without --rm=true
sudo docker build -t jupyter-debug:test /home/rooday/
|
What happens if you run without
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ROODAY
Apr 8, 2016
That gives the same error of
Directory '/usr/src/jupyter-notebook' is not installable. File 'setup.py' not found.
The command '/bin/sh -c BUILD_DEPS="nodejs-legacy npm" && apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -yq $BUILD_DEPS && pip3 install --no-cache-dir /usr/src/jupyter-notebook && pip2 install --no-cache-dir widgetsnbextension && pip3 install --no-cache-dir widgetsnbextension && npm cache clean && apt-get clean && rm -rf /root/.npm && rm -rf /root/.cache && rm -rf /root/.config && rm -rf /root/.local && rm -rf /root/tmp && rm -rf /var/lib/apt/lists/* && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false -o APT::AutoRemove::SuggestsImportant=false $BUILD_DEPS' returned a non-zero code: 1
ROODAY
commented
Apr 8, 2016
|
That gives the same error of |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ROODAY
Apr 8, 2016
A friend pointed out to me that the build error was likely due to the fact that the environment I was building it in did not have all the same folders and resources that the original was built in, and that I should instead use a dockerfile to extend the already made jupyter/notebook container.
Using this Dockerfile:
FROM jupyter/notebook
CMD ["jupyter", "notebook", "--debug"]
and this build command:
sudo docker build -t jupyter/notebook:debug /home/rooday/
I was able to build a Docker container that ran Jupyter Notebook in debug. Upon connecting to the notebook, creating a new IPython 3 Notebook and editing it, I received the same connection as before. This is the last few lines of output from the Jupyter Notebook container's logs (full logs here: http://pastebin.com/vTqc1Xug):
[W 22:31:20.357 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 1269.18ms referer=None
[D 22:31:20.358 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:21.388 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:21.403 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 15.53ms referer=None
[D 22:31:21.403 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:23.437 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:23.450 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 12.59ms referer=None
[D 22:31:23.450 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:25.284 NotebookApp] 200 GET /custom/custom.css (172.17.0.2) 6.46ms
[D 22:31:25.433 NotebookApp] 200 GET /static/notebook/js/main.min.js.map (172.17.0.2) 71.94ms
[D 22:31:25.451 NotebookApp] 304 GET /static/style/style.min.css.map (172.17.0.2) 8.74ms
[D 22:31:27.480 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:27.490 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 10.67ms referer=None
[D 22:31:27.491 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:35.538 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:35.549 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 12.31ms referer=None
[D 22:31:35.550 NotebookApp] Can "Upgrade" only to "WebSocket".
[D 22:31:51.590 NotebookApp] Initializing websocket connection /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels
[W 22:31:51.616 NotebookApp] 400 GET /api/kernels/93b69b1c-a4ab-4a85-a49b-f3a0abf592db/channels?session_id=0F0EDE29F9364D918500E3FC55EF9A3A (172.17.0.2) 30.62ms referer=None
[D 22:31:51.617 NotebookApp] Can "Upgrade" only to "WebSocket".
The difference I notice now from when it wasn't in debug is there are now lines saying
Can "Upgrade" only to "WebSocket".
Is there a setting that is preventing the connection from being upgraded to a websocket connection?
ROODAY
commented
Apr 8, 2016
|
A friend pointed out to me that the build error was likely due to the fact that the environment I was building it in did not have all the same folders and resources that the original was built in, and that I should instead use a dockerfile to extend the already made jupyter/notebook container. Using this Dockerfile: and this build command: I was able to build a Docker container that ran Jupyter Notebook in debug. Upon connecting to the notebook, creating a new IPython 3 Notebook and editing it, I received the same connection as before. This is the last few lines of output from the Jupyter Notebook container's logs (full logs here: http://pastebin.com/vTqc1Xug): The difference I notice now from when it wasn't in debug is there are now lines saying Is there a setting that is preventing the connection from being upgraded to a websocket connection? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ROODAY
Apr 10, 2016
@willingc
I've been talking to a person from the nginx-proxy repo to help figure out the issues on the nginx side, but I made some progress in that one config allowed me to make a kernel websocket connection. However, when I did that, I saw these errors in the chrome console:
actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later...
Failed to load resource: the server responded with a status of 404 (Not Found)
http://notes.rooday.com/nbextensions/widgets/extension.js?v=20160408223056
Failed to load resource: the server responded with a status of 404 (Not Found) http://notes.rooday.com/nbextensions/widgets/notebook/js/extension.js?v=20160408223056
Widgets are not available. Please install widgetsnbextension or ipywidgets 4.0
Is this something that should be ignored, or could it be a reason I'm having trouble?
ROODAY
commented
Apr 10, 2016
|
@willingc Is this something that should be ignored, or could it be a reason I'm having trouble? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
willingc
Apr 10, 2016
Member
@ROODAY In the jupyter/notebook's Dockerfile, widgetsnbextension is installed for Python 2 and 3. I would recommend installing it.
|
@ROODAY In the jupyter/notebook's |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ROODAY
Apr 10, 2016
@willingc Could this be the issue? The dockerfile for Jupyter/Notebook on DockerHub does not match the docker file from the Jupyter/Notebook repository.
https://hub.docker.com/r/jupyter/notebook/~/dockerfile/
Edit:
I run a new jupyter/notebook container with the stable tag with this command:
docker run --name jupyter -d -p 8888:8888 -v "/home/jupyter/notebooks:/notebooks" jupyter/notebook:stable
And from doing that it seemed to install the extensions as I saw this in the chrome console:
Session: kernel_created (8631fdc3-87a9-405f-bdf5-cc486238fdcc)
Starting WebSockets: ws://notes.rooday.com/api/kernels/fc593fbd-f32e-4ecb-a8fb-1a665ca916ed
Loading extension: widgets/notebook/js/extension
loaded widgets
Kernel: kernel_connected (fc593fbd-f32e-4ecb-a8fb-1a665ca916ed)
However, I still get 304 errors when trying to run code as I have to force Upgrade "websocket" to make a connection, therefore interfering with normal requests for assets like Javascript and CSS.
ROODAY
commented
Apr 10, 2016
|
@willingc Could this be the issue? The dockerfile for Jupyter/Notebook on DockerHub does not match the docker file from the Jupyter/Notebook repository. https://hub.docker.com/r/jupyter/notebook/~/dockerfile/ Edit: I run a new jupyter/notebook container with the stable tag with this command: And from doing that it seemed to install the extensions as I saw this in the chrome console: However, I still get 304 errors when trying to run code as I have to force Upgrade "websocket" to make a connection, therefore interfering with normal requests for assets like Javascript and CSS. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ROODAY
Apr 10, 2016
@willingc Nevermind, I found the issue. As it turns out, it was not related to either nginx-proxy or Jupyter, and rather Cloudflare. At the moment, Websocket connections through Cloudflare is in beta, and by default is disabled. Cloudflare recommends as a workaround to simply grey-cloud A records that make use of websocket connections, so in my case that was notes.rooday.com
https://support.cloudflare.com/hc/en-us/articles/200169466-Can-I-use-CloudFlare-with-WebSockets-
For anyone in the future attempting to use Cloudflare, Jupyter/notebook, and nginx-proxy, refer to this:
jwilder/nginx-proxy#412
Update:
Cloudflare now allows websockets for everyone, read more here.
ROODAY
commented
Apr 10, 2016
•
|
@willingc Nevermind, I found the issue. As it turns out, it was not related to either nginx-proxy or Jupyter, and rather Cloudflare. At the moment, Websocket connections through Cloudflare is in beta, and by default is disabled. Cloudflare recommends as a workaround to simply grey-cloud A records that make use of websocket connections, so in my case that was notes.rooday.com https://support.cloudflare.com/hc/en-us/articles/200169466-Can-I-use-CloudFlare-with-WebSockets- For anyone in the future attempting to use Cloudflare, Jupyter/notebook, and nginx-proxy, refer to this: Update:Cloudflare now allows websockets for everyone, read more here. |
ROODAY
closed this
Apr 10, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
willingc
Apr 10, 2016
Member
@ROODAY I'm glad you got it resolved. Thanks for passing along the content to help others too.
|
@ROODAY I'm glad you got it resolved. Thanks for passing along the content to help others too. |
minrk
added this to the
no action milestone
May 30, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
lucaswxp
Oct 11, 2016
+1 I had similar issue and the problem was solved by disabling (grey-cloud) http-proxy in cloudflare.
lucaswxp
commented
Oct 11, 2016
|
+1 I had similar issue and the problem was solved by disabling (grey-cloud) http-proxy in cloudflare. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jakocoo
May 25, 2017
If you are not on CloudFlare and still have the issue. Here's the fix:
https://stackoverflow.com/questions/22665809/how-to-configure-ipython-behind-nginx-in-a-subpath/23912400#23912400
proxy_set_header Origin ""; is the key.
jakocoo
commented
May 25, 2017
|
If you are not on CloudFlare and still have the issue. Here's the fix:
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
takluyver
May 25, 2017
Member
Does anyone know if that has security implications? IIRC, if the Origin header is unset, the server skips the usual cross-origin checks, assuming that it's a non-browser client connecting. Will the proxy server do cross-origin checks itself before passing the request on? Or will it just remove the origin header and pass it on regardless?
|
Does anyone know if that has security implications? IIRC, if the |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rgbkrk
May 25, 2017
Member
It has security implications for the websockets on the server side (where we put protections in place), however the browser will continue to do CORS checks.
|
It has security implications for the websockets on the server side (where we put protections in place), however the browser will continue to do CORS checks. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
takluyver
May 25, 2017
Member
Are the browser checks sufficient to protect users? Will it allow making requests, even if the user can't see the results?
|
Are the browser checks sufficient to protect users? Will it allow making requests, even if the user can't see the results? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rgbkrk
May 25, 2017
Member
The browser will make the requests (though our XSRF stuff should prevent that at the server side) still, then check the headers from the response.
|
The browser will make the requests (though our XSRF stuff should prevent that at the server side) still, then check the headers from the response. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
takluyver
May 26, 2017
Member
So a malicious site could e.g. start lots of kernels by making POST requests, even though it couldn't then use those kernels?
|
So a malicious site could e.g. start lots of kernels by making POST requests, even though it couldn't then use those kernels? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
takluyver
May 26, 2017
Member
Or will the XSRF prevent that? I have trouble remembering how all these different bits work?
|
Or will the XSRF prevent that? I have trouble remembering how all these different bits work? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rgbkrk
May 29, 2017
Member
The XSRF protection will prevent that, assuming they don't have the token.
|
The XSRF protection will prevent that, assuming they don't have the token. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
OK, thanks Kyle :-) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
disrupted
Jul 8, 2018
If you are not on CloudFlare and still have the issue. Here's the fix:
https://stackoverflow.com/questions/22665809/how-to-configure-ipython-behind-nginx-in-a-subpath/23912400#23912400proxy_set_header Origin ""; is the key.
I tried that but I am still having issues with Safari saying "Kernel starting, please wait..." (which I learned happens when it can't establish a secure websocket).
my config looks like this:
nginx jupyter.conf
location /jupyter/ {
proxy_pass http://localhost:8888;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Origin "";
}
location ~* /jupyter/(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
proxy_pass http://localhost:8888;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Origin "";
}
jupyter_notebook_config.py
c.NotebookApp.allow_origin = '*'
c.NotebookApp.base_url = '/jupyter'
c.NotebookApp.open_browser = False
c.NotebookApp.password = 'sha1:REDACTED'
c.NotebookApp.port = 8888
c.NotebookApp.trust_xheaders = True
disrupted
commented
Jul 8, 2018
•
I tried that but I am still having issues with Safari saying "Kernel starting, please wait..." (which I learned happens when it can't establish a secure websocket). my config looks like this: nginx
|
ROODAY commentedApr 6, 2016
I have a Digital Ocean server running Ubuntu 14.04, with Jupyter Notebook (https://hub.docker.com/r/jupyter/notebook/) running inside a Docker container. I also have the nginx-proxy (https://github.com/jwilder/nginx-proxy) Docker container to serve as my reverse proxy.
I have it configured such that notes.rooday.com would be pointed to the Jupyter container, and that works as intended. I'm able to view the application, authenticate, make and edit files, etc. However, attempting to run an ipython notebook file gives a Connection Failed error:
Looking inside the Chrome console, I see this error:
The kernel attempts to reconnect, but to no avail. Checking the docker logs for the nginx proxy yields the following relevant lines:
and the logs for the Jupyter container show
Based on this, and asking in the nginx-proxy issues page, it would seem that Jupyter is the cause of the problem. The nginx proxy is configured to allow websocket connections, but they're still not going through. Has anyone else attempted to run Jupyter behind a reverse proxy, or know enough to help me find a workaround?