Unable to establish websocket connection behind nginx reverse proxy #1311
Comments
@ROODAY Here's one approach that is documented in the jupyter/docker-stacks repo: https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes#running-behind-a-nginx-proxy cc/ @parente |
@willingc I tried following the conf but I still am not able to make a web socket connection. nginx-proxy allows mounting of a folder of conf files for specific domains that get included into the main conf, so I was able to add this to the server block for jupyter:
The only parts from the suggested conf file that I did not include were these:
Is SSL required to get websockets working? Edit: Here is my jupyter_notebook_config.py file
|
That looks right, but I'll ping @rgbkrk for any ideas about what might not be right about the nginx config for websocket forwarding. My guess is it's a tiny regex or priority bug preventing the Upgrade block from being applied correctly. |
Is there a way to run the Jupyter Notebook container in a debug mode to see more information on what happens when I load an IPython notebook? |
@ROODAY Jupyter Notebook can be run in debug mode with the command |
I tried debugging it within the Docker container since I'm sure the issue is related to everything being in a Docker container. However, when I tried to build a Docker image of Jupyter Notebook with a modified version of the Docker file that had the This is the command I ran to build the image (the docker file was at /home/rooday/dockerfile):
These were the last few lines of the build:
Edit: If necessary, here's the full console output when the build command was ran: http://pastebin.com/ybJPnwC0 |
What happens if you run without
|
That gives the same error of
|
A friend pointed out to me that the build error was likely due to the fact that the environment I was building it in did not have all the same folders and resources that the original was built in, and that I should instead use a dockerfile to extend the already made jupyter/notebook container. Using this Dockerfile:
and this build command: I was able to build a Docker container that ran Jupyter Notebook in debug. Upon connecting to the notebook, creating a new IPython 3 Notebook and editing it, I received the same connection as before. This is the last few lines of output from the Jupyter Notebook container's logs (full logs here: http://pastebin.com/vTqc1Xug):
The difference I notice now from when it wasn't in debug is there are now lines saying
Is there a setting that is preventing the connection from being upgraded to a websocket connection? |
@willingc
Is this something that should be ignored, or could it be a reason I'm having trouble? |
@ROODAY In the jupyter/notebook's |
@willingc Could this be the issue? The dockerfile for Jupyter/Notebook on DockerHub does not match the docker file from the Jupyter/Notebook repository. https://hub.docker.com/r/jupyter/notebook/~/dockerfile/ Edit: I run a new jupyter/notebook container with the stable tag with this command:
And from doing that it seemed to install the extensions as I saw this in the chrome console:
However, I still get 304 errors when trying to run code as I have to force Upgrade "websocket" to make a connection, therefore interfering with normal requests for assets like Javascript and CSS. |
@willingc Nevermind, I found the issue. As it turns out, it was not related to either nginx-proxy or Jupyter, and rather Cloudflare. At the moment, Websocket connections through Cloudflare is in beta, and by default is disabled. Cloudflare recommends as a workaround to simply grey-cloud A records that make use of websocket connections, so in my case that was notes.rooday.com https://support.cloudflare.com/hc/en-us/articles/200169466-Can-I-use-CloudFlare-with-WebSockets- For anyone in the future attempting to use Cloudflare, Jupyter/notebook, and nginx-proxy, refer to this: Update:Cloudflare now allows websockets for everyone, read more here. |
@ROODAY I'm glad you got it resolved. Thanks for passing along the content to help others too. |
+1 I had similar issue and the problem was solved by disabling (grey-cloud) http-proxy in cloudflare. |
If you are not on CloudFlare and still have the issue. Here's the fix:
|
Does anyone know if that has security implications? IIRC, if the |
It has security implications for the websockets on the server side (where we put protections in place), however the browser will continue to do CORS checks. |
Are the browser checks sufficient to protect users? Will it allow making requests, even if the user can't see the results? |
The browser will make the requests (though our XSRF stuff should prevent that at the server side) still, then check the headers from the response. |
So a malicious site could e.g. start lots of kernels by making POST requests, even though it couldn't then use those kernels? |
Or will the XSRF prevent that? I have trouble remembering how all these different bits work? |
The XSRF protection will prevent that, assuming they don't have the token. |
OK, thanks Kyle :-) |
I tried that but I am still having issues with Safari saying "Kernel starting, please wait..." (which I learned happens when it can't establish a secure websocket). my config looks like this: nginx
|
I have a Digital Ocean server running Ubuntu 14.04, with Jupyter Notebook (https://hub.docker.com/r/jupyter/notebook/) running inside a Docker container. I also have the nginx-proxy (https://github.com/jwilder/nginx-proxy) Docker container to serve as my reverse proxy.
I have it configured such that notes.rooday.com would be pointed to the Jupyter container, and that works as intended. I'm able to view the application, authenticate, make and edit files, etc. However, attempting to run an ipython notebook file gives a Connection Failed error:
Looking inside the Chrome console, I see this error:
The kernel attempts to reconnect, but to no avail. Checking the docker logs for the nginx proxy yields the following relevant lines:
and the logs for the Jupyter container show
Based on this, and asking in the nginx-proxy issues page, it would seem that Jupyter is the cause of the problem. The nginx proxy is configured to allow websocket connections, but they're still not going through. Has anyone else attempted to run Jupyter behind a reverse proxy, or know enough to help me find a workaround?
The text was updated successfully, but these errors were encountered: