Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploying behind a reverse proxy #219

Closed
danielballan opened this issue Apr 7, 2015 · 46 comments

Comments

@danielballan
Copy link
Contributor

commented Apr 7, 2015

Has anyone worked on deploying jupyterhub behind a Reverse Proxy?

I am trying to deploy it on a system that communicates with the open network through an Apache reverse proxy. Authentication works fine, and kernels are created when users open notebooks, but the notebooks cannot connect to their kernels. It looks like requests are coming in, but communication out through the Apache server is not working correctly.

@minrk

This comment has been minimized.

Copy link
Member

commented Apr 7, 2015

You need to make sure that websocket connections work, which I think is not Apache's default behavior. We've deployed it behind nginx a few times. Here is an example nginx config.

@danielballan

This comment has been minimized.

Copy link
Contributor Author

commented Apr 10, 2015

Thanks. I'm working with our busy network admins on this, and I'll revisit the issue with anything we that might be helpful to others.

@danielballan

This comment has been minimized.

Copy link
Contributor Author

commented Apr 20, 2015

If anyone else is looking to do deploy behind an Apache reverse proxy, here's our intrepid sysadmin's summary of she got this working:

Older versions of Apache such as 2.2.x do not provide WebSocket support, therefore a newer version of Apache (i,e 2.4) included in the upcoming Debian Jessie release, supplies proxy_wstunnel for WebSocket applications. For Jupyterhub, for any URL spaces /(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/?, forward to ws(s)://servername:port_number, all other standard spaces, forward to http(s)://servername:port_number, that will do the trick!

@franga2000

This comment has been minimized.

Copy link
Contributor

commented Sep 15, 2015

I know I'm a bit late, but if anyone needs it, here's the config that works for me (basically what @danielballan quoted):

ProxyPass / http://localhost:8000/
ProxyPassReverse / http://localhost:8000/

Header edit Origin {external address} localhost:8000
RequestHeader edit Origin {external address} localhost:8000

Header edit Referer {external address} localhost:8000
RequestHeader edit Referer {external address} localhost:8000
<Location ~ "/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/?">
    ProxyPass ws://localhost:8000
    ProxyPassReverse ws://localhost:8000
</Location>
@iocampomx

This comment has been minimized.

Copy link

commented Nov 11, 2015

Hello franga2000, what means {external address}? What is the value that I need to put in this place?

@franga2000

This comment has been minimized.

Copy link
Contributor

commented Nov 12, 2015

@Nafiux it's the address that you want to use to access jupyter from outside (like py.example.com)

@chandragaajula

This comment has been minimized.

Copy link

commented Jan 5, 2016

@Nafiux I have the following entry in "/etc/apache2/sites-enabled" on my Ubuntu 15.10

<VirtualHost *:80>
        ServerName "jupyter.xxxxxxxxxxxx.com"
        ProxyPass / http://192.168.254.23:8888/
        ProxyPassReverse / http://192.168.254.23:8888/
        Header edit Origin "jupyter.xxxxxxxxxxxx.com" 192.168.254.23:8888
        RequestHeader edit Origin "jupyter.xxxxxxxxxxxx.com" 192.168.254.23:8888

        Header edit Referer "jupyter.xxxxxxxxxxxx.com" 192.168.254.23:8888
        RequestHeader edit Referer "jupyter.xxxxxxxxxxxx.com" 192.168.254.23:8888
        <Location ~ "/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/?">
                ProxyPass ws://192.168.254.23:8888
                ProxyPassReverse ws://192.168.254.23:8888
        </Location>
</VirtualHost>

After making the above entry I restarted apache: "sudo service apache2 restart"

When I attempt to run my script the Kernel is unable to connect.
jupyterreverseproxy

Console Logs:
jupyterlogs

The same works when I specify the IP Address though:
jupyterwithipaddressandport

How do I get this to work with the domain name?

@franga2000

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2016

@chandragaajula are there any errors in the Apache logs? Also, have you enabled mod_proxy_wstunnel?

@chandragaajula

This comment has been minimized.

Copy link

commented Jan 7, 2016

@franga2000 - Yes, I have enabled mod_proxy_wstunnel:

 chandra@TheMilkyWay:/etc/apache2$ sudo a2enmod
 Your choices are: access_compat actions alias allowmethods asis auth_basic auth_digest auth_form authn_anon authn_core authn_dbd authn_dbm authn_file authn_socache authnz_fcgi authnz_ldap authz_core authz_dbd authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex buffer cache cache_disk cache_socache cgi cgid charset_lite data dav dav_fs dav_lock dbd deflate dialup dir dump_io echo env expires ext_filter file_cache filter headers heartbeat heartmonitor ident include info lbmethod_bybusyness lbmethod_byrequests lbmethod_bytraffic lbmethod_heartbeat ldap log_debug log_forensic lua macro mime mime_magic mpm_event mpm_prefork mpm_worker negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_express proxy_fcgi proxy_fdpass proxy_ftp proxy_html proxy_http proxy_scgi proxy_wstunnel ratelimit reflector remoteip reqtimeout request rewrite sed session session_cookie session_crypto session_dbd setenvif slotmem_plain slotmem_shm socache_dbm socache_memcache socache_shmcb speling ssl status substitute suexec unique_id userdir usertrack vhost_alias xml2enc
 Which module(s) do you want to enable (wildcards ok)?
 proxy_wstunnel
 Considering dependency proxy for proxy_wstunnel:
 Module proxy already enabled
 Module proxy_wstunnel already enabled
 chandra@TheMilkyWay:/etc/apache2$

I don't see any specific errors in logs, will review them again.

@mbmilligan

This comment has been minimized.

Copy link
Member

commented Jan 8, 2016

The config for websocket proxying quoted by @franga2000 didn't work for me either. What DID work was instead being explicit about the URL matching:

<LocationMatch "/mypath/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)(.*)">
    ProxyPassMatch ws://localhost:8999/mypath/$1/$2$3
    ProxyPassReverse ws://localhost:8999 # this may be superfluous
</LocationMatch>

For what it's worth, this is using Apache 2.2 on CentOS, with the wstunnel backport patches described here: https://gist.github.com/vitkin/6661683

@chandragaajula

This comment has been minimized.

Copy link

commented Jan 8, 2016

@mbmilligan - Thanks for the link you provided which led me to my solution. In my case, it turns out to be quite simple:

<VirtualHost *:80>
        ServerName "jupyter.xxxxxxxxxxx.com"
        ProxyPreserveHost On
        ProxyRequests off

        ProxyPass /api/kernels/ ws://192.168.254.23:8888/api/kernels/
        ProxyPassReverse /api/kernels/ http://192.168.254.23:8888/api/kernels/

        ProxyPass / http://192.168.254.23:8888/
        ProxyPassReverse / http://192.168.254.23:8888/
</VirtualHost>

Important thing to note: The entry for WebSockets should be in the beginning.

@mcapuccini

This comment has been minimized.

Copy link

commented Feb 12, 2016

Same problem on Traefik. Anyone managed to configure it using it? A generic explanation of what causes the problem would be really appreciated, so I can try to configure the proxy myself.

@mcapuccini

This comment has been minimized.

Copy link

commented Feb 12, 2016

Just found the solution. I needed to add the following to the JSON that I send to Marathon:

"labels": {"traefik.frontend.entryPoints":"http,https,ws"}

I guess this will be useful for many people, as Traefik is getting popular.

@lbiemans

This comment has been minimized.

Copy link

commented Jul 14, 2016

Had the same issue but thanks to @franga2000 this is solved. I had to enable proxy, proxy_http, headers and proxy_wstunnel.

@TheGodEmperor

This comment has been minimized.

Copy link

commented Aug 25, 2016

I notice this is old, but I've been trying to figure out how to have a server proxy to the server running JupyterHub.

Basically, one server is test-session that runs Jupyterhub and the other is the test-dev which will need to run so that if one did test-dev/notebook/ it'd link to the test-session:8888/notebooks/ link.

I've got a proxy for test-dev to test-session for other things that works (like Python scripts), but for some reason the proxy for the Jupyterhub isn't working. Wouldn't the proxy be, on the test-dev server, basically?

ProxyPass /notebooks/ http://lle-test-session:8888/
ProxyPassReverse /notebooks/ http://lle-test-session:8888/

Am I missing something? I'm not much of a server administrative guy so I'm still in the process of learning.

@ryanlovett

This comment has been minimized.

Copy link
Contributor

commented Aug 26, 2016

Just adding another works for me example. Apache 2.4.7 in Ubuntu 14.04 suffers from a websocket bug that prevents notebook sessions from working properly in this context.

ProxyPreserveHost On
ProxyRequests off

ProxyPass /api/kernels/ ws://127.0.0.1:8888/api/kernels/
ProxyPassReverse /api/kernels/ https://127.0.0.1:8888/api/kernels/

ProxyPass / https://127.0.0.1:8888/
ProxyPassReverse / https://127.0.0.1:8888/

<Location ~ "/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/?">
    ProxyPass ws://localhost:8888
    ProxyPassReverse ws://localhost:8888
</Location>
@mbmilligan

This comment has been minimized.

Copy link
Member

commented Aug 26, 2016

@TheGodEmperor You should crib from one of the working examples in this thread. You definitely do need to proxy the main paths and the websockets separately. A couple more notes:

  1. you spell the url "/notebook" in one place and "/notebooks" in another, check for misspellings in your config;
  2. the Hub may need to be configured to listen to an external interface rather than localhost;
  3. recent versions will complain if started without SSL, so if you are terminating SSL at your proxy server (which is the common thing to do here) you may need to add the --no-ssl option
@lbiemans

This comment has been minimized.

Copy link

commented Aug 29, 2016

What works for me is:

    ProxyPass / http://localhost:8000/
    ProxyPassReverse / http://localhost:8000/

    Header edit Origin HOSTNAME.TLD localhost:8000
    RequestHeader edit Origin HOSTNAME.TLD localhost:8000

    Header edit Referer HOSTNAME.TLD localhost:8000
    RequestHeader edit Referer HOSTNAME.TLD localhost:8000
    <Location ~ "/(user/[^/]+)/(api/kernels/[^/]+/channels|terminals/websocket)/?">
            ProxyPass ws://localhost:8000
            ProxyPassReverse ws://localhost:8000
    </Location>

Don't forget to enable proxy, proxy_http, headers and proxy_wstunnel.

@lordloh

This comment has been minimized.

Copy link

commented Oct 4, 2016

I am getting a "Blocking Cross Origin API request".

[W 2016-10-04 13:53:03.855 bharath handlers:288] Blocking Cross Origin API request.  Origin: https://192.168.144.3:8000, Host: jptr.mydomain.com
[W 2016-10-04 13:53:03.856 bharath log:47] 404 POST /user/bharath/api/contents (129.118.26.12) 1.39ms referer=https://192.168.144.3:8000/user/bharath/tree

what do I need to change in my jupyterhub config file?

@lordloh

This comment has been minimized.

Copy link

commented Oct 4, 2016

What worked for me was -

<VirtualHost *:443>
        ServerName jptr.example.com
        SSLEngine On
        SSLProxyEngine On
        ProxyPreserveHost On
        DocumentRoot /var/www/html

        <Location ~ "/(user/[^/]+)/(api/kernels/[^/]+/channels|terminals/websocket)/?">
                ProxyPass ws://192.168.144.3:8000
                ProxyPassReverse ws://192.168.144.3:8000
        </Location>

        ProxyPass / http://192.168.144.3:8000/
        ProxyPassReverse / http://192.168.144.3:8000/

        SSLCertificateFile /etc/letsencrypt/live/jptr.example.com/cert.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/jptr.example.com/privkey.pem
        SSLCertificateChainFile /etc/letsencrypt/live/jptr.example.com/chain.pem
        Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

The websocket part had to come before the the http part.

@slacmshankar

This comment has been minimized.

Copy link

commented Oct 13, 2016

One more Apache configuration as an example; I'm using an existing httpd-2.4.6-40 instance and placing this within /jupyterhub, i.e c.JupyterHub.base_url = '/jupyterhub' The order of the Location and LocationMatch seemed to be crucial; otherwise, I'd get a various websocket related failures (HTTP 400's) with sessions/kernels etc. Note this is the opposite of what others had experienced. So if you see websocket failures, try changing the order of your Location directives.

  ProxyRequests off
  ProxyPreserveHost On

  <Location /jupyterhub>
    ProxyPass http://jphub:8000/jupyterhub
    ProxyPassReverse http://jphub:8000/jupyterhub
  </Location>

  <LocationMatch "/jupyterhub/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)(.*)">
    ProxyPassMatch ws://jphub:8000/jupyterhub/$1/$2$3
    ProxyPassReverse ws://jphub:8000/jupyterhub/$1/$2$3
  </LocationMatch>
@ianabc

This comment has been minimized.

Copy link

commented Oct 14, 2016

@slacmshankar that almost works for me. I get the initial connection but whenever I try to restart a running kernel it fails and I see connection failed messages in the console.

kernel.js:453 Starting WebSockets: wss://jphub/jupyter/user/ianatr/api/kernels/3538a47c-33e1-4d14-bbf9-737798456261
VM681:35 WebSocket connection to 'wss://jphub/jupyter/user/ianatr/api/kernels/3538a47c-33…d14-bbf9-737798456261/channels?session_id=1BB7A309853142AFB8843CF626F67EE0' failed: One or more reserved bits are on: reserved1 = 0, reserved2 = 1, reserved3 = 1WrappedWebSocket @ VM681:35WrappedWebSocket @ VM713:35Kernel.start_channels @ kernel.js:455Kernel.reconnect @ kernel.js:349i @ jquery.min.js:4
kernel.js:100 Kernel: kernel_disconnected (3538a47c-33e1-4d14-bbf9-737798456261)
kernel.js:538 WebSocket connection failed:  wss://jphub/jupyter/user/ianatr/api/kernels/3538a47c-33e1-4d14-bbf9-737798456261 true
kernel.js:556 Connection lost, reconnecting in 2 seconds.
kernel.js:100 Kernel: kernel_connected (3538a47c-33e1-4d14-bbf9-737798456261)
manager.js:112 Widget frontend version does not match the backend.
kernel.js:100 Kernel: kernel_ready (3538a47c-33e1-4d14-bbf9-737798456261)
kernel.js:100 Kernel: kernel_reconnecting (3538a47c-33e1-4d14-bbf9-737798456261)
kernel.js:453 Starting WebSockets: wss://jphub/jupyter/user/ianatr/api/kernels/3538a47c-33e1-4d14-bbf9-7377984

...

Could you try restarting a running kernel on your instance and see if you get the same behaviour?

Edit: This is only happening to me in Chrome, Firefox seems to handle the restart just fine!

@slacmshankar

This comment has been minimized.

Copy link

commented Oct 14, 2016

Apologies; noob here. How do I "restart a running kernel on your instance"?

@ianabc

This comment has been minimized.

Copy link

commented Oct 14, 2016

Hi @slacmshankar me too! I just mean select Kernel->Restart from the menu system (screenshot attached).
screen shot 2016-10-14 at 4 03 59 pm
When I do that from Chrome it doesn't reconnect to the running kernel and I get a bunch of error messages in the console.

I think I've got the problem narrowed down to something in the way that the websockets are handled in kernel.js (jupyter/notebook) rather than jupyterhub but it would be really helpful to know if you are seeing the same behaviour. I don't see any problem with Firefox.

@slacmshankar

This comment has been minimized.

Copy link

commented Oct 14, 2016

Aah; confirmed! On Google chrome, I get the same behavior
WebSocket connection to '...' failed: One or more reserved bits are on: reserved1 = 0, reserved2 = 1, reserved3 = 1

Works just fine in Firefox; no problems there.

@remidebette

This comment has been minimized.

Copy link

commented Dec 9, 2016

Dear everyone,

Thank you for the usefull discussion.
The .conf file of @slacmshankar made the jupyterhub work for me.

But after I upgraded to jupyterhub 0.7 then suddently apache does not redirect.
"Not Found

The requested URL /jupyterhub was not found on this server."

Would you have any idea of how to update the .conf file?

Thank you

@remidebette

This comment has been minimized.

Copy link

commented Dec 12, 2016

Dear everyone,

To be more specific:
With the following apache conf (all other things unchanged after the upgrade to 0.7)

ProxyPreserveHost On
ProxyRequests off
#SSLProxyEngine On

<Location /jupyter>
        ProxyPass http://localhost:8000/jupyter
        ProxyPassReverse http://localhost:8000/jupyter
</Location>

<LocationMatch "/jupyter/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)(.*)">
        ProxyPassMatch ws://localhost:8000/jupyter/$1/$2$3
        ProxyPassReverse ws://localhost:8000/jupyter/$1/$2$3
</LocationMatch>

I get the following when calling http://localhost/jupyter

https://debette.noip.me/jupyter/hub/jupyter 404 not found

and no error if I call http://localhost/jupyter/ (with the slash)

On the other hand if I add slashes:

ProxyPreserveHost On
ProxyRequests off
#SSLProxyEngine On

<Location /jupyter/>
        ProxyPass http://localhost:8000/jupyter/
        ProxyPassReverse http://localhost:8000/jupyter/
</Location>


<LocationMatch "/jupyter/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)(.*)">
        ProxyPassMatch ws://localhost:8000/jupyter/$1/$2$3
        ProxyPassReverse ws://localhost:8000/jupyter/$1/$2$3
</LocationMatch>

Then http://localhost/jupyter
gives me:

Not Found

The requested URL /jupyter was not found on this server.

With the slash it also works fine.

All my other proxied servers work fine with the same kind of configuration, with or without slash, and once again this configuration worked just fine before the upgrade to Juptyerhub 0.7
Should I raise a bug?

Remi

@minrk

This comment has been minimized.

Copy link
Member

commented Dec 21, 2016

@remidebette can you include your JupyterHub configuration? I suspect a base_url is getting duplicated (hence /jupyter/hub/jupyter/), but it's not quite obvious where this is happening.

@minrk minrk added this to the 0.7.1 milestone Dec 21, 2016

@goulou

This comment has been minimized.

Copy link

commented Dec 21, 2016

In my setup, none of the above solutions seemed to work... This might be due to the fact that Apache runs in SSL mode.
However, the following solution works perfectly, and seems cleaner to me :

  ProxyPreserveHost On
  RewriteEngine On
  RewriteCond %{HTTP:Upgrade} =websocket [NC]
  RewriteRule /(.*)           ws://jupyter_host:8888/$1 [P,L]
  RewriteCond %{HTTP:Upgrade} !=websocket [NC]
  RewriteRule /(.*)           http://jupyter_host:8888/$1 [P,L]

It uses HTTP headers instead of the URL, so should be more generic.

Setup details :
OS : debian Jessie (8)
Apache version : Apache/2.4.10 (Debian)
The version of the notebook server is 4.3.0 and is running on:
Python 3.5.2+ (default, Nov 22 2016, 01:00:20)
[GCC 6.2.1 20161119]

I hope this will help!

@remidebette

This comment has been minimized.

Copy link

commented Feb 1, 2017

@minrk Sorry about the long delay.

Here are the uncommented parts of the configuration file I use:
(see my former comment for the apache reverse proxy code)

c.JupyterHub.base_url = '/jupyter'
c.JupyterHub.db_url = 'mysql+pymysql://[localconf]'
c.JupyterHub.spawner_class = 'sudospawner.SudoSpawner'

Others are just user and admin configuration
Jupyter 0.7, apache 2.4.18, php 7.0.13-0ubuntu0.16.10.1, Ubuntu 16.10

Remi

@remidebette

This comment has been minimized.

Copy link

commented Feb 12, 2017

@goulou Your solution works fine for me.

But this solution requires all the subpaths to be redirected to /jupyter/hub/...
In my configuration I still want to use several applications on the same host, based on the subpaths.
Which brings us back to this error since jupyterhub 0.7

@rgbkrk

This comment has been minimized.

Copy link
Member

commented Mar 9, 2017

This version works well for the first connection:

  ProxyPreserveHost On
  RewriteEngine On
  RewriteCond %{HTTP:Upgrade} =websocket [NC]
  RewriteRule /(.*)           ws://jupyter_host:8888/$1 [P,L]
  RewriteCond %{HTTP:Upgrade} !=websocket [NC]
  RewriteRule /(.*)           http://jupyter_host:8888/$1 [P,L]

However, any reconnect or restart attempt ends up with new errors. They happen on close of the original websocket.

@rgbkrk

This comment has been minimized.

Copy link
Member

commented Mar 10, 2017

Of note, we were experiencing this with Apache 2.4.7 (current Trusty Ubuntu stable)

@rgbkrk

This comment has been minimized.

Copy link
Member

commented Mar 10, 2017

I can reproduce this really quickly with an Apache proxy fronted notebook by closing the current kernel:

Jupyter.notebook.kernel.ws.close()

The WebSocket.close function is a native function and I can't seem to catch the WebSocket connection to '...' failed: One or more reserved bits are on: reserved1 = 0, reserved2 = 1, reserved3 = 1 error.

@ianabc

This comment has been minimized.

Copy link

commented Mar 10, 2017

I'm on apache Apache/2.4.6 (CentOS) and I'm using something like

ProxyPass http://127.0.0.1:8000/jupyter
ProxyPassReverse http://127.0.0.1:8000/jupyter
ProxyPreserveHost on
<LocationMatch "/jupyter/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)(.*)">
    ProxyPassMatch ws://127.0.0.1:8000/jupyter/$1/$2$3
    ProxyPassReverse ws://127.0.0.1:8000/jupyter/$1/$2$3
</LocationMatch>

I see the same behaviour closing websockets but only via the notebook restart action. Selecting kernel->restart leaves my browser in a loop trying to reconnect the kernel as described in (jupyter/notebook#1835).

If I do Jupyter.notebook.kernel.ws.close() in chrome's console I see one instance of the error message you get above, but then I do actually get reconnected without any problem

WebSocket connection to 'wss://ubc.syzygy.ca/jupyter/user/ianatr/api/kernels/a94f074d-c534-4653-8284-22309b590960/channels?session_id=D872FB9D684D451B8802474890B9EBE4' failed: One or more reserved bits are on: reserved1 = 0, reserved2 = 1, reserved3 = 1
WrappedWebSocket @ VM297:35
WrappedWebSocket @ VM328:35
Kernel.start_channels @ kernel.js:455
Kernel.reconnect @ kernel.js:349
i @ jquery.min.js:4
kernel.js:100 Kernel: kernel_disconnected (a94f074d-c534-4653-8284-22309b590960)
kernel.js:538 WebSocket connection failed:  wss://ubc.syzygy.ca/jupyter/user/ianatr/api/kernels/a94f074d-c534-4653-8284-22309b590960 true
kernel.js:556 Connection lost, reconnecting in 1 seconds.
kernel.js:100 Kernel: kernel_reconnecting (a94f074d-c534-4653-8284-22309b590960)
kernel.js:453 Starting WebSockets: wss://ubc.syzygy.ca/jupyter/user/ianatr/api/kernels/a94f074d-c534-4653-8284-22309b590960
kernel.js:100 Kernel: kernel_connected (a94f074d-c534-4653-8284-22309b590960)
manager-base.js:195 Widget backend and frontend versions are compatible
kernel.js:100 Kernel: kernel_ready (a94f074d-c534-4653-8284-22309b590960)
@Geremia

This comment has been minimized.

Copy link

commented Mar 10, 2017

If I do Jupyter.notebook.kernel.ws.close() in chrome's console I see one instance of the error message you get above, but then I do actually get reconnected without any problem

Unfortunately, that doesn't solve the issue for me in Firefox or Chromium. I use a similar configuration you did but with Apache 2.4.25.

Update: I was able to fix the issue by adding RequestHeader set Origin "http://localhost:8888" to my webserver config (fix courtesy: this answer).

@ianabc

This comment has been minimized.

Copy link

commented Apr 12, 2017

I have this problem on a bunch of hubs now, I haven't really figured out how to debug it. I've tried making local modifications to kernel.js within Chrome and saving them, but they never seem to be applied. The most likely thing I've found is this websocket-sharp issue which talks about problems with Chrome's handling of compression. If I can figure out how, I'll try disabling permessage-delate.

Here is a summary of the behaviour in my case:

  • Reloading a notebook works fine in Firefox and Chrome, the new websocket connection is created no problem
  • Using the reconnect action in Firefox is fine, the socket gets reconnected
  • Manually closing the connection in chrome (Jupyter.notebook.kernel.ws.close()) works fine, everything gets reconnected OK.
  • Using the reconnect action in chrome triggers the loop. I get an loop of Connection failed notices from the notebook, and a "failed: Invalid frame header" for the websocket in the console.
@kckern

This comment has been minimized.

Copy link

commented Oct 20, 2017

Here's the configuration that finally worked for me (Apache/2.4.18 (Ubuntu))

<VirtualHost *:80>
    ServerName jupyter.example.com
    <Location />
	    RequestHeader unset Accept-Encoding
	    ProxyPass        http://127.0.0.1:8888/
	    ProxyPassReverse http://127.0.0.1:8888/
	    ProxyPreserveHost on
	    Order allow,deny
	    Allow from all
	</Location>
	<Location /api/kernels/>
	    ProxyPass        ws://127.0.0.1:8888/api/kernels/
	    ProxyPassReverse ws://127.0.0.1:8888/api/kernels/
	</Location>
</VirtualHost>

consideRatio pushed a commit to consideRatio/jupyterhub that referenced this issue Feb 19, 2018

Merge pull request jupyterhub#219 from yuvipanda/reference-not-schema
Call reference documentation reference & not as schema
@tfabbri

This comment has been minimized.

Copy link

commented Feb 25, 2018

Here's my configuration that worked on my VPS:

<VirtualHost *:80>
    ServerName ipynb.example.ovh
    ServerAdmin my@email.com
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    ProxyPreserveHost On
    ProxyRequests off

    <Location /api/kernels/>
        ProxyPass        ws://example.ovh:9999/api/kernels/
        ProxyPassReverse ws://example.ovh:9999/api/kernels/
    </Location>

    ProxyPass       /  http://example.ovh:9999/
    ProxyPassReverse / http://example.ovh:9999/
    RequestHeader set Origin "http://example.ovh:9999"
    Redirect permanent / http://ipynb.example.ovh
</VirtualHost>
@gerroon

This comment has been minimized.

Copy link

commented Dec 19, 2018

@tfabbri

what is the ip here Redirect permanent / http://ipynb.example.ovh

@pkumar8789

This comment has been minimized.

Copy link

commented Feb 12, 2019

Went through issue #219 and related links but no success till yet.

I have a reverse proxy setup on Jupyterhub using Apache 2.4. The redirection to JupyterHUB url from Apache URL works fine but the internal Websocket connection after Kernel is created fails as it takes the Port from the Apache URL and not the redirected Port.

Apache URL: https://HOST_NAME:31000

JupyterHUB redirect URL: https://HOST_NAME:8000

but websocket connection is attempted at wss://HOST_NAME:31000 which fails. The URL redirection doesn't happen for websocket.

Apache Conf:

Listen HOST_NAME:31000

ServerName HOST_NAME

RewriteEngine On
RewriteCond %{HTTP:Connection} Upgrade [NC]
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteRule /(.*) ws://HOST_NAME:8000/$1 [P,L]

<Location ~ "/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/?">
ProxyPreserveHost on
# proxy to JupyterHub
ProxyPass https://HOST_NAME:8000/
ProxyPassReverse ws://HOST_NAME:8000/

Note: HOST_NAME is the server on which JupyterHub is running. Apache is also running on the same server.

While trying to open Notebook session, I see Kernel Created.

Starting websockets: wss://HOST_NAME:31000/user/d12333/api/kernels/d34343/channels?session...failing to connect. if the URL re-direction for websocket uses 8000 port then this should work.

But not able to configure the redirect through apache cnfiguration.

proxy_wstunnel_module, proxy_http_module and proxy_module are all enabled.
console

@lbiemans

This comment has been minimized.

Copy link

commented Feb 12, 2019

Checked one of our configs and found some differences like:

Listen HOST_NAME:31000 ==> We don't use it

Below a snippet from the websocket config:

    RewriteEngine On
    RewriteCond %{HTTP:Connection} Upgrade [NC]
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteRule /(.*) ws://HOST_NAME:8000/$1 [P,L]

    <Location "/">
            ProxyPreserveHost on
            ProxyPassReverse  http://HOST_NAME:8000/
            ProxyPass         http://HOST_NAME:8000/
    </Location>
@pkumar8789

This comment has been minimized.

Copy link

commented Feb 12, 2019

Checked one of our configs and found some differences like:

Listen HOST_NAME:31000 ==> We don't use it

Below a snippet from the websocket config:

    RewriteEngine On
    RewriteCond %{HTTP:Connection} Upgrade [NC]
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteRule /(.*) ws://HOST_NAME:8000/$1 [P,L]

    <Location "/">
            ProxyPreserveHost on
            ProxyPassReverse  http://HOST_NAME:8000/
            ProxyPass         http://HOST_NAME:8000/
    </Location>

The reason I used Listen host_name:31000 is because Apache URL is setup at that port and I want all incoming calls to be redirected to JupyterHUB URL host_name:8000.

@lbiemans

This comment has been minimized.

Copy link

commented Feb 12, 2019

Okay makes sense :-).

Have you tried the <Location "/"> instead of <Location ~ "/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/?"> ?

What does the apache debug and jupyterhub log tell you?

@pkumar8789

This comment has been minimized.

Copy link

commented Feb 12, 2019

Okay makes sense :-).

Have you tried the <Location "/"> instead of <Location ~ "/(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/?"> ?

What does the apache debug and jupyterhub log tell you?

Tried the tag earlier but later changed to regex as per few suggestions for fixing the issue.
Apache logs shows no anomalies. JupyterHUB console logs shows wss connection failed as I have attached in my original post.
Let me also check logs on JupyterHUB server.
Thanks for helping me out here. Really bugged by this issue :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.