500 / 404 error on using docker spawner

[yuvipanda]

Hello!

(First, thanks for the awesome work!)

I'm trying to set up a publicly accessible jupyterhub setup on Wikimedia Labs (jupyter.wmflabs.org). Would like to use docker to provide isolation as well as resource quotas / clustering (later on). Jupyterhub works with the default spawner, but with dockerspawner I get a '500 server error' when I click 'my server'. And if I ssh tunnel through and hit the proxied docker endpoint directly, I get a 404. No errors in jupyterhub stdout, so I'm not sure what's happening.

Thoughts?

[yuvipanda]

Aha! I found a trace in the log:

Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1346, in _execute
        result = method(*self.path_args, **self.path_kwargs)
      File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 2613, in wrapper
        if not self.current_user:
      File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1028, in current_user
        self._current_user = self.get_current_user()
      File "/usr/local/lib/python2.7/dist-packages/IPython/html/base/handlers.py", line 72, in get_current_user
        return self.login_handler.get_user(self)
      File "/usr/local/bin/jupyterhub-singleuser", line 82, in get_user
        auth_data = JupyterHubLoginHandler.verify_token(self, self.cookie_name, encrypted_cookie)
      File "/usr/local/bin/jupyterhub-singleuser", line 51, in verify_token
        data=encrypted_cookie,
      File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
        return request('get', url, **kwargs)
      File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
        return session.request(method=method, url=url, **kwargs)
      File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
        r = adapter.send(request, **kwargs)
      File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
        raise ConnectionError(e)
    ConnectionError: HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /hub/api/authorizations/cookie/jupyter-hub-token-yuvipanda (Caused by <class 'socket.error'>: [Errno 111] Connection refused)

[jhamrick]

What's your JupyterHub config? Which authenticator are you using for JupyterHub?

[yuvipanda]

I'm using the sample GitHub OAuth authenticator. https://gist.github.com/yuvipanda/08e06376ddb98f21a8c4 is the config. It works now after I fiddled around a bit - I set the ip address explicitly in the config, and also rebuilt the standalone docker container. Unsure which fixed it.

[jhamrick]

I suspect setting the IP address explicitly in the config is what fixed it -- because the hub passes it's IP address along to the containers so they can communicate with the hub. The default is localhost, I think, but localhost inside the containers isn't the same as localhost for the rest of the machine (unless the containers are run with --net=host, but I think they're not). So, they need to have the real IP address to be able to connect.

Anyway, glad you were able to fix it!

[yuvipanda]

Yup, containers are not run with --net=host, so that won't work.

Thanks!

[netoisc]

Thanks @jhamrick !, I had the same problem when I launched the jupyterHub service. But now, I have my JupyterHub running 👍 . First, I executed the ifconfig command in the host, after that, I only had to change the line 'c.JupyterHub.hub_ip' with my ip and run again.

[anaderi]

I happen to stuck with this even after changing c.JupyterHub.hub_ip to host. Until I've removed already created containers (seem hub address is cached at the creation time and those are using 'localhost' even after restart)

[kynan]

I can confirm the behavioru @anaderi is seeing: any containers need to be removed.