jupyterhub/singleuser No user token cookie?

[Spritekin]

Hi,
I run a docker swarm cluster based on mesos. I have configured my jupyterhub to authenticate with LDAP and then it starts the jupyter servers on swarm so I can see them appearing on the mesos cluster.

The problem is that those environments are unresponsive even when they seem ok. This is an example log:

I0421 14:12:49.394819 11264 exec.cpp:162] Version: 1.2.0
I0421 14:12:49.398325 11266 exec.cpp:237] Executor registered on agent b7cfea24-75c1-43d4-8727-49fb86ca66ae-S1
I0421 14:12:49.399019 11266 docker.cpp:850] Running docker -H unix:///var/run/docker.sock run --memory 2147483648 --env-file /tmp/XZKgjL -v /var/lib/mesos/slave/slaves/b7cfea24-75c1-43d4-8727-49fb86ca66ae-S1/frameworks/b2853a78-7cba-455a-99c7-d73bb5cbda95-0024/executors/jupyter-grodriguez.537d87935e09/runs/aa93541b-ab40-4903-a107-c137f1f8789b:/mnt/mesos/sandbox --net bridge --label=com.docker.swarm.mesos.name=jupyter-grodriguez --label=com.docker.swarm.mesos.task=jupyter-grodriguez.537d87935e09 --env=JPY_COOKIE_NAME=jupyter-hub-token-grodriguez --env=JPY_HUB_PREFIX=/hub/ --env=JPY_API_TOKEN=76bd38d982b947a9be0df68f06b1e721 --env=JUPYTERHUB_CLIENT_ID=user-grodriguez --env=JUPYTERHUB_HOST= --env=JPY_USER=grodriguez --env=JUPYTERHUB_API_TOKEN=76bd38d982b947a9be0df68f06b1e721 --env=MEM_LIMIT=2147483648 --env=JPY_BASE_URL=/user/grodriguez --env=JUPYTERHUB_OAUTH_CALLBACK_URL=/user/grodriguez/oauth_callback --env=JPY_HUB_API_URL=http://172.31.45.116:8081/hub/api --env=NOTEBOOK_DIR=/home/jovyan/work -p 80:8888/tcp --name mesos-b7cfea24-75c1-43d4-8727-49fb86ca66ae-S1.aa93541b-ab40-4903-a107-c137f1f8789b jupyterhub/singleuser
[W 2017-04-21 14:12:50.454 grodriguez configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[I 2017-04-21 14:12:50.459 grodriguez notebookapp:511] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[W 2017-04-21 14:12:50.499 grodriguez login:222] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[I 2017-04-21 14:12:50.525 grodriguez extension:193] JupyterLab alpha preview extension loaded from /opt/conda/lib/python3.5/site-packages/jupyterlab
[I 2017-04-21 14:12:50.529 grodriguez notebookapp:1201] Serving notebooks from local directory: /home/jovyan/work
[I 2017-04-21 14:12:50.529 grodriguez notebookapp:1201] 0 active kernels 
[I 2017-04-21 14:12:50.529 grodriguez notebookapp:1201] The Jupyter Notebook is running at: http://0.0.0.0:8888/user/grodriguez/
[I 2017-04-21 14:12:50.529 grodriguez notebookapp:1202] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2017-04-21 14:12:50.546 grodriguez log:47] 302 GET /user/grodriguez (172.31.45.116) 0.63ms
[I 2017-04-21 14:12:50.947 grodriguez log:47] 302 GET /user/grodriguez (::ffff:192.168.16.108) 0.43ms
[I 2017-04-21 14:12:51.265 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.81ms
[I 2017-04-21 14:12:52.068 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.57ms
[I 2017-04-21 14:12:52.879 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.58ms
[I 2017-04-21 14:12:53.746 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.63ms
[I 2017-04-21 14:12:54.576 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.60ms
[I 2017-04-21 14:12:55.328 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.61ms
[I 2017-04-21 14:12:56.269 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.56ms

Everything looks ok I think. However the server always responds with a 404 Not Found error and it keeps trying to redirect to the /user/grodriguez/tree page in an eternal loop.

So I enabled the debug (by passing --debug in the notebook command) and found this:

[I 2017-04-24 05:00:34.393 grodriguez notebookapp:511] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[W 2017-04-24 05:00:34.431 grodriguez login:222] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[I 2017-04-24 05:00:34.442 grodriguez notebookapp:1201] Serving notebooks from local directory: /home/jovyan/work
[I 2017-04-24 05:00:34.442 grodriguez notebookapp:1201] 0 active kernels 
[I 2017-04-24 05:00:34.442 grodriguez notebookapp:1201] The Jupyter Notebook is running at: http://0.0.0.0:8888/user/grodriguez/
[I 2017-04-24 05:00:34.442 grodriguez notebookapp:1202] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2017-04-24 05:00:34.505 grodriguez log:47] 302 GET /user/grodriguez (172.31.45.116) 0.58ms
[I 2017-04-24 05:00:35.023 grodriguez log:47] 302 GET /user/grodriguez (::ffff:192.168.16.110) 0.37ms
[D 2017-04-24 05:00:35.261 grodriguez auth:219] No token cookie
[I 2017-04-24 05:00:35.261 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.110) 0.67ms
[D 2017-04-24 05:00:36.024 grodriguez auth:219] No token cookie
[I 2017-04-24 05:00:36.025 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.110) 0.76ms
[D 2017-04-24 05:00:36.767 grodriguez auth:219] No token cookie

So it is telling me the cookie is not found? But the cookie was created in the server:

cat /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret                                                                                                          
UzrSzGx1qQ/sWL7lKdRN9JvgC6zBxiAa1KOp6qFVnB+y3/m+nRBt5ub/X+JfTugzxKlj8Jd7NZSQ
tkYjR9IMat+8Ic0MKRFCIEugJhvBE8VeSXOUUBQfUoYpVRwEIDW1Gws2DaZtG4bEo9d4f7drROg0
yPkaV52dBNLp1GETGSLNtWSSLHLu3scjpmkOAUCqdWDoTTatki5j8AzuU29gfieCA3Bmp23olrdg
aiIvAItJ9Cav+b+OnAoMNT+Jb1Vy6C4By5IGHPepnXRglkrdw51/Iak3RB8x/61rgJlvDqkNkceG
uLINnJAdmM5fyN70BVDsenlRu8Vhw59pjy15tjSqx0UCb1PcoqyuxTJuKxwVFAt04ELHqBXj3yrB
SE1PhHMsfvXLQIPdufK5+k+PfTyMWUcqOuG3d7Bz4co0j3IwZbuvh7+p0sUSkmbjIDuaeSezf0hx
0gxlXRhz0HZ9MbxDOpZQawc6+zYLk4rtZblhQyPawL+XC3MZFGWRjE6nTxF4aQY12mLDQ8V81UDb
kkANNjn23nzCq2pUlgSGuKNNdr/RFdrlx5O2yKQ+hDv1KRlMPaKzSOlQtmXV9cg4barTJjNnDEAh
PUHR/mYaO5EpzjQs/+0q/cJ8v+ByCvlvxb4KQu8FsXY3MHskabYX1+8kR/LeSRTiNVHDTOZga0Li
enG85FvCT371QG+/xi5yO/24PP1Kn2JIO76fF63eekfxT/aOZon3MDlsdd4EZhZY45aknW6Zy3DU
apt07teGl+jUXAvLoAD31qXNtKFOAW/qwkZr1pVH9bffT2GQiHt81g/FfdaGAJ5Hur1KwFJ1IB5l
CZgifdgvl1CSWKvo7fM9hKvMi9VJTns99SGgi5Q5PM/6C+M5CtOMAVFFWCPaxZG6Cq2pRk/eD86T
jU4QpGEaJgmlBMgmjEgtlgXo7ttAHKbxHizMItwkVT5xhZpZ0xPX1JOUARPojhjX0RW+fb+E9o6a
5BYQjdPHfeofSig4fWO69suLN2EVC8u4C7Tdezj6tSm24GF016Rs5z/ACjVOQVgdTl9F9ct6MIWi
qbKSqRRsRlrdHggNjyrkMDLIUtK1SsVrFKcf7Jq62tRNVBZwsOe40lFIOzZmdviECAItRVaJIBYI
o1bS/quY+7nM3BXbBBNhyBS+8B3A4xLeQs6YVcP+9TH4Zik54MfiKfr1hrT7m/JLVnECuBIj3e3Y
rr4oi00dZnnB5G0CXroocYTjdecGxZFMX3Y+PaNVQ0lHbj7FD/3vA2QNIY/0EyZIBu2ntKubg6cw
f0Cu646ROaYLEHj9WIMuh8l13baIXb+7EOy0gSqSwlOllMe7TaMh11gkvcwPU88YMihDBjwv/g==

I can only imagine that means is not in my browser? How do I get it into my browser then
Any ideas would be appreciated.

Thanks!

[Spritekin]

Hmmm... I just noticed the startup command says the cookie is --env=JPY_COOKIE_NAME=jupyter-hub-token-grodriguez

But the notebook says the cookie name is notebook_cookie_secret?

[minrk]

cookie_secret is the file containing the encryption key for secure cookies. The notebook_cookie_secret is not used with JupyterHub. cookie_name is the name of the cookie to be set / checked (the key in the cookies dict). The No token cookie suggests that the cookie is not being set in your browser, or is not being set correctly. Can you check your browser's cookies for the Hub host?

[Spritekin]

Thanks @minrk !
Indeed that seems like the problem. When I go to my browser cookies I get this:

Name:	jupyter-hub-token
Content:	"2|1:0|10:1493180628|17:jupyter-hub-token|44:N2U3NmE5ZGZjMmU2NGVlOWE0NGU0ZDczYTViMDk5YWY=|91726edc9ce5fa5979bd459c94a10abede98e0039423dd7bbac3aab56df01a6a"
Domain:	172.31.45.116
Path:	/hub/
Send for:	Any kind of connection
Accessible to script:	Yes
Created:	Wednesday, April 26, 2017 at 2:23:48 PM
Expires:	Friday, May 26, 2017 at 2:23:49 PM

As you can see the cookie name is "jupyter-hub-token", but the cookie name passed to the notebook is
--env=JPY_COOKIE_NAME=jupyter-hub-token-grodriguez

So it would look like it won't find it. Any ideas how to fix this?

Thanks!

[Spritekin]

Ok so reading the docs:

When a new browser logs in to JupyterHub, the following events take place:

1. Login data is handed to the Authenticator instance for validation
2. The Authenticator returns the username, if login information is valid
3. A single-user server instance is Spawned for the logged-in user
4. When the server starts, the proxy is notified to forward /user/[username]/* to the single-user server
5. Two cookies are set, one for /hub/ and another for /user/[username], containing an encrypted token.
6. The browser is redirected to /user/[username], which is handled by the single-user server

So from the logs, it seems steps 1, 2, 3, 4 are done correctly, however step 5 only sets the /hub/ cookie but not the /user/username cookie.
Then step 6 fails.

[Spritekin]

Sorry it was a versioning problem.
Thanks for the help!

[minrk]

@Spritekin glad you figured it out! Can you give a bit of detail about the versioning problem? I'd like to try to get better errors to users when things fail due to incompatible versions, and having some known cases would be great.

[mcs07]

I just had the same problem with jupyterhub in a container with FROM jupyterhub/jupyterhub:latest, also solved by changing to FROM jupyterhub/jupyterhub:0.7.2. I guess it must just be an issue with the current master.

[mgeplf]

I believe that I also had the same problem: running a jupyterhub 'server' from master against a singleuser.py from a previous release (a conda installed version of jupyterhub (0.7.2)) causes the cookie lookup to fail, which in turn means the /tree/ path served by the singleuser.py tries to redirect to /hub/ and then an infinite redirection loop to happens.

[minrk]

Indeed, updating to master/0.8 requires updating both the Hub and the single-user server. Before the 0.8 release, I'll try to make sure we get more informative messages about mismatched versions.