Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to add jadmin to proxy and to spawn notebook #572

Closed
shakthimaan opened this issue May 15, 2020 · 8 comments
Closed

Failed to add jadmin to proxy and to spawn notebook #572

shakthimaan opened this issue May 15, 2020 · 8 comments
Labels
bug Something isn't working

Comments

@shakthimaan
Copy link

shakthimaan commented May 15, 2020 •
edited

Bug description

Used the following method to install The Littlest JupyterHub on an Ubuntu 18.0404 LTS system:

curl https://raw.githubusercontent.com/jupyterhub/the-littlest-jupyterhub/master/bootstrap/bootstrap.py | sudo -E python3 - --admin jadmin

and the installation went fine, but, the spawner fails to start with the following error:

I am able to add users though, but, just the notebook does not get spawned.

Expected behaviour

The jadmin/ page to change the password should show up and the tree directory structure should be displayed.

Actual behaviour

The output of jouralctl -u jupyterhub.

-- Logs begin at Thu 2019-08-29 12:01:21 BST, end at Fri 2020-05-15 07:19:13 BST. --
May 15 06:28:05 foo systemd[1]: Started jupyterhub.service.
May 15 06:28:05 foo systemd[1]: Stopping jupyterhub.service...
May 15 06:28:05 foo systemd[1]: Stopped jupyterhub.service.
May 15 06:28:05 foo systemd[1]: Started jupyterhub.service.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.305 JupyterHub app:2240] Running JupyterHub version 1.1.0
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.306 JupyterHub app:2271] Using Authenticator: firstuseauthenticator.firstuseauthenticator.FirstUseAuthenticator
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.307 JupyterHub app:2271] Using Spawner: builtins.UserCreatingSpawner
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.307 JupyterHub app:2271] Using Proxy: jupyterhub_traefik_proxy.toml.TraefikTomlProxy-0+unknown
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.317 JupyterHub app:1394] Writing cookie_secret to /opt/tljh/state/jupyterhub_cookie_secret
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.340 alembic.runtime.migration migration:155] Context impl SQLiteImpl.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.340 alembic.runtime.migration migration:162] Will assume non-transactional DDL.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.356 alembic.runtime.migration migration:515] Running stamp_revision  -> 4dc2d5a8c53c
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.369 alembic.runtime.migration migration:155] Context impl SQLiteImpl.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.369 alembic.runtime.migration migration:162] Will assume non-transactional DDL.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.436 JupyterHub app:1655] Not using whitelist. Any authenticated user will be allowed.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.483 JupyterHub app:2311] Initialized 0 spawners in 0.010 seconds
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.486 JupyterHub app:2520] Not starting proxy
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.486 JupyterHub app:2556] Hub API listening on http://127.0.0.1:15001/hub/
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.486 JupyterHub app:2571] Starting managed service cull-idle
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.487 JupyterHub service:335] Starting service 'cull-idle': ['/opt/tljh/hub/bin/python3', '-m', 'jupyterhub_idle_culler', '--timeout=600', '--cull-every=60', '--concurrency=5', '--max-age=0']
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.489 JupyterHub service:121] Spawning /opt/tljh/hub/bin/python3 -m jupyterhub_idle_culler --timeout=600 --cull-every=60 --concurrency=5 --max-age=0
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.495 JupyterHub proxy:320] Checking routes
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.495 JupyterHub proxy:400] Adding default route for Hub: / => http://127.0.0.1:15001
May 15 06:28:07 foo python3[5859]: /opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/traefik_utils.py:42: RuntimeWarning: Escape character '_' cannot be a safe character. Set allow_collisions=True if you want to allow ambiguous escaped strings.
May 15 06:28:07 foo python3[5859]:   return server_type + "_" + escapism.escape(routespec, safe=safe)
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.517 JupyterHub proxy:129] Waiting for / to register with traefik
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.730 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 18.25ms
May 15 06:28:08 foo python3[5859]: [I 2020-05-15 06:28:08.556 JupyterHub app:2631] JupyterHub is now running at http://:8000
May 15 06:29:07 foo python3[5859]: [I 2020-05-15 06:29:07.725 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 14.70ms
May 15 06:30:07 foo python3[5859]: [I 2020-05-15 06:30:07.725 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 14.49ms
May 15 06:31:07 foo python3[5859]: [I 2020-05-15 06:31:07.724 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 14.04ms
May 15 06:31:49 foo python3[5859]: [I 2020-05-15 06:31:49.596 JupyterHub log:174] 200 GET /hub/login (@127.0.0.1) 51.79ms
May 15 06:32:07 foo python3[5859]: [I 2020-05-15 06:32:07.723 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 13.19ms
May 15 06:32:14 foo python3[5859]: [I 2020-05-15 06:32:14.732 JupyterHub base:707] User logged in: jadmin
May 15 06:32:14 foo python3[5859]: [I 2020-05-15 06:32:14.735 JupyterHub log:174] 302 POST /hub/login?next= -> /hub/spawn (jadmin@127.0.0.1) 357.95ms
May 15 06:32:15 foo useradd[6178]: new group: name=jupyter-jadmin, GID=1013
May 15 06:32:15 foo useradd[6178]: new user: name=jupyter-jadmin, UID=1011, GID=1013, home=/home/jupyter-jadmin, shell=/bin/sh
May 15 06:32:15 foo python3[5859]: Adding user jupyter-jadmin to group jupyterhub-users
May 15 06:32:15 foo gpasswd[6185]: user jupyter-jadmin added by root to group jupyterhub-users
May 15 06:32:15 foo python3[5859]: Adding user jupyter-jadmin to group jupyterhub-admins
May 15 06:32:15 foo gpasswd[6189]: user jupyter-jadmin added by root to group jupyterhub-admins
May 15 06:32:15 foo python3[5859]: Running as unit: jupyter-jadmin.service
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.013 JupyterHub log:174] 302 GET /hub/spawn -> /hub/spawn-pending/jadmin (jadmin@127.0.0.1) 1007.46ms
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.290 JupyterHub pages:347] jadmin is pending spawn
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.296 JupyterHub log:174] 200 GET /hub/spawn-pending/jadmin (jadmin@127.0.0.1) 14.35ms
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.562 JupyterHub log:174] 200 GET /hub/api (@127.0.0.1) 2.05ms
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.591 JupyterHub log:174] 200 POST /hub/api/users/jadmin/activity (jadmin@127.0.0.1) 20.54ms
May 15 06:32:16 foo python3[5859]: [W 2020-05-15 06:32:16.708 JupyterHub _version:56] jupyterhub version 1.1.0 != jupyterhub-singleuser version 1.0.0. This could cause failure to authenticate and result in redirect loops!
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.708 JupyterHub base:855] User jadmin took 1.698 seconds to start
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.709 JupyterHub proxy:262] Adding user jadmin to proxy /user/jadmin/ => http://127.0.0.1:50277
May 15 06:32:16 foo python3[5859]: /opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/traefik_utils.py:42: RuntimeWarning: Escape character '_' cannot be a safe character. Set allow_collisions=True if you want to allow ambiguous escaped strings.
May 15 06:32:16 foo python3[5859]:   return server_type + "_" + escapism.escape(routespec, safe=safe)
May 15 06:32:16 foo python3[5859]: [E 2020-05-15 06:32:16.711 JupyterHub base:876] Failed to add jadmin to proxy!
May 15 06:32:16 foo python3[5859]:     Traceback (most recent call last):
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 869, in finish_user_spawn
May 15 06:32:16 foo python3[5859]:         await self.proxy.add_user(user, server_name)
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/proxy.py", line 274, in add_user
May 15 06:32:16 foo python3[5859]:         {'user': user.name, 'server_name': server_name},
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/toml.py", line 191, in add_route
May 15 06:32:16 foo python3[5859]:         self.toml_dynamic_config_file, self.routes_cache
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/traefik_utils.py", line 153, in persist_routes
May 15 06:32:16 foo python3[5859]:         toml.dump(routes_dict, config_fd)
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/toml/encoder.py", line 29, in dump
May 15 06:32:16 foo python3[5859]:         d = dumps(o, encoder=encoder)
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/toml/encoder.py", line 67, in dumps
May 15 06:32:16 foo python3[5859]:         raise ValueError("Circular reference detected")
May 15 06:32:16 foo python3[5859]:     ValueError: Circular reference detected
May 15 06:32:16 foo python3[5859]:     
May 15 06:32:16 foo python3[5859]: [E 2020-05-15 06:32:16.713 JupyterHub base:878] Stopping jadmin to avoid inconsistent state
May 15 06:32:17 foo python3[5859]: [W 2020-05-15 06:32:17.788 JupyterHub web:1786] 400 GET /hub/api/users/jadmin/server/progress (127.0.0.1): jadmin is not starting...
May 15 06:32:17 foo python3[5859]: [W 2020-05-15 06:32:17.789 JupyterHub log:174] 400 GET /hub/api/users/jadmin/server/progress (jadmin@127.0.0.1) 12.50ms
May 15 06:33:07 foo python3[5859]: [I 2020-05-15 06:33:07.724 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 14.06ms
May 15 06:33:08 foo python3[5859]: /opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/toml.py:104: RuntimeWarning: Escape character '_' cannot be a safe character. Set allow_collisions=True if you want to allow ambiguous escaped strings.
May 15 06:33:08 foo python3[5859]:   escaped_routespec = escapism.escape(traefik_routespec, safe=safe)
May 15 06:33:08 foo python3[5859]: [I 2020-05-15 06:33:08.561 JupyterHub proxy:320] Checking routes

How to reproduce

Just use the install script with curl to do an installation and try to open and login to the GUI.

Your personal set up

  • OS: Ubuntu 18.04.4 LTS (x86_64)
  • Version: JupyterHub 1.1.0
@zrf1
Copy link

zrf1 commented May 15, 2020

Have you solved it? I have the same problem. Please let me know if you can solve it. It has been reinstalled several times. The problem is the same. It is the same to check the log error code.

journalctl -u jupyterhub

May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]: /opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/traefik_utils.py:42: RuntimeWarning: Escape ch
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:   return server_type + "_" + escapism.escape(routespec, safe=safe)
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]: [E 2020-05-15 20:02:26.802 JupyterHub base:876] Failed to add admin to proxy!
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:     Traceback (most recent call last):
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 869, in finish_user_spaw
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:         await self.proxy.add_user(user, server_name)
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/proxy.py", line 274, in add_user
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:         {'user': user.name, 'server_name': server_name},
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/toml.py", line 191, in add_route
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:         self.toml_dynamic_config_file, self.routes_cache
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/traefik_utils.py", line 153, in pe
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:         toml.dump(routes_dict, config_fd)
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:       File "/opt/tljh/hub/lib/python3.6/site-packages/toml/encoder.py", line 29, in dump
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:         d = dumps(o, encoder=encoder)
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:       File "/opt/tljh/hub/lib/python3.6/site-packages/toml/encoder.py", line 67, in dumps
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:         raise ValueError("Circular reference detected")
May 15 20:02:26 iZ2ze9urlacrut4s79afkbZ python3[9327]:     ValueError: Circular reference detected

@Andrew-Dupuis
Copy link

Andrew-Dupuis commented May 15, 2020 •
edited

I spun up a completely fresh instance of Ubuntu 18.04.4 LTS (x86_64), and was able to reproduce the issue with the current installer CURL script. I think the issues is in the use of "_" in line 41 of traefik_utils.py:

safe = string.ascii_letters + string.digits + "_-"
return server_type + "_" + escapism.escape(routespec, safe=safe)

@minrk merged a new unsafe warning into escapism about two weeks ago that removes the unsafe character, leading to a badly formed alias string. I think this is the same issue as #106 in traefik-proxy

I'm working on a PR to either add the required allow_collisions flag to the escapism call - but I'm also not sure if allow_collisions is actually a desired behavior here. Will update once I know what's going on and what's submitted.

@shakthimaan
Copy link
Author

Given an existing installation, please also advice on the recommended upgrade path for the fix. Thanks for the quick turnaround time. Appreciate it!

@manics manics mentioned this issue May 15, 2020
Closed
@manics manics added the bug Something isn't working label May 15, 2020
@GeorgianaElena
Copy link
Member

GeorgianaElena commented May 15, 2020 •
edited

@apd47, I managed to reproduce this using Yuvi's steps here. I think is has to do with the recent release of toml and not with the new escapism version as with the new version of escapism and the previous one of toml everything seems to be fine.

A temporary fix, until we figured out what's the issue might be pinning the toml version to 0.10.0

@ilfsn
Copy link

ilfsn commented May 15, 2020 •
edited

@apd47, I managed to reproduce this using Yuvi's steps here. I think is has to do with the recent release of toml and not with the new escapism version as with the new version of escapism and the previous one of toml everything seems to be fine.

A temporary fix, until we figured out what's the issue might be pinning the toml version to 0.10.0

Hi, can you please introducing the temporary fix in detail steps ? Thank you so much.

@GeorgianaElena
Copy link
Member

Install toml 0.10.0:

/opt/tljh/hub/bin/python3 -m pip install toml==0.10.0

Restart hub and traefik services:

systemctl restart traefik
systemctl restart jupyterhub

@ilfsn, let me know if it works 🌞

@jsstokes
Copy link

@GeorgianaElena - I was having the same issue and stumbled across this. It worked for me! 🍾

@GeorgianaElena
Copy link
Member

The toml version has been pinned in traefik-proxy and a new release has been made 🎉
Bumping traefik-proxy should solve this issue for existing installs.

@GeorgianaElena GeorgianaElena mentioned this issue May 16, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@shakthimaan @manics @GeorgianaElena @ilfsn @Andrew-Dupuis @jsstokes @zrf1