Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't connect to the kernel #290

Closed
PaluchowskiMatthew opened this issue Mar 7, 2019 · 7 comments
Closed

Can't connect to the kernel #290

PaluchowskiMatthew opened this issue Mar 7, 2019 · 7 comments
Labels
bug Something isn't working

Comments

@PaluchowskiMatthew
Copy link

I successfully installed jupyterhub on a google cloud instance however I am unable to start a working jupyter notebook as it fails to connect to a kernel with a following message:

Connection failed A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration.

image

jupyterhub.service logs:

root@jupyterhub-0:/opt/tljh/config/jupyterhub_config.d# sudo journalctl -u jupyterhub.service
-- Logs begin at Thu 2019-03-07 11:12:13 UTC, end at Thu 2019-03-07 12:07:43 UTC. --
Mar 07 11:16:46 jupyterhub-0 systemd[1]: /etc/systemd/system/jupyterhub.service:12: Failed to parse protect home value, ignoring: tmpfs
Mar 07 11:16:46 jupyterhub-0 systemd[1]: Started jupyterhub.service.
Mar 07 11:16:46 jupyterhub-0 systemd[1]: Stopping jupyterhub.service...
Mar 07 11:16:46 jupyterhub-0 systemd[1]: Stopped jupyterhub.service.
Mar 07 11:16:46 jupyterhub-0 systemd[1]: Started jupyterhub.service.
Mar 07 11:16:46 jupyterhub-0 systemd[1]: /etc/systemd/system/jupyterhub.service:12: Failed to parse protect home value, ignoring: tmpfs
Mar 07 11:16:47 jupyterhub-0 systemd[1]: /etc/systemd/system/jupyterhub.service:12: Failed to parse protect home value, ignoring: tmpfs
Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.206 JupyterHub app:1673] Using Authenticator: firstuseauthenticator.firstuseauthenticator.FirstUseAuthenticator
Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.207 JupyterHub app:1673] Using Spawner: builtins.UserCreatingSpawner
Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.209 JupyterHub app:1055] Writing cookie_secret to /opt/tljh/state/jupyterhub_cookie_secret
Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.274 alembic.runtime.migration migration:130] Context impl SQLiteImpl.
Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.276 alembic.runtime.migration migration:137] Will assume non-transactional DDL.
Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.289 alembic.runtime.migration migration:356] Running stamp_revision -> 896818069c98
Mar 07 11:17:33 jupyterhub-0 useradd[7938]: new group: name=jupyter-deeption, GID=1014
Mar 07 11:17:33 jupyterhub-0 useradd[7938]: new user: name=jupyter-deeption, UID=1011, GID=1014, home=/home/jupyter-deeption, shell=/bin/sh
Mar 07 11:17:33 jupyterhub-0 python3[7815]: Adding user jupyter-deeption to group jupyterhub-users
Mar 07 11:17:33 jupyterhub-0 gpasswd[7945]: user jupyter-deeption added by root to group jupyterhub-users
Mar 07 11:17:33 jupyterhub-0 python3[7815]: Adding user jupyter-deeption to group jupyterhub-admins
Mar 07 11:17:33 jupyterhub-0 gpasswd[7949]: user jupyter-deeption added by root to group jupyterhub-admins
Mar 07 11:17:33 jupyterhub-0 python3[7815]: Running as unit: jupyter-deeption.service

jupyter-deeption (user) logs:

root@jupyterhub-0:/opt/tljh/config/jupyterhub_config.d# sudo journalctl -u jupyter-deeption
-- Logs begin at Thu 2019-03-07 11:12:13 UTC, end at Thu 2019-03-07 12:03:15 UTC. --
Mar 07 11:17:33 jupyterhub-0 systemd[1]: Started /bin/bash -c cd /home/jupyter-deeption && exec jupyterhub-singleuser --port=54437.
Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.639 SingleUserNotebookApp extension:168] JupyterLab extension loaded from /opt/tljh/user/lib/python3.6/site-packages/jupyterlab
Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.641 SingleUserNotebookApp extension:169] JupyterLab application directory is /opt/tljh/user/share/jupyter/lab
Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.677 SingleUserNotebookApp extension:23] nteract extension loaded from /opt/tljh/user/lib/python3.6/site-packages/nteract_on_jupyter
Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.679 SingleUserNotebookApp singleuser:406] Starting jupyterhub-singleuser server version 0.9.4
Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.688 SingleUserNotebookApp notebookapp:1712] Serving notebooks from local directory: /home/jupyter-deeption
Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.688 SingleUserNotebookApp notebookapp:1712] The Jupyter Notebook is running at:
Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.689 SingleUserNotebookApp notebookapp:1712] http://127.0.0.1:54437/user/deeption/
Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.689 SingleUserNotebookApp notebookapp:1713] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.795 SingleUserNotebookApp log:158] 302 GET /user/deeption/ -> /user/deeption/tree? (@127.0.0.1) 9.27ms
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.122 SingleUserNotebookApp log:158] 302 GET /user/deeption/?redirects=1 -> /user/deeption/tree?redirects=1 (@185.127.206.173) 1.22ms
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.149 SingleUserNotebookApp log:158] 302 GET /user/deeption/tree?redirects=1 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-deeption&redirect_uri=%2Fuser%2Fdeeption%2Foauth_callback&response_type=code&state=[secret] (@185.127.206.173) 3.46m
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.318 SingleUserNotebookApp auth:875] Logged-in user {'kind': 'user', 'name': 'deeption', 'admin': True, 'groups': [], 'server': '/user/deeption/', 'pending': None, 'created': '2019-03-07T11:16:48.410639Z', 'last_activity': '2019-03-07T11:17:35.309
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.320 SingleUserNotebookApp log:158] 302 GET /user/deeption/oauth_callback?code=[secret]&state=[secret] -> /user/deeption/tree?redirects=1 (@185.127.206.173) 80.10ms
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.404 SingleUserNotebookApp log:158] 200 GET /user/deeption/tree?redirects=1 (deeption@185.127.206.173) 58.36ms
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.922 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/tree?=1551957455858 (deeption@185.127.206.173) 3.65ms
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.928 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/terminals?=1551957455860 (deeption@185.127.206.173) 3.62ms
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.931 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/common?=1551957455859 (deeption@185.127.206.173) 5.10ms
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.938 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/kernelspecs (deeption@185.127.206.173) 5.39ms
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.940 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/sessions?=1551957455861 (deeption@185.127.206.173) 6.59ms
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.942 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/terminals?=1551957455862 (deeption@185.127.206.173) 1.34ms
Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.998 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/contents?type=directory&=1551957455863 (deeption@185.127.206.173) 3.94ms
Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.523 SingleUserNotebookApp log:158] 302 GET /user/deeption/ -> /user/deeption/tree? (@185.127.206.173) 1.40ms
Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.547 SingleUserNotebookApp log:158] 200 GET /user/deeption/tree (deeption@185.127.206.173) 3.15ms
Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.653 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/tree?=1551957522610 (deeption@185.127.206.173) 4.31ms
Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.656 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/common?=1551957522611 (deeption@185.127.206.173) 6.67ms
Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.678 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/terminals?=1551957522612 (deeption@185.127.206.173) 5.24ms
Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.681 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/sessions?=1551957522613 (deeption@185.127.206.173) 2.31ms
Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.684 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/terminals?=1551957522614 (deeption@185.127.206.173) 1.54ms
Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.703 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/contents?type=directory&=1551957522615 (deeption@185.127.206.173) 2.71ms
Mar 07 11:18:47 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:47.049 SingleUserNotebookApp handlers:153] Creating new notebook in
Mar 07 11:18:47 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:47.071 SingleUserNotebookApp sign:396] Writing notebook-signing key to /home/jupyter-deeption/.local/share/jupyter/notebook_secret
Mar 07 11:18:47 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:47.083 SingleUserNotebookApp log:158] 201 POST /user/deeption/api/contents (deeption@185.127.206.173) 34.74ms
Mar 07 11:18:47 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:47.371 SingleUserNotebookApp log:158] 200 GET /user/deeption/notebooks/Untitled.ipynb?kernel_name=python3 (deeption@185.127.206.173) 261.87ms
Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.093 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/notebook?=1551957527975 (deeption@185.127.206.173) 6.34ms
Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.098 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/common?=1551957527976 (deeption@185.127.206.173) 10.02ms
Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.157 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/contents/Untitled.ipynb?type=notebook&=1551957527977 (deeption@185.127.206.173) 7.11ms
Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.274 SingleUserNotebookApp kernelmanager:164] Kernel started: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969
Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.281 SingleUserNotebookApp log:158] 201 POST /user/deeption/api/sessions (deeption@185.127.206.173) 37.64ms
Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.295 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/contents/Untitled.ipynb/checkpoints?=1551957527978 (deeption@185.127.206.173) 5.95ms
Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.346 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 8.49ms
Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.905 SingleUserNotebookApp handlers:190] Adapting to protocol v5.1 for kernel 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969
Mar 07 11:18:48 jupyterhub-0 bash[7955]: /opt/tljh/user/lib/python3.6/site-packages/notebook/base/zmqhandlers.py:284: RuntimeWarning: coroutine 'WebSocketHandler.get' was never awaited
Mar 07 11:18:48 jupyterhub-0 bash[7955]: super(AuthenticatedZMQStreamHandler, self).get(*args, **kwargs)
Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.908 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/kernels/4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969/channels?session_id=99809f1975b44be79ec70788f2ee4b01 (deeption@185.127.206.173) 580.28ms
Mar 07 11:18:50 jupyterhub-0 bash[7955]: [W 2019-03-07 11:18:50.183 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01
Mar 07 11:18:53 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:53.337 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.43ms
Mar 07 11:18:58 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:58.336 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.58ms
Mar 07 11:19:37 jupyterhub-0 bash[7955]: [W 2019-03-07 11:19:37.266 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01
Mar 07 11:19:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:19:48.347 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.56ms
Mar 07 11:19:53 jupyterhub-0 bash[7955]: [I 2019-03-07 11:19:53.346 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.56ms
Mar 07 11:20:26 jupyterhub-0 bash[7955]: [W 2019-03-07 11:20:26.303 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01
Mar 07 11:20:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:20:48.259 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/contents/Untitled.ipynb?content=0&_=1551957527979 (deeption@185.127.206.173) 4.81ms
Mar 07 11:20:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:20:48.283 SingleUserNotebookApp handlers:164] Saving file at /Untitled.ipynb
Mar 07 11:20:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:20:48.297 SingleUserNotebookApp log:158] 200 PUT /user/deeption/api/contents/Untitled.ipynb (deeption@185.127.206.173) 15.73ms
Mar 07 11:20:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:20:48.351 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.84ms
Mar 07 11:20:53 jupyterhub-0 bash[7955]: [I 2019-03-07 11:20:53.352 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 5.50ms
Mar 07 11:21:19 jupyterhub-0 bash[7955]: [W 2019-03-07 11:21:19.410 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01
Mar 07 11:21:23 jupyterhub-0 bash[7955]: [I 2019-03-07 11:21:23.358 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.60ms
Mar 07 11:21:28 jupyterhub-0 bash[7955]: [I 2019-03-07 11:21:28.358 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 5.06ms
Mar 07 11:22:20 jupyterhub-0 bash[7955]: [W 2019-03-07 11:22:20.556 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01
Mar 07 11:22:38 jupyterhub-0 bash[7955]: [I 2019-03-07 11:22:38.399 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 38.32ms
Mar 07 11:22:43 jupyterhub-0 bash[7955]: [I 2019-03-07 11:22:43.403 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 5.15ms
Mar 07 11:23:37 jupyterhub-0 bash[7955]: [W 2019-03-07 11:23:37.682 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01
Mar 07 11:25:26 jupyterhub-0 bash[7955]: [W 2019-03-07 11:25:26.791 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01

I tried downgrading tornado as suggested here https://github.com/jupyter/notebook/issues/4399 however it did not resolve my issue... Any suggestions?
@yuyueugene84
Copy link

I can confirm that the same issue also occurs when I installed Jupyterhub in AWS.

@PaluchowskiMatthew
Copy link
Author

PaluchowskiMatthew commented Mar 8, 2019 •
edited

Apparently downgrading the tornado was the right call it just needs to be done in a right way:
From within running jupyterhub login as an admin user, open new terminal window and run

sudo -E pip install tornado==5.1.1

to downgrade tornado.

Then make sure to restart the server (within jupyrehub click on the control panel in the upper right corner, stop the server then start it again. From now on you should be able to connect to the notebook)

@willirath willirath reopened this Mar 8, 2019
@willirath
Copy link
Collaborator

Let's keep open until fixed.

willirath added a commit that referenced this issue Mar 8, 2019
@willirath
Pin tornado to <6
488683a 
See #290
@willirath willirath mentioned this issue Mar 8, 2019
Merged
@willirath
Copy link
Collaborator

@PaluchowskiMatthew Do you have a chance to check #292?

Would be done as follows:

  • Create a fresh machine, but do not provide a startup script.
  • SSH into the machine
  • Set env var to point PIP to the branch and run installer
branch_name="willirath-pin-tornado-5"

export \
    TLJH_BOOTSTRAP_PIP_SPEC="git+https://github.com/jupyterhub/the-littlest-jupyterhub.git@${branch_name}"

curl \
    https://raw.githubusercontent.com/jupyterhub/the-littlest-jupyterhub/${branch_name}/bootstrap/bootstrap.py \
    | sudo -E python3 - --admin <admin-user-name>

@PaluchowskiMatthew
Copy link
Author

@willirath
Problem persists and tornado version remains unchanged (6.0.1)

image

@leportella leportella added the bug Something isn't working label Mar 22, 2019
@yuvipanda
Copy link
Collaborator

Should hopefully be fixed now with #292. Please re-open if that isn't the case!

@hyamanieu hyamanieu mentioned this issue Jan 27, 2020
Closed
@awesome-crab
Copy link

still experience this issue. downgrading tornado is not so easy since there are about 10 packages relying on the latest version of tornado. any help would be appreciated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@yuvipanda @yuyueugene84 @willirath @leportella @PaluchowskiMatthew @awesome-crab