POST api/kernel return 'Kernel does not exist'

[argenisleon]

Hi,

I installed kernel gateway via conda.

When I try to create a new kernel using POST I get:

[W 200806 14:40:23 web:1786] 404 POST /api/kernels (xxx.xxx.xxx.xxx): Kernel does not exist: <coroutine object MappingKernelManager.start_kernel at 0x7f2990577680>

I have tried using the --debug flag but I had not found any other error message.

Any hint to troubleshoot this?

PD: The kernelspec endpoint returns the kernels that already exist in the jupyter notebook installation.

[drauschenbach]

@argenisleon I started experiencing this as well on Saturday across all 4 of my isolated clusters. This is after weeks of stable running. It's as if a time bomb went off in the code somewhere.

[kevin-bates]

Hmm. Strange.

Can you share the output of jupyter --version?
I was going to suggest DEBUG, but see this has been updated accordingly.

[kevin-bates]

Are you using NB2KG or Notebook via --gateway-url to point at your JKG?

[drauschenbach]

Sure thing:

$ jupyter version

Traceback (most recent call last):
  File "/home/<redacted>/.local/bin/jupyter", line 11, in <module>
    sys.exit(main())
  File "/home/<redacted>/.local/lib/python3.6/site-packages/jupyter_core/command.py", line 247, in main
    command = _jupyter_abspath(subcommand)
  File "/home/<redacted>/.local/lib/python3.6/site-packages/jupyter_core/command.py", line 134, in _jupyter_abspath
    'Jupyter command `{}` not found.'.format(jupyter_subcommand)
Exception: Jupyter command `jupyter-version` not found.

$ ls -al ~/.local/bin/jupy*

-rwxr-xr-x 1 <redacted> <redacted> 223 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter
-rwxr-xr-x 1 <redacted> <redacted> 237 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-bundlerextension
-rwxr-xr-x 1 <redacted> <redacted> 227 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-kernel
-rwxr-xr-x 1 <redacted> <redacted> 239 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-kernelgateway
-rwxr-xr-x 1 <redacted> <redacted> 265 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-kernelspec
-rwxr-xr-x 1 <redacted> <redacted> 223 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-migrate
-rwxr-xr-x 1 <redacted> <redacted> 225 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-nbconvert
-rwxr-xr-x 1 <redacted> <redacted> 224 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-nbextension
-rwxr-xr-x 1 <redacted> <redacted> 223 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-notebook
-rwxr-xr-x 1 <redacted> <redacted> 244 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-run
-rwxr-xr-x 1 <redacted> <redacted> 228 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-serverextension
-rwxr-xr-x 1 <redacted> <redacted> 228 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-troubleshoot
-rwxr-xr-x 1 <redacted> <redacted> 256 Aug  8 09:35 /home/<redacted>/.local/bin/jupyter-trust

I run Jupyter Kernel Gateway 2.4.1, and use it from Jupyter Notebook 6.0.3. Here's a snippet from my installer DevOps:

pip3 install -I jupyter-kernel-gateway==2.4.1 --user
pip3 install -I notebook==6.0.3 --user

[drauschenbach]

I am using Notebook with --gateway-url.

[jarmitage]

Also experiencing this with --KernelGatewayApp.api='kernel_gateway.notebook_http'

It's as if a time bomb went off in the code somewhere. – @drauschenbach

Does have this feeling!

jupyter --version:

jupyter core     : 4.6.3
jupyter-notebook : 6.1.1
qtconsole        : not installed
ipython          : 7.17.0
ipykernel        : 5.3.4
jupyter client   : 6.1.6
jupyter lab      : 2.2.4
nbconvert        : 5.6.1
ipywidgets       : 7.5.1
nbformat         : 5.0.7
traitlets        : 4.3.3

[kevin-bates]

Thanks for the update. The command is jupyter --version.

I get this when attempting:

[W 200810 08:02:33 web:1786] 404 POST /api/kernels (127.0.0.1): Kernel does not exist: <coroutine object MappingKernelManager.start_kernel at 0x7fc51f21c3c0>
[W 200810 08:02:33 web:2246] 404 POST /api/kernels (127.0.0.1) 1.57ms
/opt/anaconda3/envs/kernel_gateway-2.4.1/lib/python3.8/asyncio/base_events.py:1844: RuntimeWarning: coroutine 'MappingKernelManager.start_kernel' was never awaited
  handle = self._ready.popleft()

And I believe this may be related to NB 6.1. Are you sure you're running NB 6.0.3? Please confirm.

I'll spend time with this this morning (PDT).

[drauschenbach]

Ah, OK thanks here's my corrected version info:

$ jupyter --version
jupyter core     : 4.6.3
jupyter-notebook : 6.1.1
qtconsole        : not installed
ipython          : 7.16.1
ipykernel        : 5.3.4
jupyter client   : 6.1.6
jupyter lab      : not installed
nbconvert        : 5.6.1
ipywidgets       : not installed
nbformat         : 5.0.7
traitlets        : 4.3.3

[kevin-bates]

Please try to downgrade notebook to 6.0.3 (pip install --upgrade notebook==6.0.3) in the environment in which JKG is running. I suspect you can use that as work around for now.

[jarmitage]

Thanks for looking into it, @kevin-bates.

How do you do the same downgrade in conda? Or would pip work there too?

Got it: conda install notebook=6.0.3

Can confirm downgrading works.

[kevin-bates]

I think 6.1 is the timebomb, but JKG is where the change is required - although I'm not sure what that is. Enterprise Gateway has been using the 6.1 rc but it also enables the async kernel management support - which I think I should do with JKG anyway.

[kevin-bates]

I have a fix. Will try to get JKG 2.4.2 out today.

I'm going to hold off switching to the async kernel management since this is a patch. Perhaps we can enable it in 2.5 or 3.0 if others are interested. It essentially gives you the ability to startup multiple kernels "simultaneously" instead of having to wait for the first to start before the second can. Since JKG is used with local kernels and they tend to startup in a second or two, this isn't that critical. With remote kernels spawned by Enterprise Gateway, the startup times can be 10-20 seconds sometimes so kernel startup can become a bottleneck.