Google Colab notebook using IJava stuck at "Connecting" after installation + page reload

[thomers]

All of my notebooks with the IJava kernel that worked before now are stuck at "Connecting" after the initial IJava installation and browser page refresh.

What was working previously / expected behavior

1. Execute this first cell

!wget https://github.com/SpencerPark/IJava/releases/download/v1.3.0/ijava-1.3.0.zip 
!unzip ijava-1.3.0.zip 
!python install.py --sys-prefix

2. Wait for the Installed java kernel message

3. Refresh the browser page.

4. Execute any cell with Java code and see the output.

Now what happens is

I can execute the first cell and get the Installed java kernel message, seeing the notebook status as "Connected".

But after refreshing the page, the status of the notebook is stuck at "Connecting" forever,

and thus no cells with Java code can be executed.

--

I'm using Google Colab for free, but since the initial installation still works, and the notebook status is "Connected" before the page is refreshed, I assume not being a paid subscriber should not be the problem.

Any idea what has been changed, and how I can get my Java notebooks to connect again?

Example
https://colab.research.google.com/github/vistec-AI/colab/blob/master/ijava.ipynb

[thomers]

Here's the question on SO with a bounty:

https://stackoverflow.com/questions/74674688/google-colab-notebook-using-ijava-stuck-at-connecting-after-installation-ref/74709951#74709951

[cperry-goog]

Shouldn't be an issue with running on the free of charge tier.

My guess is this broke with our Python 3.8 upgrade - do you have any way to verify? Looks like the repo you're referring to hasn't seen updates in a few years.

[SpencerPark]

Hello, author of that kernel here :) found my way over from the stack overflow question.

I was taking a peek into what is going on but I'm not too familiar with the colab infrastructure. Though I can verify that I can use the kernel within that workbook linked in that top comment, but cannot change the kernel type in colab. What I mean by that is I can run the following in a cell:

!jupyter console --kernel=java --simple-prompt

and the kernel starts up properly, jupyter client can connect, and I can execute code in that kernel.

Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop starting...
Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop started.
Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop starting...
Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop started.
Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop starting...
Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop started.
Jupyter console 6.1.0

Java 11.0.17+8-post-Ubuntu-1ubuntu218.04 :: IJava kernel 1.3.0 
Protocol v5.3 implementation by jupyter-jvm-basekernel 2.3.0
In [1]: System.out.println("test");
test

So there doesn't seem to be an issue with starting the kernel in the container via jupyter but for whatever reason via colab has trouble connecting to it.

Also tried switching via getKernel().runtimeManager.changeKernelType('<session-id>', 'java') and saw the same successful startup logs in /var/colab/app.log:

Timestamp	Level	Message
Dec 9, 2022, 2:55:23 AM	WARNING	INFO: Loop started.
Dec 9, 2022, 2:55:23 AM	WARNING	Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start
Dec 9, 2022, 2:55:23 AM	WARNING	INFO: Loop starting...
Dec 9, 2022, 2:55:23 AM	WARNING	Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start
Dec 9, 2022, 2:55:23 AM	WARNING	INFO: Loop started.
Dec 9, 2022, 2:55:23 AM	WARNING	Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start
Dec 9, 2022, 2:55:23 AM	WARNING	INFO: Loop starting...
Dec 9, 2022, 2:55:23 AM	WARNING	Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start
Dec 9, 2022, 2:55:23 AM	WARNING	INFO: Loop started.
Dec 9, 2022, 2:55:23 AM	WARNING	Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start
Dec 9, 2022, 2:55:23 AM	WARNING	INFO: Loop starting...
Dec 9, 2022, 2:55:23 AM	WARNING	Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start
Dec 9, 2022, 2:55:22 AM	INFO	Kernel shutdown: 25522ba0-5629-428b-a3a6-29827234a5d0
Dec 9, 2022, 2:55:22 AM	INFO	Kernel started: 495f9ece-6927-4283-884f-ffc29915f72c

There are no errors and I do see the kernel connection file exists at /root/.local/share/jupyter/runtime/kernel-495f9ece-6927-4283-884f-ffc29915f72c.json so I presume it is still running. Note that 25522ba0-5629-428b-a3a6-29827234a5d0 is the id for the default python3 kernel that got created, and in the logs it shows that as shutting down when switching to the java kernel.

Don't know if there is much else I can do to debug further on my end but happy to try anything if it will help!

[craigcitro]

@SpencerPark do you know if there's any python code invoked as part of ijava installation/setup?

In particular, this broke alongside our python 3.7 -> 3.8 transition, so I'm wondering if there's a chance some python code from ijava is getting into a bad state.

Relatedly: is there a way to get more logging from the ijava backend?

[SpencerPark]

There is some python code as part of the install but after that everything is implemented in java (i.e. the argv for the kernel is java ...). But I'm hesitant to believe the issue is there because I can use the jupyter commands in the colab container to find and successfully start the kernel in there.

Does colab have its own way of starting kernels or is it using the same jupyter package that is user accessible in the container?

Relatedly: is there a way to get more logging from the ijava backend?

There unfortunately isn't much logging apart from some messages during startup.

Another avenue I just tried investigating though is trying to connect to the kernel that colab was creating from another tab within the same runtime via:

!jupyter console --existing=/root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971.json --simple-prompt --debug

Output

[ZMQTerminalIPythonApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[ZMQTerminalIPythonApp] Looking for jupyter_config in /etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_config in /usr/local/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_config in /usr/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_config in /root/.local/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_config in /root/.jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /usr/local/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /usr/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /root/.local/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /root/.jupyter
[ZMQTerminalIPythonApp] Connecting to existing kernel: /root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971.json
[ZMQTerminalIPythonApp] Loading connection file /root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971.json
[ZMQTerminalIPythonApp] connecting iopub channel to ipc:///root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971-ipc-2
[ZMQTerminalIPythonApp] Connecting to: ipc:///root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971-ipc-2
[ZMQTerminalIPythonApp] connecting shell channel to ipc:///root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971-ipc-1
[ZMQTerminalIPythonApp] Connecting to: ipc:///root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971-ipc-1
[ZMQTerminalIPythonApp] connecting stdin channel to ipc:///root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971-ipc-3
[ZMQTerminalIPythonApp] Connecting to: ipc:///root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971-ipc-3
[ZMQTerminalIPythonApp] connecting heartbeat channel to ipc:///root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971-ipc-5
[ZMQTerminalIPythonApp] connecting control channel to ipc:///root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971-ipc-4
[ZMQTerminalIPythonApp] Connecting to: ipc:///root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971-ipc-4

which lead to an interesting discovery, the kernel was being started with "transport": "ipc" rather than "tcp" (which I knew was an option, but had never seen done and is not supported by jeromq) :

{
  "shell_port": 1,
  "iopub_port": 2,
  "stdin_port": 3,
  "control_port": 4,
  "hb_port": 5,
  "ip": "/root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971-ipc",
  "key": "",
  "transport": "ipc",
  "signature_scheme": "hmac-sha256",
  "kernel_name": ""
}

Starting via !jupyter console --kernel=java --simple-prompt --debug would use tcp in the connection file and is working fine.

Output

[ZMQTerminalIPythonApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[ZMQTerminalIPythonApp] Looking for jupyter_config in /etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_config in /usr/local/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_config in /usr/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_config in /root/.local/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_config in /root/.jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /usr/local/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /usr/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /root/.local/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /root/.jupyter
[ZMQTerminalIPythonApp] Connection File not found: /root/.local/share/jupyter/runtime/kernel-1734.json
[ZMQTerminalIPythonApp] Starting kernel: ['java', '-jar', '/usr/local/share/jupyter/kernels/java/ijava-1.3.0.jar', '/root/.local/share/jupyter/runtime/kernel-1734.json']
[ZMQTerminalIPythonApp] Connecting to: tcp://127.0.0.1:36409
[ZMQTerminalIPythonApp] connecting iopub channel to tcp://127.0.0.1:37369
[ZMQTerminalIPythonApp] Connecting to: tcp://127.0.0.1:37369
[ZMQTerminalIPythonApp] connecting shell channel to tcp://127.0.0.1:60365
[ZMQTerminalIPythonApp] Connecting to: tcp://127.0.0.1:60365
[ZMQTerminalIPythonApp] connecting stdin channel to tcp://127.0.0.1:42249
[ZMQTerminalIPythonApp] Connecting to: tcp://127.0.0.1:42249
[ZMQTerminalIPythonApp] connecting heartbeat channel to tcp://127.0.0.1:58877
[ZMQTerminalIPythonApp] connecting control channel to tcp://127.0.0.1:36409
[ZMQTerminalIPythonApp] Connecting to: tcp://127.0.0.1:36409

@craigcitro Do you know if this was another change in configuration on your side that rolled out with the 3.8 changes? And is there something we could specify in the kernel metadata to say that a specific kernel prefers tpc as the transport?

[thomers]

@cperry-goog and @craigcitro , any feedback or news from your side regarding

_Do you know if this was another change in configuration on your side that rolled out with the 3.8 changes?

And is there something we could specify in the kernel metadata to say that a specific kernel prefers tpc as the transport?_

[thomers]

Just for the record, SpencerPark created a workaround (thank you!) using a local proxy to forward the ipc to tcp channels, described here: https://stackoverflow.com/questions/74674688/google-colab-notebook-using-ijava-stuck-at-connecting-after-installation-ref