quarto only daemonizes `julia` and `python` kernels

[allefeld]

Bug description

I have developed a custom kernel for Matlab based on ipykernel. The kernel works well for my purposes, appears to be used by a few others, I haven't changed it in months, and I use it on a regular basis for Quarto documents.

Recently I noticed that processing of Quarto documents using the kernel is very slow, even for very simple computations, and then figured out that this is due to the kernel no longer being daemonized. Every time the document is processed, the kernel and thereby Matlab is started new.

Steps to reproduce

This assumes that Matlab is installed.

• install the kernel from https://github.com/allefeld/mkernel

• if installed in an environment (e.g. conda), activate the environment

• create a file test.qmd with the contents

---
jupyter: mkernel
---

```{matlab}
2 + 2
```

• run quarto preview test.qmd

• edit and save test.qmd

Expected behavior

After the first start of quarto preview, the kernel should be started, but on subsequent renders triggered by editing and saving the file, Quarto should re-use the kernel.

Actual behavior

The kernel is shut down after each render and restarted on the next.

$ quarto preview test.qmd

Starting mkernel kernel...Done

Executing 'test.quarto_ipynb'
  Cell 1/1: ''...Done

– snip – 

Output created: test.html

Watching files for changes
Browse at http://localhost:6681/
Opening in existing browser session.

Starting mkernel kernel...Done

Executing 'test.quarto_ipynb'
  Cell 1/1: ''...Done

– snip –
  
Output created: test.html

Watching files for changes

I checked the logs of my kernel, and indeed it receives a shutdown request, e.g.

Control received: {'header': {'msg_id': '6297427c-daf6f0379e7b84702c45a4c4_36565_0', 'msg_type': 'shutdown_request', 'username': 'ca', 'session': '6297427c-daf6f0379e7b84702c45a4c4', 'date': datetime.datetime(2024, 3, 15, 21, 40, 2, 470311, tzinfo=tzutc()), 'version': '5.3'}, 'msg_id': '6297427c-daf6f0379e7b84702c45a4c4_36565_0', 'msg_type': 'shutdown_request', 'parent_header': {}, 'metadata': {}, 'content': {'restart': False}, 'buffers': []}

To check whether this is a general problem with Jupyter kernels, in test.qmd I replaced mkernel with python3 and matlab with python, and the Python kernel is daemonized as expected.

To see what the difference between handling of the python3 kernel and my mkernel kernel is, I used the option --execute-debug.

Full output of quarto preview test.qmd --execute-debug, including modifying test.qmd once.

- Connecting to kernel

Starting mkernel kernel...2024-03-15 22:25:05,381 - root - Level 25 - starting notebook server subprocess
2024-03-15 22:25:05,381 - root - Level 25 - starting notebook server subprocess
2024-03-15 22:25:05,642 - root - Level 25 - running notebook server subprocess
2024-03-15 22:25:05,642 - root - Level 25 - creating notebook server (tcp: /run/user/1000/jt/3c0b0e66a9a6f7a52edd)
2024-03-15 22:25:05,643 - root - Level 25 - notebook server bound to port 36549
- Kernel connection successful
2024-03-15 22:25:05,723 - root - Level 25 - handling server request
- Sending execute command to kernel
- Execute command sent, reading response
2024-03-15 22:25:05,725 - root - Level 25 - executing notebook
2024-03-15 22:25:05,725 - root - Level 25 - inside notebook_execute
2024-03-15 22:25:05,760 - root - Level 25 - read notebook
2024-03-15 22:25:05,761 - root - Level 25 - not using cache
2024-03-15 22:25:05,761 - root - Level 25 - Will attempt to create notebook
2024-03-15 22:25:05,761 - root - Level 25 - type of notebook: <class 'nbformat.notebooknode.NotebookNode'>
2024-03-15 22:25:05,761 - root - Level 25 - Creating NotebookClient
2024-03-15 22:25:14,554 - root - Level 25 - NotebookClient created
2024-03-15 22:25:14,555 - root - Level 25 - Executing cell 0
2024-03-15 22:25:14,555 - root - Level 25 - Executed cell 0
2024-03-15 22:25:14,555 - root - Level 25 - Executing cell 1
Done

Executing 'test.quarto_ipynb'
  Cell 1/1: ''...2024-03-15 22:25:21,150 - root - Level 25 - Executed cell 1
2024-03-15 22:25:21,150 - root - Level 25 - Done
2024-03-15 22:25:21,150 - root - Level 25 - Notebook execution complete
2024-03-15 22:25:21,152 - root - Level 25 - notebook not persistable (exiting server)
2024-03-15 22:25:21,152 - root - Level 25 - cleaning up server resources
2024-03-15 22:25:21,152 - root - Level 25 - exiting server
Done

- Server request complete


pandoc 
  to: html
  output-file: test.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  
metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  jupyter: mkernel
  
Output created: test.html

Watching files for changes
Browse at http://localhost:6310/
Opening in existing browser session.
- Connecting to kernel

Starting mkernel kernel...2024-03-15 22:25:48,426 - root - Level 25 - starting notebook server subprocess
2024-03-15 22:25:48,426 - root - Level 25 - starting notebook server subprocess
2024-03-15 22:25:48,722 - root - Level 25 - running notebook server subprocess
2024-03-15 22:25:48,722 - root - Level 25 - creating notebook server (tcp: /run/user/1000/jt/3c0b0e66a9a6f7a52edd)
2024-03-15 22:25:48,722 - root - Level 25 - notebook server bound to port 46199
- Kernel connection successful
- Sending execute command to kernel
2024-03-15 22:25:48,771 - root - Level 25 - handling server request
- Execute command sent, reading response
2024-03-15 22:25:48,772 - root - Level 25 - executing notebook
2024-03-15 22:25:48,772 - root - Level 25 - inside notebook_execute
2024-03-15 22:25:48,815 - root - Level 25 - read notebook
2024-03-15 22:25:48,816 - root - Level 25 - not using cache
2024-03-15 22:25:48,816 - root - Level 25 - Will attempt to create notebook
2024-03-15 22:25:48,816 - root - Level 25 - type of notebook: <class 'nbformat.notebooknode.NotebookNode'>
2024-03-15 22:25:48,816 - root - Level 25 - Creating NotebookClient
2024-03-15 22:25:57,862 - root - Level 25 - NotebookClient created
Done
2024-03-15 22:25:57,863 - root - Level 25 - Executing cell 0
2024-03-15 22:25:57,864 - root - Level 25 - Executed cell 0
2024-03-15 22:25:57,864 - root - Level 25 - Executing cell 1

Executing 'test.quarto_ipynb'
  Cell 1/1: ''...2024-03-15 22:26:04,547 - root - Level 25 - Executed cell 1
2024-03-15 22:26:04,547 - root - Level 25 - Done
2024-03-15 22:26:04,547 - root - Level 25 - Notebook execution complete
Done
2024-03-15 22:26:04,548 - root - Level 25 - notebook not persistable (exiting server)

2024-03-15 22:26:04,548 - root - Level 25 - cleaning up server resources
- Server request complete


pandoc 
  to: html
  output-file: test.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  
metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  jupyter: mkernel
  
Output created: test.html

Watching files for changes

The difference is that with mkernel compared to python3, there are additional lines between Notebook execution complete and - Server request complete:

2024-03-15 22:25:21,152 - root - Level 25 - notebook not persistable (exiting server)
2024-03-15 22:25:21,152 - root - Level 25 - cleaning up server resources
2024-03-15 22:25:21,152 - root - Level 25 - exiting server
Done

The important bit seems to be notebook not persistable (exiting server), but unfortunately a web search turned up nothing.

Your environment

• Debian 12.5

Quarto check output

Quarto 1.5.25
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.1.11: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.41.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.5.25
      Path: /opt/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: (not installed)
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: Installation From Path
      Path: /usr/bin
      Version: 2022

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.10.12 (Conda)
      Path: /home/ca/Store/micromamba/envs/std/bin/python
      Jupyter: 5.3.1
      Kernels: python3, mkernel

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........OK
      Version: 4.3.3
      Path: /usr/lib/R
      LibPaths:
        - /home/ca/Store/R/4.3
        - /usr/local/lib/R/site-library
        - /usr/lib/R/site-library
        - /usr/lib/R/library
      knitr: 1.45
      rmarkdown: 2.25

[✓] Checking Knitr engine render......OK

[allefeld]

Since reproduction as described above needs Matlab, I checked whether the same occurs with another ipykernel-based kernel, and it does. New reproduction steps:

• install echo_kernel from https://github.com/jupyter/echo_kernel

  In my experience, the installation instructions are incomplete. I installed via pip into a conda environment, but the kernel was not recognized – I had to additionally run install.py in the package installation directory, python3 <...>/site-packages/echo_kernel/install.py

• if installed in an environment (e.g. conda), activate the environment

• create a file test.qmd with the contents

---
jupyter: echo
---

```{echo}
2 + 2
```

• run quarto preview test.qmd

• edit and save test.qmd

The kernel is started new on every modification of the file. With --execute-debug, the message notebook not persistable (exiting server) appears just like for my kernel.

[cscheid]

Quarto used to be too aggressive in keeping daemons running in the past and that caused problems with users expecting different jupyter versions, etc. Quarto is now more conservative about it, and that is probably causing your ipykernel to not be persisted.

The logic behind daemonization and persistence is in https://github.com/quarto-dev/quarto-cli/blob/main/src/resources/jupyter/notebook.py#L45-L302 specifically around the field kernel_deps and the exception RestartKernel.

[allefeld]

Thanks! Though I'm not sure what should have changed between executions. As I said, it's just reprocessing of the same Quarto document, the kernel implementation and kernel specification have not changed at all.

I am also not sure about the explanation. So before executing the notebook, the kernel is restarted if kernel_deps has changed. But in the log, the message notebook not persistable (exiting server) comes after the notebook has been executed. I therefore suspect it has more to do with the return value persist of notebook_execute.

This is determined as hasattr(notebook_execute, "kernel_deps") (l. 301), which in turn apparently depends on the value of nb_kernel_dependencies (l. 290-294), which returns something if there is at least one cell output with output.name == 'stdout' and output.output_type == 'stream'. I don't understand what cell outputs have to do with persistability? The comment at l. 288-289 says "picks up imports that occurred witihn the notebook cells", which seems Python-specific – my kernel is based on Python, but doesn't execute Python, and echo_kernel doesn't execute anything.

In summary, I don't see how I can translate what the code tells me into a concrete action. Is there something I should change either in the kernel or in using Quarto to work around this?

[cscheid]

In summary, I don't see how I can translate what the code tells me into a concrete action.

I don't know it either. I mentioned the file because we can't predict ahead of time how arbitrary third-party kernels are going to behave, and we almost exclusively test against the python and julia kernels.

I don't understand what cell outputs have to do with persistability? The comment at l. 288-289 says "picks up imports that occurred witihn the notebook cells", which seems Python-specific – my kernel is based on Python, but doesn't execute Python, and echo_kernel doesn't execute anything.

The kernel is persisted when there's a truthy value of nb_kernel_depenencies (sic); the output check happens only for the "cleanup" cell specifically. If there's no cleanup cell then we don't execute the block inside if cleanup_cell: which likely causes notebook.kernel_deps to be None (which is falsy). If the cleanup cell doesn't produce output, then notebook.kernel_deps is {} which is also falsy.

(Yes, this should be properly documented.)

Is there something I should change either in the kernel or in using Quarto to work around this?

Your kernel could emit a cell that's recognized by nb_cleanup_cell so that cell is executed and produces output.

EDIT: Actually, I think this isn't possible to do given the check in line 361, and will require changes on our side.

[cscheid]

I would like non-julia, non-python kernels to have access to the setup and cleanup cells that we provide to julia and python without Quarto having to interfere. I think that's the correct feature for us to implement, and then your kernel can do anything it wants to, and then it could also respond to the Quarto environment as we hard-code for python and julia kernels.

[allefeld]

I looked at the code a bit more and inserted some hacky debug messages. As expected, with my kernel, the function attribute kernel_deps is never set, so persist is always False.

But, I'm not sure the logic here is correct. It makes sense to restart the kernel if the kernel dependencies have changed. But I don't think it is correct behavior to not persist if there are no kernel dependencies to begin with. That just means possible changes can't be detected, it doesn't mean changes have definitely occurred. In this case it would be better to keep persistence and let the user specify daemon: false if they want.

For the moment, I resorted to a hacky solution: Change l. 301 to persistent = True. As far as I can tell, that shouldn't lead to problems because the restart logic is still present.

I don't think I can contribute directly to the necessary changes in Quarto, but if there's something I can do to debug a possible solution, let me know.

[cscheid]

Yeah, I really don't think there's anything you can do at the moment, which is why I changed the issue title. I'm hopeful we can make it into something that custom kernels can declare in their kernelspecs.

[cscheid]

On further thought, you could attempt to use notebook caching; if caching is enabled and succeeds, the kernel will automatically be persisted: https://github.com/quarto-dev/quarto-cli/blob/main/src/resources/jupyter/notebook.py#L154