quarto render failing to close jupyter kernel in `pystan2` code?

[betanalpha]

Bug description

When trying to render a quarto notebook with Python code from the command line I'm seeing behavior consistent with a persistent jupyter kernel. I would expect that each call to quarto render would run in a new, clean kernel.

For example consider the file test.qmd with the text

---
title: "Test"
author: "Test"
jupyter: python3
---

Test test test.

```{python}
import multiprocessing
multiprocessing.set_start_method("fork")
```

If I run

quarto render test.qmd --to html

from the command line once then everything renders fine, but I do see some Python threads that persist even after the command line prompt has returned.

If I run the same command again then quarto fails with the error

Executing 'test.ipynb'
  Cell 1/1...ERROR: 

An error occurred while executing the following cell:
------------------
import multiprocessing
multiprocessing.set_start_method("fork")
------------------

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [3], in <cell line: 2>()
      1 import multiprocessing
----> 2 multiprocessing.set_start_method("fork")

File /usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py:243, in DefaultContext.set_start_method(self, method, force)
    241 def set_start_method(self, method, force=False):
    242     if self._actual_context is not None and not force:
--> 243         raise RuntimeError('context has already been set')
    244     if method is None and force:
    245         self._actual_context = None

RuntimeError: context has already been set
RuntimeError: context has already been set

Force quitting those dangling Python threads does allow me to render again without problem.

If I include a cell like

```{python}
print(locals())
a = 2
print(locals())
```

then I also see the locals() dictionary growing from call to call, even holding variables from previous renders that are no longer in the notebook, consistent with the kernel persisting from call to call.

Operating System: MacOS 10.15.7 (19H2026)
Python Version: Python 3.9.6 (default, Jun 29 2021, 06:20:32) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Quarto Version: 1.1.251

Any help would be appreciated, and apologies if I'm missing something obvious. Many thanks for a great project.

Checklist

• Please include a minimal, fully reproducible example in a single .qmd file? Please provide the whole file rather than the snippet you believe is causing the issue.
• Please format your issue so it is easier for us to read the bug report.
• Please document the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
• Please document the operating system you're running. If on Linux, please provide the specific distribution.

[dragonstyle]

By default for Jupyter, we do run a daemon for the kernel (to address startup performance). More about this, including how you can manage the daemon here:

https://quarto.org/docs/computations/python.html#kernel-daemon

HTH!

[betanalpha]

Trying to turn the daemon off or giving it a time out, either from the command line or within the notebook itself, didn't work; instead 18 Python threads spawned and the rendering froze at the Starting python3 kernel step. On the other hand --execute-daemon-restart did the trick and allowed rendering of multiple --to targets at once.

Thanks for the prompt and informative response!

[cscheid]

On the other hand --execute-daemon-restart did the trick and allowed rendering of multiple --to targets at once.

Just to confirm, you're seeing that adding

execute:
  daemon: false

didn't work for you? If that's the case, this is a bug we should fix...

[betanalpha]

Note that I updated to quarto 1.2.269 just in case I might have been hitting an old issue.

If I reduce everything to simple examples, for instance

---
title: "Test"
author: "Test"
jupyter: python3
execute:
  daemon: false
---

Hello.

```{python}
print("Hello")
```

or

---
title: "Test"
author: "Test"
jupyter: python3
execute:
  daemon: false
---

Hello.

```{python}
import multiprocessing
multiprocessing.set_start_method("fork")
```

then the daemons appear to behave correctly. Removing the execute configuration from the YAML header but adding --no-execute-daemon to the command line call also seems to work correctly.

Unfortunately the more complicated notebook of my interest, https://github.com/betanalpha/mcmc_diagnostics/blob/main/pystan2/mcmc_diagnostics_pystan2.qmd, exhibits the failure mode I mentioned above. When executing quarto render mcmc_diagnostics_pystan2.qmd --to html,pdf --no-execute-daemon the output freezes at

Starting python3 kernel...

while about 18 Python threads are spun up and sit idle. I tried using the same YAML header with the bodies of the simpler test notebooks and everything worked fine; I haven't been able to figure out useful elements of the body to isolate given that the execution fails before any code blocks are evaluated.

The command quarto render mcmc_diagnostics_pystan2.qmd --to html,pdf --execute-daemon-restart does work, which is sufficient for my immediate needs and I'm happy to assume blame for the awkward behavior on problems with my code. I am happy, however, to try anything that might help isolate the awkward behavior above if there is interest, especially given the nontrivial dependences in the notebook.

Thanks, again and apologies for any confusion.

[cscheid]

Thank you for the added context! I'm probably only going to be able to get to this issue in a few weeks, but I'll follow up with you when I do (it might have to do with details of how pystan spawns subprocesses.)

[cscheid]

@betanalpha we recently made a (small) change on how we spawn subprocesses in the jupyter engine, and there's a small chance it fixes this issue. Would you mind trying again on https://github.com/quarto-dev/quarto-cli/releases/tag/v1.3.191 or later?

[betanalpha]

Apologies for the delay!

I downloaded v1.3.191 and gave it try. The results are a bit different but now it seems like neither approach works.

Running quarto render mcmc_diagnostics_pystan2.qmd --to html,pdf without execute: daemon: false in the YAML renders the HTML but then fails with a RuntimeError: context has already been set when trying to render the PDF as the daemon where the multiprocessing environment has already been configured is reused.

Previously running quarto render mcmc_diagnostics_pystan2.qmd --to html,pdf --no-execute-daemon without execute: daemon: false worked as expected but running quarto render mcmc_diagnostics_pystan2.qmd --to html,pdf with execute: daemon: false froze after a Starting python3 kernel... message while many Python processes spawned in the background.

Now running quarto render mcmc_diagnostics_pystan2.qmd --to html,pdf --no-execute-daemon without execute: daemon: false and running quarto render mcmc_diagnostics_pystan2.qmd --to html,pdf with execute: daemon: false give consistent results, but unfortunately not the expected results. Both freeze after the Starting python3 kernel... message and this time no Python processes seem to spawn in either case.

Happy to try on a more recent tag if that might help.

Thanks!