Interest in a lightweight, Jupyter-free Python engine extension?

[jhelvy]

Description

Hi all, I'd like to gauge interest (and check for prior art) before building something.

Main issue

Running Python in Quarto requires (I think) going through Jupyter, which brings in all the issues associated with jupyter. It seems a good deal of Python setup trouble I see in the issues traces back to needing Jupyter rather than to Quarto itself (e.g. #1366, #11804, #5140). The alternative, engine: knitr with reticulate, works well, but it requires an R installation for a pure-Python document.

Idea

Should we (I?) build a python engine extension on the new 1.9 engine API that:

• Runs chunks in a persistent subprocess of whatever Python interpreter you point it at (with no Jupyter, no kernelspecs and no extra packages to install)
• Keeps state across chunks, captures stdout/stderr/errors, and respects Quarto's standard chunk options (echo, fig-width, output: asis, etc.)
• Handles rich output through the standard _repr_mimebundle_ / _repr_html_ / _repr_latex_ protocols, with format-aware output for HTML, PDF and Typst
• Gives first-class support to the more modern stack (polars, GreatTables, plotnine) alongside pandas, matplotlib and seaborn, and lets users register their own display adapters

Questions

1. Has anyone built / started something like this?
2. Would you use it? What would it need to support for you to switch?
3. Is there anything about the engine extension API, or about freeze/caching and inline code, that would make this harder than it looks?
4. Will TypeScript engine extensions carry over to Quarto 2, or would a Q2 version need a different approach?

I just want to use Quarto with Python chunks (and no Jupyter) so it feels like how it feels to use R chunks with knitr, so that's kind of what I'm going for here.

[mcanouil]

1. As far as I know, there are only two "engine extensions":
   • https://github.com/jimjam-slam/q2-ojs (https://github.com/orgs/quarto-dev/discussions/14877#discussioncomment-19210375)
   • https://github.com/PumasAI/quarto-julia-engine
   • https://github.com/mcanouil/quarto-typst-render (branch not published)
2. I would say, possibly but hard to say as it means quite a lot of work for graphics/tables, etc.
3. You'll have to discover those pain points unfortunately, but the Julia engine extension seems to work well, so I would bet it should be mostly ok.
4. I don't have the exact answer to that as there are several things under active development in that area.

Note that Jupyter is to Python what knitr is to R, but with some difference of behaviours which can be seen in how Quarto displays some computations.

[jhelvy]

Thanks, great feedback.

Note that Jupyter is to Python what knitr is to R

Is that a proper comparison though? Because Jupyter comes with a whole bunch of other stuff and wasn't purpose-built for use with Quarto, whereas my understanding of knitr is that it was built for RMarkdown (and eventually Quarto). Jupyter has to start it's own kernel in a quarto doc, which feels totally different.

[mcanouil]

That's not a one to one comparison no.

There are several tweaks made to knitr on its side and Quarto side which was made easy because there is a shared maintainer on both projects which is not the case for Jupyter.

Note that Jupyter could be used also for R, there are several kernel including "ark" which is also developed by Posit.
There are discussions about providing better support for it as an alternative to knitr. The main benefit is that it's one protocol (Jupyter) which allows more consistent computing behaviour.
There are different paths possible and since most of them were not taken by anyone (yet), hard to have a thorough comparison.

The above is one of the reason behind the engine extension feature, i.e., ease the development of engine by community to fit their needs on whatever they want.
The current main downside in Q1 is "1 document = 1 engine"¹, but with Q2, this limitation might go away which would mean computations could happen in several engines (no more R/reticulate/python/rpy2).

Footnotes

1. this is the main reason the public facing API of Typst Render is a filter and not the extension engine. ↩