Allow code execution to generate rich outputs that are incorporated into the AST

[choldgraf]

Background

Code-cells can produce rich output like Markdown, LaTeX, and tables. Right now, these outputs are only rendered at display time, meaning that each static export and web build needs to interpret the MIME bundle outputs. As a result, the output content cannot participate in a MyST build, i.e. the Markdown or LaTeX output cannot generate or consume referencing labels.

We should parse this output along with the rest of the Markdown in the document. This would allow for programmatic generation of content, and along with the {embed} directive could allow users to nicely stitch together MyST content via any Jupyter Kernel.

Personas & Stories

Personas and stories for computation in MyST

An arbitrary persona wants to publish an online textbook

• These notebooks may be executable.
• They want to have reliable reproducible builds in CI.
• They may want to pre-execute in a particular compute environment.

An instructor publishes their courseware

• Their courseware is predominantly in notebooks with some markdown content.
• They have worksheets that are executable, and students are expected to fill in the blanks with their solutions.
• Instructors have reference solutions and want to be able to validate them e.g. in CI
• Reference solutions should/may be visible to students to copy-paste and/or otherwise pull in.
• Instructors may prefer students to remain inside the MyST site to do this.

A community maintainer creates a dashboard with MyST

• Usage statistics of their GitHub Repo are compiled by a notebook.
• The code-cells are run at build time to analyse data and generate content:
  • Tables; Pandas, Markdown
  • Arbitrary content
  • Figures: interactive plots and widgets
• The resulting document is pushed to the web as an HTML build.

An Executive Director creates a monthly report

• Financial data for the last month is scraped using an executable MyST document.
• Some Markdown content outside of the code cells includes {eval} roles which utilise the data computed in other code-cells.
• The code-cells are run at build time to analyse data and generate content:
  • Figures (plots)
  • Tables; Pandas, Markdown
  • Equations (TeX-math, Typst?)
  • Arbitrary content
• The resulting document is published to a PDF via Typst.

A scientist publishes a reproducible paper

• They have some (maybe hidden) notebooks generating interesting outputs like figures and tables.
• The notebooks are run at build time to ensure a reproducible build
• They either write their main manuscript in Markdown or TeX embedding:
  • Notebook outputs as figures to tables.
  • Code cells and programs.
• OR, they write the paper in a top-to-bottom notebook(s).

A scientist publishes a computational article

• The authors have included a mix of static and interactive figures, all of which are generated by notebooks.
• The notebooks are run at build time to ensure a reproducible build.
• The code-cells analyse data and generate interesting figures (plots, tables) that will be interactive.
• They include interactive figures for data exploration, visualisation etc.
  • Some outputs are useable as static representations of the interactive cells (static plots)
  • Other outputs require manual placeholders (widgets)
• They either write their main manuscript in Markdown:
  • Notebook outputs as figures to tables.
  • Code cells and programs.
• OR, they write the paper in a top-to-bottom notebook(s)

Definition of done

• It is possible to generate Markdown in a code-cell, and see it in a PDF build
• The generated Markdown can define and consume reference targets that are visible to the rest of the project

Related issues

• Generate MyST AST from cell execution #1633 is about generating AST directly rather than text/markdown
• Special characters are escaped in table cells #1555
• IPython display(Markdown(f"###Sample text {something to interpolate}")) not rendering correctly jupyter-book#2585

[HelgeGehring]

continuing #1559 here:

An example for this would be the following:

• Right now I have a nice notebook which analyzes exactly one physical structure.
• The analysis results in several graphs / tables / images / ..., each generated by a code cell

My problem now is, that I have 2-20 different structures. For each I get graphs based on the same code.
Right now the only solution is to put them all below each other, leading to a quite long doc.

Better would be to have

• tabs in which I can select which structure I want to see

and/or

• a table which has only the main results (maybe broken down in a number or two), but on click on a row, I see a detailed analysis for that structure.

A simplified example might be statistics about bank accounts (not scientific, but less abstract).

• It's simple to write a notebook which produces graphs / tables, ... based on an account number
• Now I have 5 accounts and want this one notebook to do the same analysis with all of them. An overview would be the balance of each one in a table. But I'd like to be able to switch tabs/expand rows of the different account to see the detailed graphs.
• Alternatively, I'd like to have automatically one notebook per account. As the code is the same, I don't wanna copy/paste

For different use-cases different approaches might be best. I'm wondering what might be possible:

• have on the complete notebook level tabs, i.e. execute the notebook like github actions matrix with different parameters
• have tabs within each output (which might be syncronized when switching (?))
• Having outputs combined with hidden content, e.g. a table in which i can expand individual rows to see more details.

[choldgraf]

Workaround: Write to temporary text files and then use {include}

I discovered this workaround today, it's a bit cludgy but isn't too hacky. It takes advantage of the fact that code cells are executed before a page is parsed by MyST. This means that you can do something like the following:

## Generate content with Jupyter

```{code-cell} python
from pathlib import Path
p = Path("../_build/txt/tmp.txt")
p.parent.mkdir(parents=True, exist_ok=True)
_ = p.write_text("- **Testing**\n- Testing two\n- Testing three")
```

And then include it in the page with MyST markdown like so:

```{include} ../_build/txt/tmp.txt
```

This will:

1. Generate some MyST Markdown in Jupyter
2. Write it to a .txt file
3. And later in the page, in MyST MD, we reference that .txt file with an {include} statement

So the file is first executed, the txt file is created, and MyST then includes it

[agoose77]

@choldgraf we should also make it possible to include files with .myst.json extensions, to support pulling in AST.

[choldgraf]

Yeah I was thinking that too. It also made me wonder if the notebook cell MyST support could be more like the plugin structure, rather than making it language specific.

For example, a cell tag like output-myst or output-myst-ast that would tell MyST to parse stdout as MyST (similar to what executable plugins do).

Does that make sense? If so I can update the issue body to reflect that suggestion.

[agoose77]

@choldgraf the natural way to do this would be to define a MIME type for MyST AST, and recognise it in our transforms!

[choldgraf]

That sounds like a good idea. Though if it were the only way, then each kernel would need to have a package that outputs MyST right? The benefit of tags and stdout is that anybody in any language could use it without developing anything specific.

[agoose77]

Although we don't need a package for this (certainly with ipython, you can just use display), I'm curious to understand your thought process - are you picturing a user printing stringified json to the stdout? Or myst markup, I.e text/markdown?

[choldgraf]

My idea was inspired by the way that you handled "black box" outputs for the executable plugins infrastructure. So below I'll share a Python and an R cell that would generate either MyST MD or MyST AST. At build time, if the tag was identified on a cell, then any text/plain output (or however stdout gets logged) would be parsed differently. Without those tags, the output would just be parsed like any other stdout.

Below I'll use Print, but I think since Jupyter "returns" the result of the final executed line, it may not be necessary. It'd also cool if this worked with variables too, so that you could insert generated MyST elsewhere.

Python - MyST MD

```{code-cell} python
:tags: output-myst-md
print("- **Bolded** list item")
```

Python - MyST AST¹

```{code-cell} python
:tags: output-myst-ast
ast = {
      "type": "list",
      "ordered": false,
      "spread": false,
      "children": [
        {
          "type": "listItem",
          "spread": True,
          "children": [
            {
              "type": "strong",
              "children": [
                {
                  "type": "text",
                  "value": "Bolded"
                }
              ]
            },
            {
              "type": "text",
              "value": " list item"
            }
          ]
        }
      ]
    }
print(ast)
```

Footnotes

1. This is also a good demonstration of why I think it's way nicer to be able to parse MyST MD directly and not only MyST AST :-) ↩