Support myst-nb tags and settings for executable code cell block

[GYHHAHA]

Context

Currently, the executed result directly ignores the myst-nb tags and settings. It would be great if we bring these features into the mystjs rendering process, especially for tags like hide-*/skip-execution/raises-exception/remove-* (* can be input/output/cell).

Proposal

No response

Tasks and updates

No response

[rowanc1]

@fwkoch do we have this tagged somewhere in the mystjs repo as well? I remember talking about it!

[davidtrem]

Yes taking into account the myst-nb cell tags like hide-input, remove-cell... would be really great 👍
It's a roadblock for me today :-(
Is there any way around it for the time being?

Update: I just come to realize my question is most probably more related to mystjs
jupyter-book/mystmd#160

[GYHHAHA]

@fwkoch do we have this tagged somewhere in the mystjs repo as well? I remember talking about it!

After a relatively long search, I don't think we currently have this feature. However, I basically and gradually clear about how to implement this. From my perspective ,

• we can add a transform function here to bring the meta tags down to the nodes with code-cell and output type
  https://github.com/executablebooks/mystjs/blob/89a541f41079adb9a5ac8630352e12777ae664f9/packages/myst-transforms/src/blocks.ts#L32
• then add the new transform at here
  https://github.com/executablebooks/mystjs/blob/89a541f41079adb9a5ac8630352e12777ae664f9/packages/myst-cli/src/process/mdast.ts#L145
• finally get the meta data by Props in code component and by JupyterOutput param in output component
  https://github.com/executablebooks/myst-theme/blob/e95834a58ae08d3b2a381854aa8fb0d4725ad8d6/packages/myst-to-react/src/code.tsx#L34
  https://github.com/executablebooks/myst-theme/blob/e95834a58ae08d3b2a381854aa8fb0d4725ad8d6/packages/jupyter/src/output.tsx#L76
  If we are all satisfied with this, I'll do the implementation this week then pr. @rowanc1 @fwkoch

[fwkoch]

Hey @GYHHAHA - thanks for the issue! Finally spent some time chewing on this, and I think you are on the right track. In particular, it's important to identify most of the work needs to go into myst-cli/myst-transforms: this should take all the input tags, etc, and boil them down to simple properties on mdast code and output nodes. Then, in myst-to-react/myst-theme the frontend mapping should be pretty straightforward, with all the info we need already present on the separate code/output nodes.

I'm pretty sure that's basically parroting your previous comment?

---

Getting slightly more into the weeds:

We want to support jupyterbook-style metadata tags. We also probably want to support code cell options provided inline, i.e.:

#| hide-code: true

print('hello world')

These are already nicely consolidated into block.data:

• metadata tags (as you found) are added here: https://github.com/executablebooks/mystjs/blob/89a541f41079adb9a5ac8630352e12777ae664f9/packages/myst-transforms/src/blocks.ts#L38 - after being pulled here: https://github.com/executablebooks/mystjs/blob/89a541f41079adb9a5ac8630352e12777ae664f9/packages/myst-cli/src/process/notebook.ts#L26
• code cell options are lifted out here: https://github.com/executablebooks/mystjs/blob/89a541f41079adb9a5ac8630352e12777ae664f9/packages/myst-cli/src/transforms/code.ts#L82

This means to propagate this info down to the code/output nodes, we would just need an additional transform that traverses each block, resolves the various fields from tags (e.g. tags: ['remove-code'] resolving to remove-code: true), then finds the code/output nodes it contains, and updates those nodes accordingly.

What does it mean to "update those nodes accordingly" though...? I suspect we will need to update the node specs to include additional fields to contain this info. The code type is here: https://github.com/executablebooks/mystjs/blob/89a541f41079adb9a5ac8630352e12777ae664f9/packages/myst-spec-ext/src/types.ts#L71 and the output type isn't actually defined anywhere, I don't think... it's just a flexible GenericNode type everywhere.

What do these new fields look like? It doesn't make a ton of sense to keep the full name, I don't think... for example, maybe hide-code just becomes a hide field on code nodes, remove-output becomes remove on output nodes, maybe raises-exception stays raises-exception on code nodes? Does that make sense, or is it confusing that we are changing tags that folks are already familiar with to new names...?

---

One additional small thought: We may want to venture outside notebooks and add this as an option to the code-block directive, i.e.:

```{code-block} python
:raises-exception:

print('hello world')
```

However, this would probably require implementing on the code-block directive, i.e. here: https://github.com/executablebooks/mystjs/blob/89a541f41079adb9a5ac8630352e12777ae664f9/packages/myst-directives/src/code.ts#L51 Pretty straightforward (especially once we have decided on the new code node fields), but probably outside the initial scope of this work. Really what we care about supporting is the existing jupyterbook-style tags.

---

Anyway! Quite keen to see your implementation if you are willing to make a PR! Happy to help out along the way however you need.

[GYHHAHA]

Thanks for your informative response! It's very clear. Since myst-cli and myst-theme are separated repos, so I will split into 2 prs, one for transformer stuff + type spec, and one for react rendering.

Personally speaking, I vote on keeping the origin jupyter meta tags, both beneficial for new comer to verify or search and semantic ambiguity-free. Another reason is the jupyter output can be html/text/picture/... even std err or stream, user might want to remove-stderr as documented here, simple "output" will result to difficulty for extending.

For the raises-exception and skip-execution, I have verified that they actually have been working currently as long as you put them input the tags like the following example. The reason why this works could be it is controlled by run-time setting which is taken care by nbclient instead of rendering-time which should be processed by sphinx or mystjs. We maybe need not to do anything for this.

```{code-cell} ipython3
:tags: [raises-exception]
# ...
```

```{code-cell} ipython3
:tags: [skip-execution]
# ...
```

Thanks again for your help! @fwkoch

[rowanc1]

Excited for this, thanks @GYHHAHA and @fwkoch. :)

[GYHHAHA]

In order to make hide-cell valid, we should add block component in myst-to-react. However, I find here is another ContentBlock component in site package:
https://github.com/executablebooks/myst-theme/blob/ddf09f5ed5deba04d732aaed4246448d35ef0405/packages/site/src/components/ContentBlocks.tsx#L19
If I control the visibility and hidden-state style in the new block component, whether will it have breaking impact on the site package's old Block? If so, what's your suggestion to make them work? @fwkoch @rowanc1

--- update

I see this is a hidden div. Maybe is used just for debug?
https://github.com/executablebooks/myst-theme/blob/ddf09f5ed5deba04d732aaed4246448d35ef0405/packages/site/src/components/ContentBlocks.tsx#L46

[MatusGasparik]

This would be realy great. Many times I'd like to present the results from a live notebook (running as a webpage with myst start and with a local jupyterlab server) but to reduce the visual clutter from the code cells, I'd like to keep them hidden or collapsed.

Is there any roadmap for this?

[GYHHAHA]

Hi @MatusGasparik , the roadmap is in here. Unfortunately, I'm in busy days these weeks, I hope I can finish the rest of work in here at the middle of July and take a overall look on that with @rowanc1 .

[rowanc1]

Thanks @GYHHAHA and @MatusGasparik! Our attention has mostly been on getting the execution working directly in MyST, and I was a bit hesitant to do both of these at the same time. The execution work is starting to settle down now, and I think this is one of the things that we can turn to next! @GYHHAHA has got a great start on that already. 🚀

@MatusGasparik are you using any ipywidgets in your work by chance?

[MatusGasparik]

Hi @rowanc1. Not yet, or not directly. I’m also just getting started. But I will definitely be using some of the PyViz tools (HoloViews, Bokeh, …) and Pyvista for interactive visualization. The idea of having MyST based interactive webpages connected to a Jupyter server is beyond amazing!

[rowanc1]

@banesullivan is ideally going to be at scipy next week helping out with the MyST poster there! Very excited to be working with him a bit to support rich visualizations & making them fully interactive!

Will try to get to that hidden code/outputs soon!