🔀 MERGE: Add inline execution and variable evaluation (#414)

.pre-commit-config.yaml

@@ -50,6 +50,7 @@ repos:
       - importlib_metadata
       - myst-parser~=0.17.2
       - "sphinx~=4.3.2"
+      - nbclient
       - types-PyYAML
       files: >
         (?x)^(

CHANGELOG.md

@@ -1,5 +1,12 @@
 # Change Log
 
+## v0.15.0 - 2022-05-05
+
+[Full changelog](https://github.com/executablebooks/MyST-NB/compare/v0.14.0...v0.15.0)
+
+✨ NEW: Add `inline` execution mode and `eval` role/directive, for inserting code variables directly into the text flow of your documentation!
+See [Inline variable evaluation](docs/render/inline.md) for more information.
+
 ## v0.14.0 - 2022-04-27
 
 [Full changelog](https://github.com/executablebooks/MyST-NB/compare/v0.13.2...v0.14.0)

docs/_static/custom.css

@@ -0,0 +1,15 @@
+/** Add a counter before subsections **/
+h1 {
+    counter-reset: subsection;
+}
+h2 {
+    counter-reset: subsubsection;
+}
+h2::before {
+    counter-increment: subsection;
+    content: counter(subsection) ". ";
+}
+h3::before {
+    counter-increment: subsubsection;
+    content: counter(subsection) "." counter(subsubsection) ". ";
+}

docs/authoring/jupyter-notebooks.md

@@ -162,7 +162,7 @@ fig, ax = plt.subplots()
 ax.scatter(*data, c=data[2])
 ```
 
-### Raw cells
+## Raw cells
 
 The [raw cell type](https://nbformat.readthedocs.io/en/latest/format_description.html#raw-nbconvert-cells) can be used to specifically render the content as a specific [MIME media type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types).
 

docs/computation/execute.md

@@ -13,12 +13,20 @@ Notebooks can either be run each time the documentation is built, or cached loca
 Execution and caching behaviour is controlled with configuration at a global or per-file level, as outlined in the [configuration section](config/intro).
 See the sections below for a description of each configuration option and their effect.
 
-(execute/config)=
+(execute/modes)=
 
-## Triggering notebook execution
+## Notebook execution modes
 
 To trigger the execution of notebook pages, use the global `nb_execution_mode` configuration key, or per-file `execution_mode` key:
 
+|   Mode   |                             Description                              |
+| -------- | -------------------------------------------------------------------- |
+| `off`    | Do not execute the notebook                                          |
+| `force`  | Always execute the notebook (before parsing)                         |
+| `auto`   | Execute notebooks with missing outputs (before parsing)              |
+| `cache`  | Execute notebook and store/retrieve outputs from a cache             |
+| `inline` | Execute the notebook during parsing (allows for variable evaluation) |
+
 By default this is set to:
 
 ```python
@@ -28,27 +36,39 @@ nb_execution_mode = "auto"
 This will only execute notebooks that are missing at least one output.
 If a notebook has *all* of its outputs populated, then it will not be executed.
 
-**To force the execution of all notebooks, regardless of their outputs**, change the above configuration value to:
+To force the execution of all notebooks, regardless of their outputs, change the above configuration value to:
 
 ```python
 nb_execution_mode = "force"
 ```
 
-**To cache execution outputs with [jupyter-cache]**, change the above configuration value to:
+To cache execution outputs with [jupyter-cache], change the above configuration value to:
 
 ```python
 nb_execution_mode = "cache"
 ```
 
 See {ref}`execute/cache` for more information.
 
-**To turn off notebook execution**, change the above configuration value to:
+To execute notebooks inline during parsing, change the above configuration value to:
+
+```python
+nb_execution_mode = "inline"
+```
+
+This allows for the use of `eval` roles/directives to embed variables, evaluated from the execution kernel, inline of the Markdown.
+
+See {ref}`render/eval` for more information.
+
+To turn off notebook execution, change the above configuration value to:
 
 ```python
 nb_execution_mode = "off"
 ```
 
-**To exclude certain file patterns from execution**, use the following configuration:
+## Exclude notebooks from execution
+
+To exclude certain file patterns from execution, use the following configuration:
 
 ```python
 nb_execution_excludepatterns = ['list', 'of', '*patterns']
@@ -124,7 +144,7 @@ By default, the command working directory (cwd) that a notebook runs in will be
 However, you can set `nb_execution_in_temp=True` in your `conf.py`, to change this behaviour such that, for each execution, a temporary directory will be created and used as the cwd.
 
 (execute/timeout)=
-## Execution Timeout
+## Execution timeout
 
 The execution of notebooks is managed by {doc}`nbclient <nbclient:client>`.
 
@@ -184,7 +204,7 @@ print(thisvariabledoesntexist)
 ```
 
 (execute/raise_on_error)=
-## Error Reporting: Warning vs. Failure
+## Error reporting: Warning vs. Failure
 
 When an error occurs in a context where `nb_execution_allow_errors=False`,
 the default behaviour is for this to be reported as a warning.

docs/conf.py

@@ -122,6 +122,7 @@
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
+html_css_files = ["custom.css"]
 
 copybutton_selector = "div:not(.output) > div.highlight pre"
 

docs/index.md

@@ -72,9 +72,9 @@ Cache execution outputs, for fast re-builds.
 :link: render/code-cells
 :link-type: ref
 
-Convert Jupyter execution outputs to embedded content.\
-Insert outputs as variables into your documents.\
-Build single or collections of documents into multiple formats, including HTML websites and PDF books.
+Convert Jupyter execution outputs to rich embedded content.\
+Insert computed variables within the document flow.\
+Build single or collections of documents into multiple formats (HTML, PDF, ...).
 
 +++
 [Learn more »](render/code-cells)

docs/reference/api.rst

@@ -8,11 +8,10 @@ The parsing of a notebook consists of a number of stages, with each stage separa
 1. The configuration is set (from a file or CLI)
 2. The parser is called with an input string and source
 3. The parser reads the input string to a notebook node
-4. The notebook code outputs are potentially updated, via execution or from a cache
-5. The notebook is "pre-processed" in-place (e.g. to coalesce output streams and extract glue outputs)
-6. The notebook is converted to a Markdown-It tokens syntax tree
-7. The syntax tree is transformed to a docutils document AST (calling the renderer plugin)
-8. The docutils document is processed by docutils/sphinx, to create the desired output format(s)
+4. The notebook is converted to a Markdown-It tokens syntax tree
+5. The notebook code outputs are potentially updated, via execution or from a cache
+6. The syntax tree is transformed to a docutils document AST (calling the renderer plugin)
+7. The docutils document is processed by docutils/sphinx, to create the desired output format(s)
 
 Configuration
 -------------
@@ -44,15 +43,23 @@ Read
 Execute
 -------
 
-.. autoclass:: myst_nb.core.execute.ExecutionResult
+.. autofunction:: myst_nb.core.execute.create_client
+
+.. autoclass:: myst_nb.core.execute.base.NotebookClientBase
     :members:
 
-.. autofunction:: myst_nb.core.execute.execute_notebook
+.. autoclass:: myst_nb.core.execute.direct.NotebookClientDirect
+
+.. autoclass:: myst_nb.core.execute.cache.NotebookClientCache
+
+.. autoclass:: myst_nb.core.execute.inline.NotebookClientInline
 
-Pre-process
------------
+.. autoexception:: myst_nb.core.execute.base.ExecutionError
 
-.. autofunction:: myst_nb.core.preprocess.preprocess_notebook
+.. autoexception:: myst_nb.core.execute.base.EvalNameError
+
+.. autoclass:: myst_nb.core.execute.base.ExecutionResult
+    :members:
 
 Render plugin
 -------------