sagemathgh-37083: HTML documentation: Show preparsed doctests using i…

…nline tabs

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->
Every doctest block is decorated with tabs, by default showing the
original doctest in **Sage** syntax, and offering a preparsed, pure
**Python** version of it. The tabs are synchronized across the page.

If @kwankyu's live documentation is being built, it is offered as
another tab **Sage (live)**. When this tab is selected, it automatically
starts the Thebe/Binder; it is not necessary to find and push the "Make
live" button.

![tabs-anim](https://github.com/sagemath/sage/assets/8345221/4252ae92-
eeb7-417a-ba24-c141726f714d)

[Preview](https://deploy-preview-37083--sagemath.netlify.app/html/en/ref
erence/function_fields/sage/rings/function_field/function_field_rational
)

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
Fixes sagemath#35791
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
- Depends on sagemath#37056
- Depends on sagemath#37065

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#37083
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Matthias Köppe, Tobias Diez

.ci/merge-fixes.sh

@@ -42,6 +42,7 @@ for REPO in ${SAGE_CI_FIXES_FROM_REPOSITORIES:-sagemath/sage}; do
                     # Considered alternative: Use https://github.com/$REPO/pull/$a.diff,
                     # which squashes everything into one diff without commit metadata.
                     PULL_URL="https://github.com/$REPO/pull/$a"
+                    PULL_SHORT="$REPO#$a"
                     PULL_FILE="$REPO_FILE-$a"
                     PATH=build/bin:$PATH build/bin/sage-download-file --quiet "$PULL_URL.patch" $PULL_FILE.patch
                     date -u +"%Y-%m-%dT%H:%M:%SZ" > $PULL_FILE.date  # Record the date, for future reference
@@ -67,7 +68,7 @@ for REPO in ${SAGE_CI_FIXES_FROM_REPOSITORIES:-sagemath/sage}; do
                         git am --signoff --show-current-patch=diff
                         echo "--------------------------------------------------------------------8<-----------------------------"
                         echo "::endgroup::"
-                        echo "Failure applying $PULL_URL as a patch, resetting"
+                        echo "Failure applying $PULL_SHORT as a patch, resetting"
                         git am --signoff --abort
                     fi
                 done

.github/workflows/build.yml

@@ -35,13 +35,13 @@ jobs:
         uses: actions/checkout@v4
       - name: Merge CI fixes from sagemath/sage
         run: |
-          .ci/merge-fixes.sh
+          mkdir -p upstream
+          .ci/merge-fixes.sh 2>&1 | tee upstream/ci_fixes.log
         env:
           GH_TOKEN: ${{ github.token }}
           SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}
       - name: Store CI fixes in upstream artifact
         run: |
-          mkdir -p upstream
           if git format-patch --stdout test_base > ci_fixes.patch; then
             cp ci_fixes.patch upstream/
           fi

.github/workflows/doc-build-pdf.yml

@@ -29,13 +29,13 @@ jobs:
         uses: actions/checkout@v4
       - name: Merge CI fixes from sagemath/sage
         run: |
-          .ci/merge-fixes.sh
+          mkdir -p upstream
+          .ci/merge-fixes.sh 2>&1 | tee upstream/ci_fixes.log
         env:
           GH_TOKEN: ${{ github.token }}
           SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}
       - name: Store CI fixes in upstream artifact
         run: |
-          mkdir -p upstream
           if git format-patch --stdout test_base > ci_fixes.patch; then
             cp ci_fixes.patch upstream/
           fi

.github/workflows/doc-build.yml

@@ -24,13 +24,13 @@ jobs:
         uses: actions/checkout@v4
       - name: Merge CI fixes from sagemath/sage
         run: |
-          .ci/merge-fixes.sh
+          mkdir -p upstream
+          .ci/merge-fixes.sh 2>&1 | tee upstream/ci_fixes.log
         env:
           GH_TOKEN: ${{ github.token }}
           SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}
       - name: Store CI fixes in upstream artifact
         run: |
-          mkdir -p upstream
           if git format-patch --stdout test_base > ci_fixes.patch; then
             cp ci_fixes.patch upstream/
           fi

src/doc/common/static/custom-jupyter-sphinx.css

@@ -1,5 +1,5 @@
 div.jupyter_container {
-  margin: .5rem 0;
+  border: 0;
 }
 
 div.jupyter_container + div.jupyter_container {

src/doc/common/static/jupyter-sphinx-furo.js

@@ -112,3 +112,16 @@ thebelab.on("status", function (evt, data) {
     kernel.requestExecute({code: "%display latex"});
   }
 });
+
+// Activate Thebe when "Sage (live)" tab is clicked
+document.querySelectorAll('input[class="tab-input"]').forEach((elem) => {
+    elem.addEventListener("click", function(event) {
+        if (elem.nextElementSibling) {
+            if (elem.nextElementSibling.nextElementSibling) {
+                if (elem.nextElementSibling.nextElementSibling.querySelector('div[class="thebelab-code"]')) {
+                    initThebelab();
+                }
+            }
+        }
+    });
+});

src/doc/en/installation/source.rst

@@ -940,11 +940,19 @@ Environment variables controlling the documentation build
 
   The value of this variable is passed as an
   argument to ``sage --docbuild all html`` or ``sage --docbuild all pdf`` when
-  you run ``make``, ``make doc``, or ``make doc-pdf``.  For example, you can
-  add ``--no-plot`` to this variable to avoid building the graphics coming from
-  the ``.. PLOT`` directive within the documentation, or you can add
-  ``--include-tests-blocks`` to include all "TESTS" blocks in the reference
-  manual. Run ``sage --docbuild help`` to see the full list of options.
+  you run ``make``, ``make doc``, or ``make doc-pdf``.  For example:
+
+  - add ``--no-plot`` to this variable to avoid building the graphics coming from
+    the ``.. PLOT`` directive within the documentation,
+
+  - add ``--no-preparsed-examples`` to only show the original Sage code of
+    "EXAMPLES" blocks, suppressing the tab with the preparsed, plain Python
+    version, or
+
+  - add ``--include-tests-blocks`` to include all "TESTS" blocks in the reference
+    manual.
+
+  Run ``sage --docbuild help`` to see the full list of options.
 
 .. envvar:: SAGE_SPKG_INSTALL_DOCS
 

src/sage_docbuild/__main__.py

@@ -289,6 +289,9 @@ def setup_parser():
     standard.add_argument("--no-plot", dest="no_plot",
                           action="store_true",
                           help="do not include graphics auto-generated using the '.. plot' markup")
+    standard.add_argument("--no-preparsed-examples", dest="no_preparsed_examples",
+                          action="store_true",
+                          help="do not show preparsed versions of EXAMPLES blocks")
     standard.add_argument("--include-tests-blocks", dest="skip_tests", default=True,
                           action="store_false",
                           help="include TESTS blocks in the reference manual")
@@ -478,6 +481,8 @@ def excepthook(*exc_info):
         build_options.ALLSPHINXOPTS += "-n "
     if args.no_plot:
         os.environ['SAGE_SKIP_PLOT_DIRECTIVE'] = 'yes'
+    if args.no_preparsed_examples:
+        os.environ['SAGE_PREPARSED_DOC'] = 'no'
     if args.live_doc:
         os.environ['SAGE_LIVE_DOC'] = 'yes'
     if args.skip_tests:

src/sage_docbuild/conf.py

@@ -39,6 +39,9 @@
 # General configuration
 # ---------------------
 
+SAGE_LIVE_DOC = os.environ.get('SAGE_LIVE_DOC', 'no')
+SAGE_PREPARSED_DOC = os.environ.get('SAGE_PREPARSED_DOC', 'yes')
+
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = [
@@ -57,7 +60,7 @@
 
 jupyter_execute_default_kernel = 'sagemath'
 
-if os.environ.get('SAGE_LIVE_DOC', 'no') == 'yes':
+if SAGE_LIVE_DOC == 'yes':
     SAGE_JUPYTER_SERVER = os.environ.get('SAGE_JUPYTER_SERVER', 'binder')
     if SAGE_JUPYTER_SERVER.startswith('binder'):
         # format: "binder" or
@@ -230,7 +233,7 @@ def sphinx_plot(graphics, **kwds):
 # console lexers. 'ipycon' is the IPython console, which is what we want
 # for most code blocks: anything with "sage:" prompts. For other IPython,
 # like blocks which might appear in a notebook cell, use 'ipython'.
-highlighting.lexers['ipycon'] = IPythonConsoleLexer(in1_regex=r'sage: ', in2_regex=r'[.][.][.][.]: ')
+highlighting.lexers['ipycon'] = IPythonConsoleLexer(in1_regex=r'(sage:|>>>)', in2_regex=r'([.][.][.][.]:|[.][.][.])')
 highlighting.lexers['ipython'] = IPyLexer()
 highlight_language = 'ipycon'
 
@@ -305,7 +308,7 @@ def set_intersphinx_mappings(app, config):
 multidocs_is_master = True
 
 # https://sphinx-copybutton.readthedocs.io/en/latest/use.html
-copybutton_prompt_text = r"sage: |[.][.][.][.]: |\$ "
+copybutton_prompt_text = r"sage: |[.][.][.][.]: |>>> |[.][.][.] |\$ "
 copybutton_prompt_is_regexp = True
 copybutton_exclude = '.linenos, .c1'  # exclude single comments (in particular, # optional!)
 copybutton_only_copy_prompt_lines = True
@@ -789,8 +792,6 @@ class will be properly documented inside its surrounding class.
     return skip
 
 
-from jupyter_sphinx.ast import JupyterCellNode, CellInputNode
-
 class SagecodeTransform(SphinxTransform):
     """
     Transform a code block to a live code block enabled by jupyter-sphinx.
@@ -828,29 +829,87 @@ def apply(self):
         if self.app.builder.tags.has('html') or self.app.builder.tags.has('inventory'):
             for node in self.document.traverse(nodes.literal_block):
                 if node.get('language') is None and node.astext().startswith('sage:'):
-                    source = node.rawsource
-                    lines = []
-                    for line in source.splitlines():
-                        newline = line.lstrip()
-                        if newline.startswith('sage: ') or newline.startswith('....: '):
-                            lines.append(newline[6:])
-                    cell_node = JupyterCellNode(
-                                execute=False,
-                                hide_code=True,
-                                hide_output=True,
-                                emphasize_lines=[],
-                                raises=False,
-                                stderr=True,
-                                code_below=False,
-                                classes=["jupyter_cell"])
-                    cell_input = CellInputNode(classes=['cell_input','live-doc'])
-                    cell_input += nodes.literal_block(
-                        text='\n'.join(lines),
-                        linenos=False,
-                        linenostart=1)
-                    cell_node += cell_input
-
-                    node.parent.insert(node.parent.index(node) + 1, cell_node)
+                    from docutils.nodes import container as Container, label as Label, literal_block as LiteralBlock, Text
+                    from sphinx_inline_tabs._impl import TabContainer
+                    parent = node.parent
+                    index = parent.index(node)
+                    if isinstance(node.previous_sibling(), TabContainer):
+                        # Make sure not to merge inline tabs for adjacent literal blocks
+                        parent.insert(index, Text(''))
+                        index += 1
+                    parent.remove(node)
+                    # Tab for Sage code
+                    container = TabContainer("", type="tab", new_set=False)
+                    textnodes = [Text('Sage')]
+                    label = Label("", "", *textnodes)
+                    container += label
+                    content = Container("", is_div=True, classes=["tab-content"])
+                    content += node
+                    container += content
+                    parent.insert(index, container)
+                    if SAGE_PREPARSED_DOC == 'yes':
+                        # Tab for preparsed version
+                        from sage.repl.preparse import preparse
+                        container = TabContainer("", type="tab", new_set=False)
+                        textnodes = [Text('Python')]
+                        label = Label("", "", *textnodes)
+                        container += label
+                        content = Container("", is_div=True, classes=["tab-content"])
+                        example_lines = []
+                        preparsed_lines = ['>>> from sage.all import *']
+                        for line in node.rawsource.splitlines() + ['']:  # one extra to process last example
+                            newline = line.lstrip()
+                            if newline.startswith('....: '):
+                                example_lines.append(newline[6:])
+                            else:
+                                if example_lines:
+                                    preparsed_example = preparse('\n'.join(example_lines))
+                                    prompt = '>>> '
+                                    for preparsed_line in preparsed_example.splitlines():
+                                        preparsed_lines.append(prompt + preparsed_line)
+                                        prompt = '... '
+                                    example_lines = []
+                                if newline.startswith('sage: '):
+                                    example_lines.append(newline[6:])
+                                else:
+                                    preparsed_lines.append(line)
+                        preparsed = '\n'.join(preparsed_lines)
+                        preparsed_node = LiteralBlock(preparsed, preparsed, language='ipycon')
+                        content += preparsed_node
+                        container += content
+                        parent.insert(index + 1, container)
+                    if SAGE_LIVE_DOC == 'yes':
+                        # Tab for Jupyter-sphinx cell
+                        from jupyter_sphinx.ast import JupyterCellNode, CellInputNode
+                        source = node.rawsource
+                        lines = []
+                        for line in source.splitlines():
+                            newline = line.lstrip()
+                            if newline.startswith('sage: ') or newline.startswith('....: '):
+                                lines.append(newline[6:])
+                        cell_node = JupyterCellNode(
+                                    execute=False,
+                                    hide_code=False,
+                                    hide_output=True,
+                                    emphasize_lines=[],
+                                    raises=False,
+                                    stderr=True,
+                                    code_below=False,
+                                    classes=["jupyter_cell"])
+                        cell_input = CellInputNode(classes=['cell_input','live-doc'])
+                        cell_input += nodes.literal_block(
+                            text='\n'.join(lines),
+                            linenos=False,
+                            linenostart=1)
+                        cell_node += cell_input
+                        container = TabContainer("", type="tab", new_set=False)
+                        textnodes = [Text('Sage Live')]
+                        label = Label("", "", *textnodes)
+                        container += label
+                        content = Container("", is_div=True, classes=["tab-content"])
+                        content += cell_node
+                        container += content
+                        parent.insert(index + 1, container)
 
 
 # This replaces the setup() in sage.misc.sagedoc_conf
@@ -864,7 +923,7 @@ def setup(app):
         app.connect('autodoc-process-docstring', skip_TESTS_block)
     app.connect('autodoc-skip-member', skip_member)
     app.add_transform(SagemathTransform)
-    if os.environ.get('SAGE_LIVE_DOC', 'no') == 'yes':
+    if SAGE_LIVE_DOC == 'yes' or SAGE_PREPARSED_DOC == 'yes':
         app.add_transform(SagecodeTransform)
 
     # When building the standard docs, app.srcdir is set to SAGE_DOC_SRC +