Add notebooks to docs download #652

AoifeHughes · 2025-09-29T20:32:10Z

This was to try and fix #642 as I was afraid of a caching issue, please see that PR for previous discussions

Co-authored-by: Penelope Yong <penelopeysm@gmail.com>

…into add_notebook_dl

…_notebook_dl

github-actions · 2025-09-30T12:39:29Z

Preview the changes: https://turinglang.org/docs/pr-previews/652
Please avoid using the search feature and navigation bar in PR previews!

Copilot

Pull Request Overview

This PR adds notebook generation and download functionality to the documentation site by creating Jupyter notebooks from Quarto .qmd files and providing download links in the rendered HTML pages.

Introduces a Python script to convert .qmd files to .ipynb format with proper cell structure
Adds shell scripts to generate notebooks and inject download links into HTML pages
Integrates notebook generation into both publish and preview GitHub workflows

Reviewed Changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
assets/scripts/qmd_to_ipynb.py	Python converter that parses .qmd files and creates Jupyter notebooks with appropriate cell types
assets/scripts/generate_notebooks.sh	Shell script that finds .qmd files and converts them to notebooks in the _site directory
assets/scripts/add_notebook_links.sh	Shell script that adds download links to HTML pages using Perl regex replacement
.github/workflows/publish.yml	Adds notebook generation and link injection steps to the publish workflow
.github/workflows/preview.yml	Adds notebook generation and link injection steps to the preview workflow

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-01T08:27:09Z

assets/scripts/qmd_to_ipynb.py

+                    "display_name": "Julia 1.11",
+                    "language": "julia",
+                    "name": self.kernel_name


The hardcoded display name 'Julia 1.11' doesn't match the dynamic kernel name when using Python. Consider making the display name dynamic based on the kernel.

Copilot · 2025-10-01T08:27:09Z

assets/scripts/qmd_to_ipynb.py

+                "language_info": {
+                    "file_extension": ".jl",
+                    "mimetype": "application/julia",
+                    "name": "julia",
+                    "version": "1.11.0"
+                }


The language_info is hardcoded for Julia even when the kernel is Python. This should be conditional based on the detected engine/kernel type.

Copilot · 2025-10-01T08:27:10Z

assets/scripts/add_notebook_links.sh

+            # Insert the notebook link AFTER the "Report an issue" link
+            # This ensures it goes in the right place in the sidebar toc-actions
+            # The download="index.ipynb" attribute forces browser to download instead of navigate
+            perl -i -pe 's/(<a href="[^"]*issues\/new"[^>]*><i class="bi[^"]*"><\/i>Report an issue<\/a><\/li>)/$1<li><a href="index.ipynb" class="toc-action" download="index.ipynb"><i class="bi bi-journal-code"><\/i>Download notebook<\/a><\/li>/g' "$html_file"


This complex regex replacement is brittle and hard to maintain. Consider using a more robust HTML parsing approach or breaking this into multiple steps for better readability.

Suggested change

perl -i -pe 's/(<a href="[^"]*issues\/new"[^>]*><i class="bi[^"]*"><\/i>Report an issue<\/a><\/li>)/$1<li><a href="index.ipynb" class="toc-action" download="index.ipynb"><i class="bi bi-journal-code"><\/i>Download notebook<\/a><\/li>/g' "$html_file"

python3 - <<EOF

import sys

from bs4 import BeautifulSoup

html_file = "$html_file"

with open(html_file, "r", encoding="utf-8") as f:

soup = BeautifulSoup(f, "html.parser")

# Find the "Report an issue" link

report_link = None

for a in soup.find_all("a"):

if a.get_text(strip=True) == "Report an issue":

report_link = a

break

if report_link is not None:

# Insert the new link after the parent <li> of the report_link

parent_li = report_link.find_parent("li")

if parent_li is not None and parent_li.parent is not None:

new_li = soup.new_tag("li")

new_a = soup.new_tag("a", href="index.ipynb", **{"class": "toc-action", "download": "index.ipynb"})

new_i = soup.new_tag("i", **{"class": "bi bi-journal-code"})

new_a.append(new_i)

new_a.append("Download notebook")

new_li.append(new_a)

parent_li.insert_after(new_li)

with open(html_file, "w", encoding="utf-8") as f:

f.write(str(soup))

EOF

Copilot · 2025-10-01T08:27:10Z

assets/scripts/generate_notebooks.sh

+echo "Generating Jupyter notebooks from .qmd files..."
+
+# Find all .qmd files in tutorials, usage, and developers directories
+find tutorials usage developers -name "index.qmd" | while read qmd_file; do


Using 'while read' with 'find' can fail with filenames containing spaces or special characters. Use 'find ... -print0 | while IFS= read -r -d "" qmd_file; do' for safer file handling.

Suggested change

find tutorials usage developers -name "index.qmd" | while read qmd_file; do

find tutorials usage developers -name "index.qmd" -print0 | while IFS= read -r -d '' qmd_file; do

AoifeHughes · 2025-10-01T08:32:52Z

This PR adds a download button to get jupyter notebooks from the docs.

Some notes:

quarto doesn't like either using convert or the _quarto.yml config options to make notebooks as well as html
To avoid running each file 2x in the deployment I created a few scripts, the main of which is https://github.com/TuringLang/docs/pull/652/files#diff-fa790f7c53fe8322e42e5baf383fa2350e43f4f279c3ee25e4aeac1d4680f3a4 which handles the conversion and builds the notebooks
The CI has been updated to perform this
I would have much rather used quarto's built in solutions, but they're just not fit for purpose in this instance (please see the dozens of commits in which I tried every solution I could find before raising them as possible solutions @mhauru can attest to the numerous attempts I made in the last week).

@shravanngoswamii if you'd have some time to check these out, I'd appreciate it.

AoifeHughes · 2025-10-01T08:33:54Z

I should also mention, I had a few issues with the CI versions of bash/posix vs deploying on my mac, so some of the scripting is done to appease github actions moreso than how users may run locally.

AoifeHughes · 2025-10-01T08:36:18Z

I think this partially satisfies some of the points from #641 ~ it might be worth adding some note about these being available in the docs / have a quick way to export into online systems, as discussed in that issue.

Aoife and others added 29 commits September 18, 2025 13:31

Add download button for Jupyter notebooks in tutorial pages

54f23e3

Refactor Quarto rendering and remove download notebook script

ff55d5a

Merge branch 'main' into add_notebook_dl

8dd8fbf

Update to Turing 0.40

091e1a1

Update version in _quarto.yml

fc4df94

Fix DPPL 0.37 run_ad change

df1f9cd

Fix VI tutorial

14973ad

Fix model-manual's use of contexts

06bb33d

Fix references to __context__

e7519c9

Fix use of addlogprob for log prior

4530247

Fix typo

57c3162

Regenerate manifest

2065a14

Remove version pin of DelayDiffEq and update Manifest

2babb2d

Fix call to evaluate

b41a4cb

Add note about contexts tutorial being out of date

425539d

Apply suggestions from code review

dde0ed3

Co-authored-by: Penelope Yong <penelopeysm@gmail.com>

Add ipynb format support

a32ec1b

maybe added links to dl?

1e7f351

Merge branch 'add_notebook_dl' of https://github.com/TuringLang/docs …

e455881

…into add_notebook_dl

reset mainifest

d1bacac

dont execute ipynb

a78fac4

bump local changes

98bf999

Merge branch 'main' into add_notebook_dl

60a07a8

add post-render script for converting .quarto_ipynb files to .ipynb

af7ceb4

Merge branch 'add_notebook_dl' of https://github.com/TuringLang/docs …

4459016

…into add_notebook_dl

Refactor notebook generation process and clean up tutorial metadata

7f00453

Merge branch 'main' into add_notebook_dl

708d22b

please work :(

64c62a4

Merge branch 'add_notebook_dl' of github.com:TuringLang/docs into add…

1c6fad9

…_notebook_dl

AoifeHughes self-assigned this Sep 29, 2025

AoifeHughes changed the title ~~Add notebooks to docs downloasd~~ Add notebooks to docs download Sep 30, 2025

Aoife added 4 commits September 30, 2025 09:51

freeze?

26cdf37

last try

01e6954

forgot to add links...

f55de5c

grep...

848dc87

Aoife and others added 5 commits September 30, 2025 13:43

fixed edit loc

fdbee40

placement

2a56d63

added a script to do notebooks justice

9ed76f1

posix

27b9d8b

Merge branch 'main' into add_notebook_dl

e06a333

AoifeHughes marked this pull request as ready for review October 1, 2025 08:25

AoifeHughes requested review from shravanngoswamii and Copilot October 1, 2025 08:26

Copilot AI reviewed Oct 1, 2025

View reviewed changes

AoifeHughes requested a review from penelopeysm October 1, 2025 08:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add notebooks to docs download #652

Add notebooks to docs download #652

AoifeHughes commented Sep 29, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Oct 1, 2025

Uh oh!

Copilot AI Oct 1, 2025

Uh oh!

Copilot AI Oct 1, 2025

Uh oh!

Copilot AI Oct 1, 2025

Uh oh!

AoifeHughes commented Oct 1, 2025

Uh oh!

AoifeHughes commented Oct 1, 2025

Uh oh!

AoifeHughes commented Oct 1, 2025

Uh oh!

Uh oh!

-            perl -i -pe 's/(<a href="[^"]*issues\/new"[^>]*><i class="bi[^"]*"><\/i>Report an issue<\/a><\/li>)/$1<li><a href="index.ipynb" class="toc-action" download="index.ipynb"><i class="bi bi-journal-code"><\/i>Download notebook<\/a><\/li>/g' "$html_file"
+            python3 - <<EOF
+import sys
+from bs4 import BeautifulSoup
+html_file = "$html_file"
+with open(html_file, "r", encoding="utf-8") as f:
+    soup = BeautifulSoup(f, "html.parser")
+# Find the "Report an issue" link
+report_link = None
+for a in soup.find_all("a"):
+    if a.get_text(strip=True) == "Report an issue":
+        report_link = a
+        break
+if report_link is not None:
+    # Insert the new link after the parent <li> of the report_link
+    parent_li = report_link.find_parent("li")
+    if parent_li is not None and parent_li.parent is not None:
+        new_li = soup.new_tag("li")
+        new_a = soup.new_tag("a", href="index.ipynb", **{"class": "toc-action", "download": "index.ipynb"})
+        new_i = soup.new_tag("i", **{"class": "bi bi-journal-code"})
+        new_a.append(new_i)
+        new_a.append("Download notebook")
+        new_li.append(new_a)
+        parent_li.insert_after(new_li)
+        with open(html_file, "w", encoding="utf-8") as f:
+            f.write(str(soup))
+EOF

	find tutorials usage developers -name "index.qmd" \| while read qmd_file; do
	find tutorials usage developers -name "index.qmd" -print0 \| while IFS= read -r -d '' qmd_file; do

Add notebooks to docs download #652

Are you sure you want to change the base?

Add notebooks to docs download #652

Conversation

AoifeHughes commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

AoifeHughes commented Oct 1, 2025

Uh oh!

AoifeHughes commented Oct 1, 2025

Uh oh!

AoifeHughes commented Oct 1, 2025

Uh oh!

Uh oh!

AoifeHughes commented Sep 29, 2025 •

edited

Loading