Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 20 additions & 18 deletions .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,44 +25,46 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v5
- name: Set up Ruby and install gems
uses: ruby/setup-ruby@v1
with:
ruby-version: '3.4'
bundler-cache: true
working-directory: ./docs
cache-version: 0 # Increment this number if you need to re-download cached gems
- name: Build with Jekyll
run: bundle exec jekyll build
working-directory: ./docs
- name: Set up Node.js
uses: actions/setup-node@v4
uses: actions/setup-node@v6
with:
node-version: '22'
cache: 'npm'
cache-dependency-path: package-lock.json
- name: Install Node.js dependencies
run: npm ci
cache-dependency-path: |
package-lock.json
builder/package-lock.json
- name: Install dependencies
run: |
npm ci
npm ci --prefix builder
- name: Build with tbdocs
run: node builder/index.mjs --src docs
- name: Check links (check_links.mjs)
# Three passes run in parallel via /sep/:
# 1. Online (_site/): --fallback-extensions html mirrors GitHub Pages'
# extensionless-URL behaviour. No --base-path needed -- this
# workflow builds without --baseurl (contrast jekyll-gh-pages.yml).
# 2. Offline (_site-offline/): strict, no extension fallback. --forbid
# catches any surviving https://docs.twinbasic.com/<path> link the
# offlinify rewrite missed (bare root URL is exempt).
# Integrity checks (--check-html/a11y/ids/sitemap/search) run here.
# 2. Offline (_site-offline/): --forbid catches any surviving
# https://docs.twinbasic.com/<path> link the offline rewrite missed
# (bare root URL is exempt). Integrity checks run here too (minus
# sitemap/search -- the offline tree has neither).
# 3. Book (_site-pdf/book.html): --no-fail makes failures informational
# (some links are not yet fully resolved).
run: >-
node scripts/check_links.mjs
--offline --include-fragments
--check-html --check-a11y --check-ids
--check-sitemap --check-search --check-canonical
--fallback-extensions html
--index-files 'index.html,.'
--root-dir docs/_site
docs/_site
/sep/
--offline --include-fragments
--index-files index.html
--check-html --check-a11y --check-ids
--index-files 'index.html,.'
--fallback-extensions html
--forbid 'https://docs.twinbasic.com'
--root-dir docs/_site-offline
docs/_site-offline
Expand Down
86 changes: 39 additions & 47 deletions .github/workflows/jekyll-gh-pages.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
name: Deploy Jekyll site to Pages
# Starting point: https://github.com/actions/starter-workflows/blob/main/pages/jekyll.yml
name: Build & deploy docs

on:
push:
Expand Down Expand Up @@ -35,63 +34,57 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v5
- name: Set up Ruby and install gems
uses: ruby/setup-ruby@v1
with:
ruby-version: '3.4'
bundler-cache: true
working-directory: ./docs
cache-version: 0 # Increment this number if you need to re-download cached gems
- name: Setup Pages
id: pages
uses: actions/configure-pages@v6
- name: Write secondary config if available
run: |
echo "${{ vars.JEKYLL_SITE_CONFIG }}" > _site_config.yml
echo "SITE_CONFIG=_site_config.yml" >> "$GITHUB_ENV"
if: ${{ vars.JEKYLL_SITE_CONFIG != '' }}
working-directory: ./docs
- name: Build with Jekyll
run: bundle exec jekyll build --config _config.yml,$SITE_CONFIG --baseurl "${{ steps.pages.outputs.base_path }}"
working-directory: ./docs
env:
JEKYLL_ENV: production
PAGES_REPO_NWO: "${{ github.repository }}"
- name: Set up Node.js
uses: actions/setup-node@v4
uses: actions/setup-node@v6
with:
node-version: '22'
cache: 'npm'
cache-dependency-path: package-lock.json
- name: Install Node.js dependencies and Chromium
# Install npm deps first so the link checks (check_links.mjs)
# can use htmlparser2. Chromium download runs in the same step
# so the cache hit / miss is one decision.
cache-dependency-path: |
package-lock.json
builder/package-lock.json
- name: Install dependencies and Chromium
run: |
npm ci
npm ci --prefix builder
sudo npx puppeteer browsers install chrome --install-deps
- name: Check links (check_links.mjs)
- name: Setup Pages
id: pages
uses: actions/configure-pages@v6
- name: Build with tbdocs
# --url + --baseurl together make canonical / og:url / sitemap
# entries point at the actual deployment (e.g. on a fork that
# deploys to kubao.github.io/twinBASIC-docs/, both bits move
# so the page advertises its own URL rather than the config'd
# production host).
run: node builder/index.mjs --src docs --url '${{ steps.pages.outputs.origin }}' --baseurl '${{ steps.pages.outputs.base_path }}'
- name: Check links and site integrity
# Three passes run in parallel via /sep/:
# 1. Online (_site/): --fallback-extensions html mirrors GitHub Pages'
# extensionless-URL behaviour. --base-path strips the Pages baseurl
# (e.g. `/twinBASIC-docs`) from absolute URLs before resolving.
# 2. Offline (_site-offline/): strict, no extension fallback. --forbid
# catches any surviving https://docs.twinbasic.com/<path> link the
# offlinify rewrite missed (bare root URL is exempt).
# Integrity checks (--check-html/a11y/ids/sitemap/search) run here.
# 2. Offline (_site-offline/): --forbid catches any surviving
# https://docs.twinbasic.com/<path> link the offline rewrite missed
# (bare root URL is exempt). Integrity checks run here too (minus
# sitemap/search -- the offline tree has neither).
# 3. Book (_site-pdf/book.html): --no-fail makes failures informational
# (some links are not yet fully resolved).
run: >-
node scripts/check_links.mjs
--offline --include-fragments
--check-html --check-a11y --check-ids
--check-sitemap --check-search --check-canonical
--fallback-extensions html
--index-files 'index.html,.'
--base-path '${{ steps.pages.outputs.base_path }}'
--root-dir docs/_site
docs/_site
/sep/
--offline --include-fragments
--index-files index.html
--check-html --check-a11y --check-ids
--forbid 'https://docs.twinbasic.com'
--fallback-extensions html
--index-files 'index.html,.'
--root-dir docs/_site-offline
docs/_site-offline
/sep/
Expand All @@ -109,30 +102,29 @@ jobs:
path: ./docs/_site
# The next two steps run only on manual dispatch -- they
# package the offline copy of the site (`docs/_site-offline/`,
# produced by the always-on `also_build_offline: true` flag in
# `_config.yml` via `_plugins/offlinify.rb`) and ship it as a
# workflow artifact for the `release` job to attach to a new
# GitHub release. Pushes to `staging` skip this and only
# deploy to Pages.
# produced by the `also_build_offline: true` flag in
# `_config.yml`) and ship it as a workflow artifact for the
# `release` job to attach to a new GitHub release. Pushes to
# `staging` skip this and only deploy to Pages.
- name: Package offline site
if: github.event_name == 'workflow_dispatch'
run: |
if [ ! -d ./docs/_site-offline ]; then
echo "::error::./docs/_site-offline not found -- ensure also_build_offline is true in _config.yml (or in the JEKYLL_SITE_CONFIG override)"
echo "::error::./docs/_site-offline not found -- ensure also_build_offline is true in _config.yml"
exit 1
fi
(cd ./docs/_site-offline && zip -rq "${{ runner.temp }}/twinbasic-docs-offline.zip" .)
- name: Upload offline-site workflow artifact
if: github.event_name == 'workflow_dispatch'
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: twinbasic-docs-offline-zip
path: ${{ runner.temp }}/twinbasic-docs-offline.zip
# Workflow-internal hand-off to the release job; the
# release itself carries the long-lived copy of the zip.
retention-days: 7
- name: Upload book PDF workflow artifact
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: twinbasic-docs-book-pdf
path: ./docs/_pdf/book.pdf
Expand All @@ -150,7 +142,7 @@ jobs:
uses: actions/deploy-pages@v5

# Release job -- runs only on manual dispatch, after a successful
# Pages deploy. Each manual deploy cuts a new GitHub release whose
# Pages deploy. Each manual deploy cuts a new GitHub release whose
# tag tracks a publicly visible documentation snapshot at
# https://docs.twinbasic.com, with the offline-browsable copy of
# the site attached as twinbasic-docs-offline.zip.
Expand All @@ -160,16 +152,16 @@ jobs:
runs-on: ubuntu-latest
permissions:
# softprops/action-gh-release needs contents:write to create
# tags and publish releases. The top-level permissions block
# tags and publish releases. The top-level permissions block
# only grants contents:read for the build/deploy path.
contents: write
steps:
- name: Download offline-site workflow artifact
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
name: twinbasic-docs-offline-zip
- name: Download book PDF workflow artifact
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
name: twinbasic-docs-book-pdf
- name: Compute release tag and name
Expand Down
38 changes: 9 additions & 29 deletions WIP.md
Original file line number Diff line number Diff line change
Expand Up @@ -424,42 +424,22 @@ WIP.md itself (and other files outside `docs/`) is not part of the Jekyll site a

Python scripts are reserved for non-render concerns: one-off content conversion (e.g. `scripts/convert_em_dash_separators.py`), repo audits, dev tooling, link checks beyond `check.bat`, anything that runs *outside* a Jekyll build. They should never be a prerequisite for the render pipeline.

## JS builder port (shipped, Phase 9 cleanup)
## Build pipeline

The Jekyll + Ruby build pipeline has been ported to a custom single-purpose Node.js tool that lives at the repo root in [builder/](builder/) (sibling of `docs/`, not inside it). All eight build phases land on the production tree and produce byte-equivalent output to Jekyll modulo the entries in [builder/accepted-divergences.mjs](builder/accepted-divergences.mjs). Phase 9 is the QoL / documentation / cleanup consolidation pass that adds CLI flags (`--no-offline`, `--no-pdf`, `--serving`, `--profile-offline`), a Phase 7 nav-block cache, a generic `_data/*.yml` loader (`data.mjs`), a multi-divergence audit tool (`_audit_accepted.mjs`), `_diff.mjs`'s `--against-disk` and `--multi` modes, a PDF cross-reference completeness check in `verify-phase8.mjs`, and [builder/README.md](builder/README.md). Phase 10 (planned) picks up the output-changing FUTURE-WORK items (Shiki theming generated from upstream `.twin` source files, mermaid auto-gen, copy-code SSR, linkify, search-data minification, AST-based JTD patcher). The Jekyll-to-tbdocs cutover is tracked in [builder/FUTURE-WORK.md §C1](builder/FUTURE-WORK.md).
The site builds via [builder/](builder/), a custom Node.js static site generator. See [builder/PLAN.md](builder/PLAN.md) for the architecture overview and [builder/README.md](builder/README.md) for the quickstart.

See [builder/README.md](builder/README.md) for the quickstart, [builder/PLAN.md](builder/PLAN.md) for the architecture overview, and [builder/PLAN-1.md](builder/PLAN-1.md) .. [builder/PLAN-9.md](builder/PLAN-9.md) for the per-phase specs.
`builder/one-offs/` contains dev-test scripts used during the port phases; these are not part of the production build path.

Until the cutover runs, the Jekyll pipeline below remains the canonical build path.
The diff and verify harnesses (`_triage.mjs`, `_diff.mjs`, `_diff_all.mjs`, `_audit_accepted.mjs`, `_sitemap_diff.mjs`, `_spot.mjs`, `verify-phase{1..8}.mjs`, `accepted-divergences.mjs`) were retired in the Phase 10 cutover commit. Regression detection now relies on `scripts/check_links.mjs` (expanded into a site-integrity checker; see [docs/check.bat](docs/check.bat)).

### Builder diff / triage / verify tools
### Historical note

When iterating on a phase or chasing a divergence vs Jekyll, **reach for one of these tools before reading source by hand**. They drive the in-memory tbdocs pipeline through whatever phase the requested target needs, then byte-diff against the matching file in Jekyll's `docs/_site/`, `docs/_site-offline/`, or `docs/_site-pdf/`. Listed in order of "what do I run first?":
The site was originally built with Jekyll + just-the-docs. The Jekyll source set (`docs/_plugins/`, `docs/_includes/`, `docs/_layouts/`, `docs/_sass/`, `docs/Gemfile`) was retired in the Phase 10 cutover commit; the directories were kept for one release cycle as reference and then deleted in a follow-up cleanup commit. Search the git log for `Phase 10` to find both commits.

- [builder/_triage.mjs](builder/_triage.mjs) — bulk classifier covering Phases 3-8 in one run. Walks every page, finds the first divergence from Jekyll, classifies it into a coarse bucket so remaining work can be ranked by pattern frequency × visual severity. After the per-page bucket table prints, an **auxiliary audit** section covers sitemap, redirect stubs, robots.txt, search index, the **offline tree** (offline pages, offline redirects, offline CSS, offline JTD JS, offline search-data.js), and the **PDF tree** (book.html with per-article accepted-divergence handling, the two stylesheets, the image inventory). Each auxiliary line reports MATCH or a one-line DIFFER summary with counts; a clean run prints MATCH at every line. Flags: `--all` (print every example per bucket; default caps at 3), `--help`. **Start here** when something looks off across the board.
- [builder/_diff.mjs](builder/_diff.mjs) — single-target diff. Pinpoints one file's first divergence with ~200 chars of context. Modes — online site: default page (`<srcRel>`, `--full` keeps the sidebar), `--redirect=<fromPath>`, `--robots`, `--search=<srcRel>`. Offline mirror: `--offline=<srcRel>`, `--offline-redirect=<fromPath>`, `--offline-css=<themeRel>`, `--offline-jtd`, `--offline-search`. PDF book: `--book` (build-info normalised), `--book=full` (no normalisation), `--pdf-image=<rel>` (MATCH / MISS / MISSING-IN-INVENTORY), `--pdf-css=<rel>`. Page-diff modifiers: `--against-disk[=<root>]` (diff the on-disk write rather than the in-memory render — catches write-time encoding bugs), `--multi` (continue past the first divergence and report every distinct region). Always available: `--help`. **Run after triage** to drill into one representative file from the largest bucket.
- [builder/_diff_all.mjs](builder/_diff_all.mjs) — per-bucket divergence audit. Aggregates Phase 3 / Phase 4 divergences across all pages.
- [builder/_audit_accepted.mjs](builder/_audit_accepted.mjs) — accepted-divergence audit. Diffs every page in `ACCEPTED_DIVERGENCE_PATHS` against Jekyll's `_site/<destPath>` and reports EVERY divergence region (not just the first), so a hidden secondary divergence behind an existing accepted entry doesn't stay masked. Flags: `--all` (print every region per page; default caps at 5), `--help`.
- [builder/_sitemap_diff.mjs](builder/_sitemap_diff.mjs) — sitemap URL-set diff against Jekyll's `sitemap.xml`.
- [builder/_spot.mjs](builder/_spot.mjs) — single-page output dump (useful for piping into a paginator when investigating a specific page).
- [builder/verify-phase{1..8}.mjs](builder/) — per-phase acceptance harness. Each Phase N has a matching `verify-phaseN.mjs` that drives Phases 1..N into a scratch destination and asserts the acceptance checks from `PLAN-N.md §10`. Phase 8's harness does per-article byte-diff vs `_site-pdf/book.html` with accepted-divergence skipping plus structural checks (file count, image presence, CSS parity). Run before merging phase changes; treat failures as blockers.
### Migration notes

Common workflows:

| Question | Tool |
|---|---|
| "Is everything still byte-perfect vs Jekyll?" | `cd builder && node _triage.mjs` |
| "Did Phase N regress?" | `cd builder && node verify-phaseN.mjs` |
| "Why does this one page differ?" | `cd builder && node _diff.mjs <srcRel>` |
| "Why does this offline page differ?" | `cd builder && node _diff.mjs --offline=<srcRel>` |
| "Are the redirect stubs right?" | `cd builder && node _diff.mjs --redirect=<fromPath>` |
| "Are the offline redirect stubs right?" | `cd builder && node _diff.mjs --offline-redirect=<fromPath>` |
| "Is the lunr search index byte-equal?" | `cd builder && node _diff.mjs --search=<srcRel>` |
| "Does book.html match Jekyll?" | `cd builder && node _diff.mjs --book` |
| "Does the PDF tree have a given image?" | `cd builder && node _diff.mjs --pdf-image=<rel>` |
| "Is a PDF-tree CSS byte-equal?" | `cd builder && node _diff.mjs --pdf-css=<themeRel>` |

All tools live in `builder/` and expect to be run from inside it (`cd builder && node <tool>.mjs ...`). They read `../docs/_site/`, `../docs/_site-offline/`, and `../docs/_site-pdf/` as the Jekyll references; all three trees must be up-to-date (run `build.bat` once if not). The tools never write to `docs/`; scratch destinations are under `docs/_site-verify*` and `docs/_site-diff-scratch` and are cleaned up on exit.
- `_site-new/` is no longer used. Run `rm -rf docs/_site-new/` on first sync after the cutover.
- The eight `verify-phase{N}.mjs` harnesses were retired in the same cutover commit. Regression detection now relies on `scripts/check_links.mjs` (expanded into a site-integrity checker; see [docs/check.bat](docs/check.bat)).

## Build / preview

Expand Down
12 changes: 0 additions & 12 deletions _ktest.rb

This file was deleted.

21 changes: 0 additions & 21 deletions _mtest.mjs

This file was deleted.

Loading