importer: shared TeX-plan library + parsing fixes from real-blueprint dogfooding - #6
Conversation
… dogfooding
New scripts/lib/tex-plan.mjs (comment stripping, recursive \input resolution,
custom-macro expansion, directive/environment parsing with brace-aware heading
titles, single-quoted YAML scalars) with unit tests; import-blueprint.mjs plan
mode now consumes it. Fixes, each reproduced on RemyDegenne/brownian-motion or
Sphere-Packing-Lean:
- plan mode failed with 'no items parsed' on multi-file blueprints (content.tex
as a pure \input sequence — the layout leanblueprint itself recommends);
\input is now resolved recursively, and --macros=a.tex,b.tex expands the
project's custom shorthands so they don't leak into statements raw
- frontmatter titles with LaTeX math broke the site build ('unknown escape
sequence' — double-quoted YAML treats backslash as an escape); fm() now emits
single-quoted YAML scalars
- scrape mode titled chapters after \part entries and lost every real chapter
name ('Chapter 3'…'Chapter 15' on brownian-motion): parseChapterToc read only
sub-toc-0; the chapter level is now the deepest integer-numbered TOC level
- scrape mode defaulted --base-url and the label to the Sphere-Packing site;
--base-url is now required and a missing --label derives from the site title
- after an import replaces the demo blueprint, lakefile.toml roots still name
the deleted .lean chapters and lake build / CI fail silently later — the
importer now prints a loud warning with the docs pointer
- \mathlibok / \notready marker macros are stripped instead of leaking into
rendered statements
- hast-util-from-html was imported but undeclared (worked via hoisting luck);
now a real dependency
- main() is guarded so tests can import fm/parseChapterToc; --chapter-level=
section supports plastex split-level=1 blueprints in plan mode
Verified: 112/112 tests; live scrape of brownian-motion now yields 15 correctly
titled chapters + derived label + stale-roots warning; plan mode on its raw
multi-file content.tex yields 620 items / 15 chapters with no manual flattening.
Co-Authored-By: Shannon's Claude <257597027+shannonshen49@users.noreply.github.com>
|
Do you want Sepo to review this PR? Add the |
![sepo-agent-app[bot]](https://avatars.githubusercontent.com/in/3527007?s=60&v=4){kind=small}
| const args = [] | ||
| if (def.optDefault !== null) { | ||
| if (src[j] === "[") { | ||
| const close = src.indexOf("]", j) |
There was a problem hiding this comment.
When src[j] === "[" but no matching ] exists, indexOf returns -1; this slices nearly the remaining source as the optional arg and resets j to 0, so a malformed call like \ball[r{x} expands to corrupted output (B_{r{x}(\)ball...). Please guard close === -1 and either use the default optional arg, leave the macro unexpanded, or fail fast, with a regression test.
| const chapters = [] | ||
| const parts = [] | ||
| let current = null | ||
| const T = "(?:[^{}]|\\{[^{}]*\\})*" |
There was a problem hiding this comment.
T only accepts one nested brace group inside a heading. A valid title like \chapter{A \frac{\sqrt{n}}{2}} does not match the chapter arm, so the following items are parsed under fallback Blueprint. Since this file already has readBalanced, can the heading scan use balanced groups instead of a bounded-depth regex?
This comment has been minimized.
This comment has been minimized.
|
Re the INFO finding on missing |
| args.push(g.inner) | ||
| j = g.end | ||
| } else { | ||
| args.push(src[j]) |
There was a problem hiding this comment.
This branch treats every unbraced required argument as one character. In TeX, an unbraced argument can be a full control sequence, so a valid call like \norm\mu for \newcommand{\norm}[1]{...#1...} becomes corrupted output. Please consume a control word/control symbol as one argument token when src[j] === "\\", or leave the macro unexpanded if the argument token is incomplete, and add a regression test.
AI Review Synthesis
Summary of PR/Issue
ReviewThe PR is close, but several parser edge cases can silently corrupt imported TeX or blueprint structure. The most important fixes are localized and suitable for an automated follow-up.
Progress
Issue DetailsUnbraced macro control-sequence arguments are corruptedCause: In Candidate solutions: When Optional macro argument without closing bracket corrupts expansionCause: Candidate solutions: Guard Deeply nested heading titles parse under fallback BlueprintCause: Candidate solutions: Parse heading arguments with the existing New importer flags are not documented in user docsCause: The new flags are present in the script usage but absent from the migration/reference docs. Candidate solutions: Document Comments: Reviewers differed on severity here: one rated this WARNING, one INFO because the importer is legacy. Synthesis keeps it WARNING because Same-line repeated \input is only partially resolvedCause: Candidate solutions: Optional hardening: scan all directive tokens on the line or explicitly document one- Comments: This is INFO-level and not recommended as required automation work. Recommended Next Step
Final Verdict
Action Items
|
…lan-mode flags
Review findings on the shared parser:
- expandMacros consumed one character for unbraced required args, corrupting
\norm\mu-style TeX; an unbraced control sequence is now taken whole.
- An unterminated optional [arg falls back to the macro default instead of
slicing to -1 and corrupting the output.
- Heading titles now nest braces two levels (\chapter{A \frac{\sqrt{n}}{2}});
one level made such chapters fall through, attaching their items to the
previous chapter.
- migrate-existing-blueprint.md documents --macros and --chapter-level for
plan mode (previously only in the script header).
Same-line repeated \input stays line-leading-only by design (leanblueprint
convention is one \input per line).
Co-Authored-By: Shannon's Claude <257597027+shannonshen49@users.noreply.github.com>
|
Round-2 findings addressed in 56028a0: unbraced control-sequence args ( |
… plan-mode flag paragraphs) Co-Authored-By: Shannon's Claude <257597027+shannonshen49@users.noreply.github.com>
Rubrics UpdateNo changes were committed to Added one active rubric:
It captures the trusted maintainer preference from PR #6: blueprint importer work should preserve warn-and-continue behavior for missing TeX No commit was made.
|
1 participant
Summary
Second PR in the migration-fix series (after #5). Every fix below was reproduced by importing two real blueprints — RemyDegenne/brownian-motion (a
\part{}-structured, multi-file plan) and thefundamentaltheor3m/Sphere-Packing-Lean (plastex split-level=1) — into fresh template copies.New
scripts/lib/tex-plan.mjs— TeX-plan parsing toolkit (comment stripping, recursive\inputresolution,\newcommand/\DeclareMathOperatorexpansion, environment/directive parsing with brace-aware heading titles, single-quoted YAML scalars), unit-tested, and shared with the upcoming native-chapter migration script.Fixes in
import-blueprint.mjs:no items parsedon\input-sequence layouts (the one leanblueprint's own template comments recommend) — inputs now resolve recursively;--macros=expands project shorthands.$\mathcal{V}$) emitted invalid double-quoted YAML and broke the site build;fm()now single-quotes.sub-toc-0, so\part-structured sites got part titles glued onto chapters 1–2 andChapter Nfallbacks for the rest; the chapter level is now the deepest integer-numbered TOC level (LaTeX chapter counters run globally through parts, matching the dep graph'sN.Mitem numbers).--base-urlno longer silently defaults to the Sphere-Packing site, and a missing--labelderives from the published site's<title>.lakefile.tomlrootsnaming deleted.leanchapters (CI fails later, silently) — now detected with a loud warning pointing at the adoption tutorial.\mathlibok/\notreadymarkers are stripped instead of leaking into statements verbatim.hast-util-from-htmldeclared (was an undeclared transitive dependency).main()import-guarded;fm/parseChapterTocexported for tests;--chapter-level=sectionfor split-level-1 plans.Verification
npm run checkand full test suite pass (112/112, including 10 new tests:\inputchains, macro args/optional-defaults/operators, marker stripping,\part+ flat TOC fixtures, YAML escaping).content.tex: 620 items / 15 chapters with no manual flattening.🤖 Generated with Claude Code