tex: the primitives and rules real package code is built on by tannevaled · Pull Request #2 · go-tex/engine

tannevaled · 2026-08-16T23:51:54Z

Continuation of the pgf/TikZ work: the pgfkeys blocker is lifted, the whole pgf stack loads from its own sources, and a figure draws.

Seven divergences from TeX, each measured before the fix

Each behavior was isolated in a few lines and then compared against a real TeX (tectonic); the expected values in the tests are that engine's, not a guess.

Rule	What was breaking
`\ifx` compares meanings, and a character token has one (category + character) — so `\ifx\next/` is true if `\next` was `\let` to a slash	pgfkeys split every key path wrong, then executed the value as if it were a command
The alphabetic constant `a/`\a was missing	`\catcode`@=11` was changing the category of character 0
Reading a category (`\the\catcode`@`)	files that restore a category would restore emptiness → `@` became an escape character
`\afterassignment`	a scanner that restarts after `\let` never resumed
Dimension → integer coercion (`\number\pgf@x`)	every computed coordinate came out wrong
`\newbox` handle = register number	the contents of a package's boxes disappeared (everything went into box 0)
`\global` forgets pending saves	`{…\global\pgf@x=\pgf@x}` — the idiom that carries a value out of a group — returned nothing

In addition: a file name is expanded before it is resolved (a package names the file to load through a macro), and the engine's named colors are published in the form a color reader expects, so that a drawing package can ask for the model and the values instead of declaring it unsupported.

Result

pgfrcs, pgfkeys, pgfsys, pgfmath, pgfcore and pgf load in full under \documentclass{article}.
Milestone (a) reached: \pgfpathmoveto{\pgfpoint{0pt}{0pt}}\pgfpathlineto{\pgfpoint{50pt}{30pt}}\pgfusepath{stroke} reaches the page as <path d="M 0.0 0.0 L 50.0 30.0" fill="none"/>, within the picture's scope, at the right coordinates.
Remaining for TikZ: the startup of \begin{tikzpicture} (\draw is installed by that startup) — next step identified.

The actual loading stays opt-in via GOTEX_PGF; by default nothing changes for an ordinary document.

Verification

746 tests pass, 0 skips, 0 failures; go vet and gofmt clean.
Conformance ratchet and document corpus unchanged.
scripts/fidelity.sh --check (CI gate, gotex vs real LaTeX): OK.
New code at 100% coverage; the tests document the real-TeX value in a comment.

Running the real pgf/TikZ sources turned up seven places where the engine answered differently from TeX. Each was isolated to a few lines, checked against a real TeX (tectonic) before being implemented, and is a rule that package code leans on constantly — the failures they caused looked like anything but their cause. \ifx compares meanings, and a character token has one: its category and its character, the same meaning a control sequence \let to that character has. So \ifx\next/ is true when \next was \let to a slash. Every one-token lookahead tests what it peeked at this way; without it pgfkeys mis-split every key path it was given and then ran the key's value as if it were a command. TeX's alphabetic constant — `<character>, or `<single-character control sequence> — was missing entirely, so \catcode`\@=11 quietly set the category of character 0 and left @ alone. Reading a category was missing too, so the files that put a category back (\edef\saved{\the\catcode`\@} … \catcode`\@= \saved) restored garbage. \afterassignment holds one token until the next assignment has been carried out, which is how a scanner resumes itself after \let has swallowed a token. An internal dimension coerces to an integer, its value in scaled points, so \number\pgf@x and \ifnum\wd0>0 read it. A box handle from \newbox is a register *number* (TeX allocates it with \chardef), so \box\mybox reaches the right register instead of box 0, where a package's boxed material used to vanish. A global assignment must outlive every open group, so a local assignment made earlier in one of them is no longer restored over it at the closing brace. The idiom that carries a computed value out of a group — {…\global\pgf@x=\pgf@x} — depended on it entirely. Alongside those: a file name is expanded before it is looked up (a package names the file to load through a macro), and the engine's named colours are published in the form a colour-reading package expects, so a drawing package can ask which model and values a colour has rather than reporting the model as unsupported. With these, the whole pgf stack — pgfrcs, pgfkeys, pgfsys, pgfmath, pgfcore and pgf itself — loads from its own sources under \documentclass{article}, and a picture draws: \pgfpathmoveto…\pgfusepath{stroke} now reaches the page as a real stroked path at the right coordinates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@for

\usepackage{beamerthemesplit} lost the body of the talk. So did the fifteen other beamer themes built the same way. The theme was not at fault: the file is 24 lines, and the loss came from \ProcessOptionsBeamer, from the \@for inside it, and from one control sequence the engine never defined. \@for over an UNDEFINED control sequence does not iterate over nothing — it swallows what follows. LaTeX's own \ProcessOptions guards the loop, \ifx\@classoptionslist\relax\else \@for\CurrentOption:=\@classoptionslist\do{…} but beamer's \ProcessOptionsBeamer guards on \@currext instead and runs the loop unguarded, as does \beamer@filterclassoptions. Both therefore rely on \documentclass having set the list — and the engine never set it. ltclass.dtx is followed on all four points: \@classoptionslist and \@raw@classoptionslist both start at \relax (§547), \documentclass fills them on the FIRST class load, \ifx\@classoptionslist\relax \protected@xdef\@classoptionslist{\zap@space#2 \@empty}% \gdef\@raw@classoptionslist{#2}% and the \ifx guard means a class that loads another class does not overwrite the document's own list. A test pins that last part, which is easy to miss. Measured over 200 real beamer talks: 0 lost, one MORE renders, 12 outputs change — a narrow change, as it should be — and the page-count error against tectonic on the 14 talks with a reference falls 11 -> 6, with none moving away and 9 of the 14 landing exactly on it. Over 200 real arXiv papers: 0 lost, two more render, 29 outputs change. Not fixed here, and pre-existing: a frame TITLE does not render, with or without a theme, before or after this change.

\global\setlength\xx{5pt} inside a group was a LOCAL assignment: it took effect inside the group and was undone on the way out, in silence. \global\xx=5pt and \global\advance\xx by4pt both worked, which is what kept the hole out of sight. In TeX a prefix applies to the assignment it finds after EXPANDING what follows — tex.web §1211 takes "the next non-blank non-relax NON-CALL token", so a macro is expanded away. \setlength is a macro there (ltlength.dtx: \def\setlength#1#2{#1 #2\relax}), so \global reaches the register assignment by itself. Here it is a primitive, which getXToken does not expand, and doGlobal's switch had no case for it: the default branch put the token back and ran it locally. The machinery was already in place — assignLength takes a global flag that nothing ever passed as true. \setlength, \addtolength, \settowidth, \settoheight and \settodepth now receive the prefix, so they reach both a \newlength register and one of the engine's dimension parameters (\textwidth is \let to \hsize, a different path, same rule). Measured over 200 real beamer talks: 0 documents lost, 0 outputs changed, page count identical — the change is inert on that corpus, which is what a fix this narrow should look like. One test case of mine was wrong before the engine was: I asserted \the\xx after setting \textwidth. Removed; the parameter path has its own test.

@let

…mes expand (#99) Two defects that turn out to be one story: no beamer theme had ever been loaded, and every keyval value was silently lost. 1. \@ifnextchar does not INSERT the branch it picks — it stores it first. ltdefns.dtx: \long\def\@ifnextchar#1#2#3{\let\reserved@d=#1 \def\reserved@a{#2}\def\reserved@b{#3}\futurelet\@let@token\@ifnch} and \def\reserved@b{#3} SCANS the branch as a macro body, which halves ## a second time (tex.web §479: a # followed by a # stores one #). This engine's \@ifnextchar is a primitive that inserted the branch verbatim, skipping it. keyval is built on exactly that: \def\define@key#1#2{\@ifnextchar[{\KV@def{#1}{#2}}{\long\@namedef{KV@#1@#2}####1}} The #### is halved once into \define@key's own body and a second time by \@ifnextchar, leaving the one # that \def reads as the parameter text #1. Without it the generated macro's parameter text was ##1 and bound nothing: \setkeys{fam}{k=VAL} gave <<>>. Real LaTeX gives \long macro:#1-><<#1>> where this engine gave macro:##1-><<#1>>. \@ifstar and \@ifnextbracket pick their branch the same way and are halved for the same reason. 2. \usepackage, \documentclass and \LoadClass now EXPAND their {name}. Checked in one line: \def\nm{marker}\usepackage{zz\nm} loads zzmarker.sty in real LaTeX and loaded nothing here. beamer builds every theme file name that way — \beamer@calltheme does \usepackage[{#1}]{beamertheme\beamer@themename} — so \usetheme{default} asked for a package literally named "beamertheme\beamer@themename". Together: the frametitle template (beamerouterthemedefault.sty) finally exists, so a FRAME TITLE renders; and because key values no longer leak, the theme files stop contributing four pages of debris. On the minimal document that is exactly the reference — n frames make n pages, title and body both present, page 364.2x273.15. Measured over 200 real beamer talks: 0 content lost, pages 1061 -> 1095, vector paths 155106 -> 174863. Eight documents stop producing a sized page and each was checked by hand: four are corpus FRAGMENTS with no \begin{document} (0 pages is what real LaTeX gives them), one is a talk with no frame at all, one is a fragile frame whose verbatim body this engine cannot render either way, one is unchanged when re-run cold, and one (039a234b02c5…) trips the runaway guard on main TOO — the guard now fires sooner, so 40KB of debris becomes none. Over 200 arXiv papers: the single apparent loss renders 2.46MB in both binaries when re-run cold; it had exceeded the sweep's time cap.

\url, \nolinkurl, \href and \hypertarget read their argument literally out of the base input, so ~, % , #, _ and & keep the value they have in the file. Only the SOURCE can be read that way. When something is already pending on the input stack the argument is coming from an EXPANSION, and the base text at the mouth's position is whatever follows the macro that produced it — so the scan skipped the argument and ate the next braced group in the document. Two ways that shows up, both measured: - A link whose target a macro builds. This paper defines \newcommand{\pyref}[2]{\href{https://…/discopy.#1.html}{\py{#2}}}, and the URL was typeset in the running text ("The class https://… is defined by:") because \href never consumed it. - beamer's navigation lays a \hypertarget per frame whose name it composes. With an outer theme that draws a head or foot line (split, infolines — so \usetheme{Berlin}, Madrid, Warsaw, Copenhagen) the target is emitted while a [fragile] frame's body is being copied out verbatim, and the group eaten was the frame's own: \textbf{gras} reached \jobname.vrb as \textbf, and the frame lost everything after its first control sequence. Issue #103. With a token list pending, the argument is read as tokens instead. Reading the source raw is kept for the case it is meant for — a \url written in the document — where % must not start a comment. Measured over 200 real talks and 200 arXiv papers: beamer 1121 → 1131 pages 5 talks up, none down arXiv 4052 → 4051 pages one paper, and it is a correction: three \href targets it used to typeset in the page are gone, no text lost (word counts unchanged) The glyph count falls in 61 documents for that same reason — a URL that was never meant to be on the page.

@input

Two things this engine did not model, both straight from the reference, and both reached through the same document: a beamer talk with a [fragile] frame. ## A file is a level of input (tex.web §537) start_input does begin_file_reading — "set up cur_file and new level of input" — and the file is read to its end before the level underneath resumes; §329 end_file_reading pops back. This mouth merged a file into ONE character buffer at the mouth's position, and it reads pending token lists before that buffer, so a file \input from inside a macro arrived AFTER the rest of that macro's body. It is not a corner case: \frame<*>[…][{…,fragile=false}]{\begingroup\input{\jobname.vrb}\endgroup} is how beamer re-reads a fragile frame's body, and the body arrived after the frame had closed — into a box nothing places. pushInputLevel/popInputLevel make the file a level of its own, carrying the buffer, the mouth's position, the line table, the pending lists and the no-progress guard's baseline (which watches e.bpos, and a new buffer starts at 0 again). \input, \InputIfFileExists, the .bbl reader and class/package loading all go through it, so the line-number bookkeeping that discounted a loaded file's lines from the document's own (inputNL/loadedNL/nlCount) is gone: a file no longer shifts the document's lines because it is no longer in its buffer. \InputIfFileExists now runs its then-code BEFORE the file, which is what ltfiles.dtx does: \IfFileExists{#1}{#2\@addtofilelist{#1}\@@input\@FilEf@und}. ## \begin{env} … \end{env} is a group (ltmiscen.dtx) \protected\def\begin#1{… \begingroup\@endpefalse\reserved@a} where \reserved@a is \def\@Currenvir{#1}… \csname #1\endcsname \def\end#1{\csname end#1\endcsname\@Checkend{#1}\expandafter\endgroup …} \begingroup comes first and \@Currenvir is defined inside it. Without the group \@Currenvir was never restored, so a class that leaves an environment early by closing its group could not — and beamer's fragile frame does exactly that: \def\beamer@checkforfragile#1fragile#2\relax{… \endgroup% end environment \expandafter\beamer@framecommand\beamer@frameoptions\bgroup} then calls \frame, which picks its syntax with \ifx\@Currenvir\beamer@frametext. With \@Currenvir still "frame" the command form took the ENVIRONMENT path a second time and \beamer@doseveralframes was handed the bare \bgroup instead of the frame's body — measured, the frame went on the page empty. The group is opened and closed by the Go-side probes \gotex@checkenv and \gotex@endenv, not by \begingroup/\endgroup tokens, because \begin and \end must stay expandable here: as tokens they would print as literal text wherever \begin is only expanded (inside \message or \edef). Four consequences, each from the reference: - Allocation is GLOBAL. ltplain.dtx's \e@alloc ends with `\global#2#6\allocationnumber`; ltcounts.dtx's \@definecounter makes \cl@<c>, \p@<c> and \the<c> global; ltthm.dtx's \@xnthm/\@ynthm use \xdef and \global\@namedef. So \newcount, \newdimen, \newskip, \newlength, \newcounter and \newtheorem now define globally — otherwise a counter declared inside \begin{document} (which is an environment) vanished with it. - Every alignment entry is a group (tex.web §791: a template's u-part and v-part are inserted inside braces), so a font switch in a cell stops there. buildCellHList terminated the cell with a bare } sentinel and no group; that stray brace was silently swallowed until a \begingroup was open. - An environment the engine implements in Go swallows its own \end, so \end — and the \endgroup with it — never runs: tabular, tabularx, verbatim, lstlisting, equation, align, minipage and the gobbled comment environments each close the group themselves. - setSrcPos no longer discounts loaded lines: a level reports its own file's line, which is where a diagnostic about that file belongs. ## Measured 200 real beamer talks and 200 arXiv papers, against main: beamer 1121 → 1031 pages 182 743 → 171 499 glyphs 16 up, 14 down arXiv 4052 → 4057 pages 9 485 942 → 9 234 865 14 up, 7 down NOT a net win yet, and the reason is a defect that predates this branch and is reproducible on main: with an outer theme of the split/infolines family (\usetheme{Berlin}, Madrid, Warsaw, Copenhagen) the verbatim capture of a fragile frame's FIRST line loses everything after its first control sequence — \textbf{gras} is written to the .vrb as \textbf. Reading that back where it belongs (inside the frame) then swallows the next line, which is why correcting the ORDER makes those talks worse. See the issue.

@input

Two things this engine did not model, both straight from the reference, and both reached through the same document: a beamer talk with a [fragile] frame. ## A file is a level of input (tex.web §537) start_input does begin_file_reading — "set up cur_file and new level of input" — and the file is read to its end before the level underneath resumes; §329 end_file_reading pops back. This mouth merged a file into ONE character buffer at the mouth's position, and it reads pending token lists before that buffer, so a file \input from inside a macro arrived AFTER the rest of that macro's body. It is not a corner case: \frame<*>[…][{…,fragile=false}]{\begingroup\input{\jobname.vrb}\endgroup} is how beamer re-reads a fragile frame's body, and the body arrived after the frame had closed — into a box nothing places. pushInputLevel/popInputLevel make the file a level of its own, carrying the buffer, the mouth's position, the line table, the pending lists and the no-progress guard's baseline (which watches e.bpos, and a new buffer starts at 0 again). \input, \InputIfFileExists, the .bbl reader and class/package loading all go through it, so the line-number bookkeeping that discounted a loaded file's lines from the document's own (inputNL/loadedNL/nlCount) is gone: a file no longer shifts the document's lines because it is no longer in its buffer. \InputIfFileExists now runs its then-code BEFORE the file, which is what ltfiles.dtx does: \IfFileExists{#1}{#2\@addtofilelist{#1}\@@input\@FilEf@und}. ## \begin{env} … \end{env} is a group (ltmiscen.dtx) \protected\def\begin#1{… \begingroup\@endpefalse\reserved@a} where \reserved@a is \def\@Currenvir{#1}… \csname #1\endcsname \def\end#1{\csname end#1\endcsname\@Checkend{#1}\expandafter\endgroup …} \begingroup comes first and \@Currenvir is defined inside it. Without the group \@Currenvir was never restored, so a class that leaves an environment early by closing its group could not — and beamer's fragile frame does exactly that: \def\beamer@checkforfragile#1fragile#2\relax{… \endgroup% end environment \expandafter\beamer@framecommand\beamer@frameoptions\bgroup} then calls \frame, which picks its syntax with \ifx\@Currenvir\beamer@frametext. With \@Currenvir still "frame" the command form took the ENVIRONMENT path a second time and \beamer@doseveralframes was handed the bare \bgroup instead of the frame's body — measured, the frame went on the page empty. The group is opened and closed by the Go-side probes \gotex@checkenv and \gotex@endenv, not by \begingroup/\endgroup tokens, because \begin and \end must stay expandable here: as tokens they would print as literal text wherever \begin is only expanded (inside \message or \edef). Four consequences, each from the reference: - Allocation is GLOBAL. ltplain.dtx's \e@alloc ends with `\global#2#6\allocationnumber`; ltcounts.dtx's \@definecounter makes \cl@<c>, \p@<c> and \the<c> global; ltthm.dtx's \@xnthm/\@ynthm use \xdef and \global\@namedef. So \newcount, \newdimen, \newskip, \newlength, \newcounter and \newtheorem now define globally — otherwise a counter declared inside \begin{document} (which is an environment) vanished with it. - Every alignment entry is a group (tex.web §791: a template's u-part and v-part are inserted inside braces), so a font switch in a cell stops there. buildCellHList terminated the cell with a bare } sentinel and no group; that stray brace was silently swallowed until a \begingroup was open. - An environment the engine implements in Go swallows its own \end, so \end — and the \endgroup with it — never runs: tabular, tabularx, verbatim, lstlisting, equation, align, minipage and the gobbled comment environments each close the group themselves. - setSrcPos no longer discounts loaded lines: a level reports its own file's line, which is where a diagnostic about that file belongs. ## Measured 200 real beamer talks and 200 arXiv papers, against main: beamer 1121 → 1031 pages 182 743 → 171 499 glyphs 16 up, 14 down arXiv 4052 → 4057 pages 9 485 942 → 9 234 865 14 up, 7 down NOT a net win yet, and the reason is a defect that predates this branch and is reproducible on main: with an outer theme of the split/infolines family (\usetheme{Berlin}, Madrid, Warsaw, Copenhagen) the verbatim capture of a fragile frame's FIRST line loses everything after its first control sequence — \textbf{gras} is written to the .vrb as \textbf. Reading that back where it belongs (inside the frame) then swallows the next line, which is why correcting the ORDER makes those talks worse. See the issue.

@input

Two things this engine did not model, both straight from the reference, and both reached through the same document: a beamer talk with a [fragile] frame. ## A file is a level of input (tex.web §537) start_input does begin_file_reading — "set up cur_file and new level of input" — and the file is read to its end before the level underneath resumes; §329 end_file_reading pops back. This mouth merged a file into ONE character buffer at the mouth's position, and it reads pending token lists before that buffer, so a file \input from inside a macro arrived AFTER the rest of that macro's body. It is not a corner case: \frame<*>[…][{…,fragile=false}]{\begingroup\input{\jobname.vrb}\endgroup} is how beamer re-reads a fragile frame's body, and the body arrived after the frame had closed — into a box nothing places. pushInputLevel/popInputLevel make the file a level of its own, carrying the buffer, the mouth's position, the line table, the pending lists and the no-progress guard's baseline (which watches e.bpos, and a new buffer starts at 0 again). \input, \InputIfFileExists, the .bbl reader and class/package loading all go through it, so the line-number bookkeeping that discounted a loaded file's lines from the document's own (inputNL/loadedNL/nlCount) is gone: a file no longer shifts the document's lines because it is no longer in its buffer. \InputIfFileExists now runs its then-code BEFORE the file, which is what ltfiles.dtx does: \IfFileExists{#1}{#2\@addtofilelist{#1}\@@input\@FilEf@und}. ## \begin{env} … \end{env} is a group (ltmiscen.dtx) \protected\def\begin#1{… \begingroup\@endpefalse\reserved@a} where \reserved@a is \def\@Currenvir{#1}… \csname #1\endcsname \def\end#1{\csname end#1\endcsname\@Checkend{#1}\expandafter\endgroup …} \begingroup comes first and \@Currenvir is defined inside it. Without the group \@Currenvir was never restored, so a class that leaves an environment early by closing its group could not — and beamer's fragile frame does exactly that: \def\beamer@checkforfragile#1fragile#2\relax{… \endgroup% end environment \expandafter\beamer@framecommand\beamer@frameoptions\bgroup} then calls \frame, which picks its syntax with \ifx\@Currenvir\beamer@frametext. With \@Currenvir still "frame" the command form took the ENVIRONMENT path a second time and \beamer@doseveralframes was handed the bare \bgroup instead of the frame's body — measured, the frame went on the page empty. The group is opened and closed by the Go-side probes \gotex@checkenv and \gotex@endenv, not by \begingroup/\endgroup tokens, because \begin and \end must stay expandable here: as tokens they would print as literal text wherever \begin is only expanded (inside \message or \edef). Four consequences, each from the reference: - Allocation is GLOBAL. ltplain.dtx's \e@alloc ends with `\global#2#6\allocationnumber`; ltcounts.dtx's \@definecounter makes \cl@<c>, \p@<c> and \the<c> global; ltthm.dtx's \@xnthm/\@ynthm use \xdef and \global\@namedef. So \newcount, \newdimen, \newskip, \newlength, \newcounter and \newtheorem now define globally — otherwise a counter declared inside \begin{document} (which is an environment) vanished with it. - Every alignment entry is a group (tex.web §791: a template's u-part and v-part are inserted inside braces), so a font switch in a cell stops there. buildCellHList terminated the cell with a bare } sentinel and no group; that stray brace was silently swallowed until a \begingroup was open. - An environment the engine implements in Go swallows its own \end, so \end — and the \endgroup with it — never runs: tabular, tabularx, verbatim, lstlisting, equation, align, minipage and the gobbled comment environments each close the group themselves. - setSrcPos no longer discounts loaded lines: a level reports its own file's line, which is where a diagnostic about that file belongs. ## Measured 200 real beamer talks and 200 arXiv papers, against main: beamer 1121 → 1031 pages 182 743 → 171 499 glyphs 16 up, 14 down arXiv 4052 → 4057 pages 9 485 942 → 9 234 865 14 up, 7 down NOT a net win yet, and the reason is a defect that predates this branch and is reproducible on main: with an outer theme of the split/infolines family (\usetheme{Berlin}, Madrid, Warsaw, Copenhagen) the verbatim capture of a fragile frame's FIRST line loses everything after its first control sequence — \textbf{gras} is written to the .vrb as \textbf. Reading that back where it belongs (inside the frame) then swallows the next line, which is why correcting the ORDER makes those talks worse. See the issue.

@input

Two things this engine did not model, both straight from the reference, and both reached through the same document: a beamer talk with a [fragile] frame. ## A file is a level of input (tex.web §537) start_input does begin_file_reading — "set up cur_file and new level of input" — and the file is read to its end before the level underneath resumes; §329 end_file_reading pops back. This mouth merged a file into ONE character buffer at the mouth's position, and it reads pending token lists before that buffer, so a file \input from inside a macro arrived AFTER the rest of that macro's body. It is not a corner case: \frame<*>[…][{…,fragile=false}]{\begingroup\input{\jobname.vrb}\endgroup} is how beamer re-reads a fragile frame's body, and the body arrived after the frame had closed — into a box nothing places. pushInputLevel/popInputLevel make the file a level of its own, carrying the buffer, the mouth's position, the line table, the pending lists and the no-progress guard's baseline (which watches e.bpos, and a new buffer starts at 0 again). \input, \InputIfFileExists, the .bbl reader and class/package loading all go through it, so the line-number bookkeeping that discounted a loaded file's lines from the document's own (inputNL/loadedNL/nlCount) is gone: a file no longer shifts the document's lines because it is no longer in its buffer. \InputIfFileExists now runs its then-code BEFORE the file, which is what ltfiles.dtx does: \IfFileExists{#1}{#2\@addtofilelist{#1}\@@input\@FilEf@und}. ## \begin{env} … \end{env} is a group (ltmiscen.dtx) \protected\def\begin#1{… \begingroup\@endpefalse\reserved@a} where \reserved@a is \def\@Currenvir{#1}… \csname #1\endcsname \def\end#1{\csname end#1\endcsname\@Checkend{#1}\expandafter\endgroup …} \begingroup comes first and \@Currenvir is defined inside it. Without the group \@Currenvir was never restored, so a class that leaves an environment early by closing its group could not — and beamer's fragile frame does exactly that: \def\beamer@checkforfragile#1fragile#2\relax{… \endgroup% end environment \expandafter\beamer@framecommand\beamer@frameoptions\bgroup} then calls \frame, which picks its syntax with \ifx\@Currenvir\beamer@frametext. With \@Currenvir still "frame" the command form took the ENVIRONMENT path a second time and \beamer@doseveralframes was handed the bare \bgroup instead of the frame's body — measured, the frame went on the page empty. The group is opened and closed by the Go-side probes \gotex@checkenv and \gotex@endenv, not by \begingroup/\endgroup tokens, because \begin and \end must stay expandable here: as tokens they would print as literal text wherever \begin is only expanded (inside \message or \edef). Four consequences, each from the reference: - Allocation is GLOBAL. ltplain.dtx's \e@alloc ends with `\global#2#6\allocationnumber`; ltcounts.dtx's \@definecounter makes \cl@<c>, \p@<c> and \the<c> global; ltthm.dtx's \@xnthm/\@ynthm use \xdef and \global\@namedef. So \newcount, \newdimen, \newskip, \newlength, \newcounter and \newtheorem now define globally — otherwise a counter declared inside \begin{document} (which is an environment) vanished with it. - Every alignment entry is a group (tex.web §791: a template's u-part and v-part are inserted inside braces), so a font switch in a cell stops there. buildCellHList terminated the cell with a bare } sentinel and no group; that stray brace was silently swallowed until a \begingroup was open. - An environment the engine implements in Go swallows its own \end, so \end — and the \endgroup with it — never runs: tabular, tabularx, verbatim, lstlisting, equation, align, minipage and the gobbled comment environments each close the group themselves. - setSrcPos no longer discounts loaded lines: a level reports its own file's line, which is where a diagnostic about that file belongs. ## Measured 200 real beamer talks and 200 arXiv papers, against main: beamer 1121 → 1031 pages 182 743 → 171 499 glyphs 16 up, 14 down arXiv 4052 → 4057 pages 9 485 942 → 9 234 865 14 up, 7 down NOT a net win yet, and the reason is a defect that predates this branch and is reproducible on main: with an outer theme of the split/infolines family (\usetheme{Berlin}, Madrid, Warsaw, Copenhagen) the verbatim capture of a fragile frame's FIRST line loses everything after its first control sequence — \textbf{gras} is written to the .vrb as \textbf. Reading that back where it belongs (inside the frame) then swallows the next line, which is why correcting the ORDER makes those talks worse. See the issue.

@input

Two things this engine did not model, both straight from the reference, and both reached through the same document: a beamer talk with a [fragile] frame. ## A file is a level of input (tex.web §537) start_input does begin_file_reading — "set up cur_file and new level of input" — and the file is read to its end before the level underneath resumes; §329 end_file_reading pops back. This mouth merged a file into ONE character buffer at the mouth's position, and it reads pending token lists before that buffer, so a file \input from inside a macro arrived AFTER the rest of that macro's body. It is not a corner case: \frame<*>[…][{…,fragile=false}]{\begingroup\input{\jobname.vrb}\endgroup} is how beamer re-reads a fragile frame's body, and the body arrived after the frame had closed — into a box nothing places. pushInputLevel/popInputLevel make the file a level of its own, carrying the buffer, the mouth's position, the line table, the pending lists and the no-progress guard's baseline (which watches e.bpos, and a new buffer starts at 0 again). \input, \InputIfFileExists, the .bbl reader and class/package loading all go through it, so the line-number bookkeeping that discounted a loaded file's lines from the document's own (inputNL/loadedNL/nlCount) is gone: a file no longer shifts the document's lines because it is no longer in its buffer. \InputIfFileExists now runs its then-code BEFORE the file, which is what ltfiles.dtx does: \IfFileExists{#1}{#2\@addtofilelist{#1}\@@input\@FilEf@und}. ## \begin{env} … \end{env} is a group (ltmiscen.dtx) \protected\def\begin#1{… \begingroup\@endpefalse\reserved@a} where \reserved@a is \def\@Currenvir{#1}… \csname #1\endcsname \def\end#1{\csname end#1\endcsname\@Checkend{#1}\expandafter\endgroup …} \begingroup comes first and \@Currenvir is defined inside it. Without the group \@Currenvir was never restored, so a class that leaves an environment early by closing its group could not — and beamer's fragile frame does exactly that: \def\beamer@checkforfragile#1fragile#2\relax{… \endgroup% end environment \expandafter\beamer@framecommand\beamer@frameoptions\bgroup} then calls \frame, which picks its syntax with \ifx\@Currenvir\beamer@frametext. With \@Currenvir still "frame" the command form took the ENVIRONMENT path a second time and \beamer@doseveralframes was handed the bare \bgroup instead of the frame's body — measured, the frame went on the page empty. The group is opened and closed by the Go-side probes \gotex@checkenv and \gotex@endenv, not by \begingroup/\endgroup tokens, because \begin and \end must stay expandable here: as tokens they would print as literal text wherever \begin is only expanded (inside \message or \edef). Four consequences, each from the reference: - Allocation is GLOBAL. ltplain.dtx's \e@alloc ends with `\global#2#6\allocationnumber`; ltcounts.dtx's \@definecounter makes \cl@<c>, \p@<c> and \the<c> global; ltthm.dtx's \@xnthm/\@ynthm use \xdef and \global\@namedef. So \newcount, \newdimen, \newskip, \newlength, \newcounter and \newtheorem now define globally — otherwise a counter declared inside \begin{document} (which is an environment) vanished with it. - Every alignment entry is a group (tex.web §791: a template's u-part and v-part are inserted inside braces), so a font switch in a cell stops there. buildCellHList terminated the cell with a bare } sentinel and no group; that stray brace was silently swallowed until a \begingroup was open. - An environment the engine implements in Go swallows its own \end, so \end — and the \endgroup with it — never runs: tabular, tabularx, verbatim, lstlisting, equation, align, minipage and the gobbled comment environments each close the group themselves. - setSrcPos no longer discounts loaded lines: a level reports its own file's line, which is where a diagnostic about that file belongs. ## Measured 200 real beamer talks and 200 arXiv papers, against main: beamer 1121 → 1031 pages 182 743 → 171 499 glyphs 16 up, 14 down arXiv 4052 → 4057 pages 9 485 942 → 9 234 865 14 up, 7 down NOT a net win yet, and the reason is a defect that predates this branch and is reproducible on main: with an outer theme of the split/infolines family (\usetheme{Berlin}, Madrid, Warsaw, Copenhagen) the verbatim capture of a fragile frame's FIRST line loses everything after its first control sequence — \textbf{gras} is written to the .vrb as \textbf. Reading that back where it belongs (inside the frame) then swallows the next line, which is why correcting the ORDER makes those talks worse. See the issue.

@array

#153 said "a matrix is not braced". That is true of this engine and FALSE of TeX, and the difference is worth stating where the code is read. tex.web §6738-6742: align_state is increased by 1 for a { and decreased by 1 for a }, and §7259-7264: an alignment entry ends at a tab or \cr only when align_state is back where it started. A nested matrix is protected by a real brace, an implicit one: \newenvironment{bmatrix}{\left[\env@matrix}{\endmatrix\right]} amsmath.sty:1082 \def\env@matrix{…\array{*\c@MaxMatrixCols c}} amsmath.sty:1058 \def\@array[#1]#2{…\bgroup …} latex.ltx:12101 This engine never expands \bmatrix — the maths layer parses the source text itself — so that \bgroup never arrives and the counter cannot see it. Counting the environment is the same protection by another road, and the comment now says so. The reading also turned up something the code did not do: tex.web §7492-7493 counts IMPLICIT braces, \bgroup and \egroup, in align_state. The cell scanner looked only at category-1 and -2 characters, so a & inside \bgroup…\egroup cut a cell that TeX would not have cut. implicitChar is what the box scanner already uses for \bgroup; the cell scanner now uses it too. Measured over 200 arXiv papers and 200 beamer talks: not one document changes — the corpus does not exercise it. The unit test does, on the scanner directly: with \bgroup x & y\egroup a row has two cells, and had three.

#155) \pmb does not switch font. It typesets its argument THREE times, offset by a fraction of an ex, so the ink looks heavier: \def\pmb@#1#2{\setbox8\hbox{$\m@th#1{#2}$}… \mkern-.8mu\copy8 \kern\dimen@\mkern.4mu\raise\pmbraise@\copy8 …} (amsbsy.sty:40-57) It is the fallback the package offers when no bold math version exists. One exists here: the maths layer renders \boldsymbol, which is amsbsy's own real bold (\mathversion{bold}, amsbsy.sty:28). Aliasing \pmb to it gives the formula the thing \pmb was imitating instead of dropping it — 348 of the 4172 formulas the arXiv corpus drops are this one command. Measured over 200 arXiv papers and 200 beamer talks: 4 documents change and 2 repaginate (one closer to tectonic, one further) for +3373 glyphs of formulas that were absent. The page error against the 157 tectonic renders is 670 → 669. beamer does not move.

@Protected

\newcommand{\cmd}[n][default] makes the FIRST of the n parameters optional: \cmd is \@Protected@testopt, which supplies {default} when no [ follows (latex.ltx:1187-1199, \@xargdef), and the remaining n-1 are grabbed as usual. The token path already does this (engine.go's optArg/optDefault); the maths source path read all n as MANDATORY. That is worse than not expanding at all. With the optional argument written out the macro still "matched", by taking the brackets themselves as arguments: \qbin [p]{N}{k} -> \genfrac []{0pt}{}{p}{]}_{[}{N}{k} \qbin {N}{k}+1 -> \genfrac []{0pt}{}{k}{+}_{N}1 so the formula came out silently wrong, the + of an expression becoming an argument. Without it, nothing matched and the formula was dropped — which is how a paper's own \newcommand{\qbinom}[3][q]{\genfrac[]{0pt}{}{#2}{#3}_{#1}} lost 12 equations. Measured over 200 arXiv papers and 200 beamer talks: 5 documents change for +1234 glyphs. One repaginates, 362 -> 363 against tectonic's 333 — a document already 29 pages long of it — and the page error against the 157 renders moves 669 -> 670. beamer does not move by one glyph.

Three families reached the maths layer as unknown commands, and an unknown command costs the WHOLE formula: - \setlength{\len}{dim} is \def\setlength#1#2{#1 #2\relax} (latex.ltx:7347), an assignment with two arguments — \addtolength beside it (:7348); - \hfil and its kin are \hskip with a fixed glue: primitive("hfil",hskip,fil_code) with fil_glue = 0pt plus 1fil (tex.web:20547, §3318), so they take no argument at all. The maths layer cannot stretch, so what they contribute here is nothing; - \leftskip and the other glue PARAMETERS are assignments that read a <glue> (tex.web §224), with TeX's optional equals before it, which scanMathGlue now accepts. Measured over 200 arXiv papers and 200 beamer talks: 11 documents change and 3 repaginate for +1290 glyphs; the page error against the 157 tectonic renders falls 656 -> 655. beamer gains 17 glyphs on one talk.

…#208) \def\tikzstyle#1=[#2]{} matches its delimiters as CHARACTERS, and real documents write \tikzstyle{thmbox} = [ draw=black, fill=white ] with spaces around the =. The scan then hunts a literal "=[" that never comes and takes the rest of the file with it: measured on the 157-paper corpus, 2311.08855 went from 18 pages and 38829 glyphs to ONE page and NOTHING. The stub landed in #128 without a corpus measurement; this is that measurement, and the fix. \@ifnextchar skips spaces, so the = and the bracket group are now read however they are spaced, and a \tikzstyle with neither is left alone. The paper is back to 17 pages and 38765 glyphs. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

@BeginTheorem

…#222) amsthm's starred form was unhandled. The star stopped readBraceName dead: no name was read, the environment stayed undefined, and the star and the heading were left in the input to be TYPESET. One paper opened on a spurious page of its own carrying *theoremTheorem *namedconjectureConjecture and then set the three bodies of its theorem environment as plain paragraphs. The star is now read, and a starred \newtheorem defines the same environment with no counter, no \the<env>, and a heading with no number: \@beginthmnonum, a sibling of \@BeginTheorem without its "\ #2", so the head reads "Theorem." rather than "Theorem ." — the optional note ("Theorem (Smith).") goes through the same continuation as a numbered one. 39 uses across 17 of the 200 arXiv papers. Measured over the 157 with a tectonic reference: page error 574 → 535, exact 23 → 24, within 1 page 61 → 66, within 2 pages 96 → 103; the signed bias crosses from +36 to −19. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

tannevaled merged commit 3a28ac1 into main Aug 16, 2026
18 checks passed

Navigation Menu

Uh oh!

tex: the primitives and rules real package code is built on - #2

tex: the primitives and rules real package code is built on#2
tannevaled merged 1 commit into
mainfrom
fix/pgfkeys-parse

tannevaled commented Aug 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

tannevaled commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Seven divergences from TeX, each measured before the fix

Result

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

tannevaled commented Aug 16, 2026 •

edited

Loading