engine: a file is an input LEVEL, and \begin{env} is a group

[tannevaled]

Two things this engine did not model, both from the reference, both reached through the same document: a beamer talk with a [fragile] frame.

Draft on purpose. The whole suite is green and every piece is backed by the reference, but measured over 200 real talks it is NOT yet a net win — it is blocked on #103, a defect that predates this branch and reproduces on main. Details at the end.

A file is a level of input (tex.web §537)

start_input does begin_file_reading — "set up cur_file and new level of input" — and the file is read to its end before the level underneath resumes; §329 end_file_reading pops back.

This mouth merged a file into ONE character buffer at the mouth's position, and it reads pending token lists BEFORE that buffer, so a file \input from inside a macro arrived after the rest of that macro's body:

\frame<*>[…][{…,fragile=false}]{\begingroup\input{\jobname.vrb}\endgroup}

That is how beamer re-reads a fragile frame's body, and the body arrived after the frame had closed — into a box nothing places.

pushInputLevel/popInputLevel make the file a level of its own, carrying the buffer, the mouth's position, the line table, the pending lists and the no-progress guard's baseline (it watches e.bpos, and a new buffer starts at 0 again — without that, every file after the first looked like a loop making no headway). \input, \InputIfFileExists, the .bbl reader and class/package loading all go through it, so the bookkeeping that discounted a loaded file's lines from the document's own (inputNL/loadedNL/nlCount) is gone: a file no longer shifts the document's lines because it is no longer in its buffer.

\InputIfFileExists now runs its then-code BEFORE the file, as ltfiles.dtx does: \IfFileExists{#1}{#2\@addtofilelist{#1}\@@input\@filef@und}.

\begin{env} … \end{env} is a group (ltmiscen.dtx)

\protected\def\begin#1{… \begingroup\@endpefalse\reserved@a}
  where \reserved@a is \def\@currenvir{#1}… \csname #1\endcsname
\def\end#1{\csname end#1\endcsname\@checkend{#1}\expandafter\endgroup …}

\begingroup comes first and \@currenvir is defined inside it. Without the group \@currenvir was never restored, so a class that leaves an environment early by closing its group could not — and beamer's fragile frame does exactly that (\endgroup% end environment), then calls \frame, which picks its syntax with \ifx\@currenvir\beamer@frametext. With \@currenvir still frame, the command form took the ENVIRONMENT path a second time and \beamer@doseveralframes was handed the bare \bgroup instead of the frame's body — measured, the frame went on the page empty.

The group is opened and closed by the Go-side probes \gotex@checkenv and \gotex@endenv, not by \begingroup/\endgroup tokens, because \begin/\end must stay expandable here: as tokens they print as literal text wherever \begin is only expanded (inside \message or \edef).

Four consequences, each from the reference:

• Allocation is GLOBAL. ltplain.dtx's \e@alloc ends with \global#2#6\allocationnumber; ltcounts.dtx's \@definecounter makes \cl@<c>, \p@<c> and \the<c> global; ltthm.dtx's \@xnthm/\@ynthm use \xdef and \global\@namedef. Otherwise a counter declared inside \begin{document} — an environment — vanished with it.
• Every alignment entry is a group (tex.web §791). buildCellHList terminated a cell with a bare } sentinel and no group; that stray brace was silently swallowed until a \begingroup was open.
• A Go-side environment closes its own group: it swallows its own \end, so \end never runs — tabular, tabularx, verbatim, lstlisting, equation, align, minipage and the gobbled comment environments.
• setSrcPos no longer discounts loaded lines: a level reports its own file's line, which is where a diagnostic about that file belongs.

Tests

12 new tests: the level order (a file read before the rest of the macro that asked for it, what a file leaves pending, nested files returning through each level, a long file after a long one not mistaken for a loop, \InputIfFileExists's then-code first), and the environment group (a definition ending with its environment, \@currenvir following the nesting, \endgroup leaving the environment, allocation surviving it, alignment cells, and every Go-side environment leaving no group open).

Measured — why this is a draft

200 real beamer talks and 200 arXiv papers, against main:

	pages	glyph paths	up	down
beamer	1121 → 1031	182 743 → 171 499	16	14
arXiv	4052 → 4057	9 485 942 → 9 234 865	14	7

The talks that lose are the fragile ones with a split/infolines outer theme, and the cause is #103: with those themes the verbatim capture of a fragile frame's FIRST line already loses everything after its first control sequence (\textbf{gras} is written to the .vrb as \textbf) — on main, before this branch. Correcting the ORDER puts that corrupt line back INSIDE the frame, where it swallows the next one; with the wrong order it landed outside and did less damage.

So #103 first, then this. Individually verifiable meanwhile: a fragile frame with no theme now renders its body (it produced an empty page before), and one talk in the 200 goes from 0 pages to 10.

[tannevaled]

Rebased on main with #105 merged, so the .vrb is now intact with every theme. Re-measured, same 200 talks and 200 papers:

	pages	glyph paths	up	down
beamer, #105 → this	1131 → 1088	180 959 → 180 888	18	10
arXiv, #105 → this	4051 → 4056	9 479 347 → 9 228 406	14	7

Better than before (16 up/14 down became 18 up/10 down, and one talk still goes 0 → 10 pages), but still not a net win on beamer. Two distinct remainders now, both isolated:

1. Five arXiv papers collapse to almost nothing — 2406.07418 renders 55 232 glyphs on main and 221 on this branch. Bisected: the document's body is never reached. Keeping the first N lines of its preamble, it breaks at the closing brace of a multi-line

\author{Name\\ Affiliation\\ \And \textbf{Name}\thanks{Corresponding Author}\\
  % examples of more authors
  % \And
  …
}

with neurips_2024.sty (a local file in the paper's directory). Replace that block with \author{Simple} and the branch matches main exactly, 1414 glyphs both. 116 preamble lines: 1402 glyphs; 117: 221. A reduced version of the block on its own does NOT reproduce — the class's own \author handling is part of it.

2. Ten talks lose pages — the largest 02369dc5 (33 → 17 pages, 11 073 → 5 072 glyphs) and 031d6afb (27 → 11). Not yet diagnosed.

Both are the same shape as everything else this branch has turned up: material that used to arrive somewhere harmless now arrives where it belongs, and something downstream cannot take it. Staying a draft until they are.

[tannevaled]

Chased the first remainder to the bottom and it does not yield here — recording it so nobody pays for it twice.

The five arXiv collapses are one defect, and it is reproducible in four lines. NeurIPS's style splits the author block with

\def\And{\end{tabular}\hfil\linebreak[0]\hfil\begin{tabular}[t]{c}\bf\rule{\z@}{24\p@}\ignorespaces}

used inside \begin{tabular}[t]{c}\bf\rule{\z@}{24\p@}\@author\end{tabular}. So \author{A \And B} carries an \end{tabular} that closes the table LaTeX is building, not a group of its own. Minimal case, with the paper's own neurips_2024.sty:

\documentclass{article}
\usepackage[preprint]{neurips_2024}
\title{T}\author{A \And B}
\begin{document}\maketitle CORPSVISIBLE\end{document}

main: the body renders. This branch: it does not. Bisecting the real paper's preamble lands on exactly the line carrying \And (116 preamble lines → 1402 glyphs, 117 → 221).

This engine builds tabular in Go, so by the time the cell holding \And is typeset the environment's group is already closed and the innermost group is the cell's box — and \end takes it down.

Two guards were implemented and measured, both repair arXiv and both wreck beamer:

	beamer pages	arXiv pages	the five collapses
no guard (this branch)	1088	4056	still there
\end closes only the group its own \begin opened	949	4052	fixed
\end may not close a BOX	950	4052	fixed

−140 pages either way, over 32 talks. The reason is now in the code: beamer's frame IS a box (\global\setbox\beamer@framebox=\vbox\bgroup …) and the \end that ends the frame meets it — beamer depends on closeSemiSimple's "Missing } inserted" recovery to close it. Refusing to close a box from \end is the more faithful rule and it is what TeX does; this engine is not yet in a state where it can afford it.

So the real fix is upstream of both: the Go-side tabular has to model a table that is split by \And the way \halign does — the \end{tabular} inside a cell must end that table and the \begin{tabular} that follows must start the next one, as they do in LaTeX. That is its own piece of work, and it is what stands between this branch and a merge.

Branch is back to the unguarded state (the better of the three), rebased, suite green.

[tannevaled]

#106 fixed on this branch, and it moves the needle a long way.

TeX's alignment scanner expands as it looks for & and \cr; this engine's tabular collector read raw, so an \end{tabular} carried two levels down (\@author → \And → \end) never reached it. collectTabularBody now expands a parameterless macro at depth 0 when it leads to this environment's \end directly or through one more parameterless macro — the same narrowness as before (no parameters anywhere on the path, and the \end must be the first token of the macro carrying it), so verbatim cells stay untouched.

Re-measured against main (#105):

	pages	glyph paths	up	down
beamer	1131 → 1088 (−43)	180 959 → 180 888	18	10
arXiv	4051 → 4159 (+108)	9 479 347 → 9 487 258	22	5

The five collapsed papers are back: 2406.07418 renders 55 253 glyphs against main's 55 232. arXiv is now a clear gain, and no guard on \end was needed — the fix is where the defect was.

What is left is beamer's frame box. Instrumented on 02369dc5 (33 → 17 pages): main ends with 0 open groups, this branch with 12 — five beamer@frameslide environments, each with a box group above it. \end{beamer@frameslide} runs all 24 times, but at 5 of them the innermost group is the frame's own \global\setbox\beamer@framebox=\vbox\bgroup, still open: its \egroup has not run, so the environment's group cannot close either and the frames after it are lost.

That is the last thread. It is also the same shape as #106 — a piece of beamer's own bookkeeping that this engine short-circuits — and it is where I would look next: why the frame box's \egroup is missed on those five.

[tannevaled]

Counted the group opens and closes per environment on a reduced case (two consecutive frames of 02369dc5, 8 pages against main's 12, 15 groups left open):

beamer@frameslide     ouverts=12  fermes=0
beamer@framepauses    ouverts=12  fermes=10
frame                 ouverts=4   fermes=2

\end{beamer@frameslide} never closes its group — not once in twelve slides. It runs every time; what it meets is the frame's own box, still open:

\global\setbox\beamer@framebox=\vbox\bgroup … \begin{beamer@framepauses} …
  \beamer@checkframetitle}%          <- fin du code de DEBUT de l'environnement
  {\par \end{beamer@framepauses}%    <- code de FIN
  \egroup … \box\beamer@framebox}

so the order beamer relies on is \end{beamer@framepauses} → \egroup (closes the box) → \end{beamer@frameslide}. In this engine the \egroup does not land on the box, closeSemiSimple refuses to close a semi-simple group through a box group, and the frameslide's group stays open. Each leak deepens the stack, and after a few frames the overlay bookkeeping of the NEXT frame reads stale local values — which is why frame 4 renders 6 slides on main and 2 here, and why frame 4 ALONE is identical on both.

That is the whole of the remaining beamer regression: 10 talks, −43 pages, one cause. It is beamer's frame-box bookkeeping meeting an engine that now has a group where it did not before, and it wants its own pass — the frame box has to be opened and closed by the same discipline \begin/\end now use.

arXiv is unaffected by it (+108 pages, 22 up, 5 down), so the branch is one bounded piece of work away.

[tannevaled]

Traced one level deeper: the frame box is a symptom, not the cause. Printing the group stack at each \end:

ENDENV beamer@framepauses  prof=7  pile=[k2 actionenv][k2 actionenv][k2 itemize][k2 beamer@framepauses][k1 box]
ENDENV beamer@frameslide   prof=5  pile=[k2 itemize][k2 beamer@framepauses][k1 box][k2 beamer@frameslide][k2 document]

At the first \end{beamer@framepauses} there are two actionenv and an itemize still open above it, so that \end closes an actionenv instead of its own group. Everything after is off by two, and the stack grows by two per frame — which is the leak, the unclosed frame box, and the lost slides.

actionenv, uncoverenv and altenv are beamer's overlay wrappers, one per \item<2->. Their \ends DO all run (68 actionenv closes for 68 opens), but out of order: beamer closes them through \aftergroup and \beamer@eoenv, and \beamer@eoenv is in this engine's skipped list — 7 times on this talk. beamerbaseoverlay.sty:

\long\def\beamer@@altenv#1#2#3#4#5{\alt#1{\long\def\beamer@eoenv{#3}#2}{\long\def\beamer@eoenv{#5}#4}}
\protected\def\endaltenv{\beamer@eoenv}

So the environment's closing code is a macro DEFINED by its opening code, and when that definition does not survive to \end{altenv} the wrapper never unwinds where it should. Without a group at \begin that cost nothing; with one it scrambles the nesting.

That is the next piece, and it is beamer's overlay layer rather than anything in this branch: make actionenv/uncoverenv/altenv close where beamer intends, starting with why \beamer@eoenv is missing at \endaltenv.

Branch unchanged (suite green, arXiv +108, beamer −43); this is diagnosis, recorded so the next pass starts here.

[tannevaled]

Reduced the beamer remainder to four lines. This document:

\documentclass{beamer}
\begin{document}
\begin{frame}
\begin{itemize}\item<2-> Un\item<3-> Deux\end{itemize}
\end{frame}
\end{document}

renders 3 pages on both main and this branch, but leaves 0 groups open on main and 6 here:

DESEQ beamer@frameslide    ouverts=3 fermes=1
DESEQ beamer@framepauses   ouverts=3 fermes=2
reste : [document][beamer@frameslide][boîte][beamer@framepauses][beamer@frameslide][boîte]

One overlay item per slide is enough — no theme, no fragile, no \pause. Each extra slide leaves a beamer@frameslide and its frame box behind, and after a few frames the next frame's overlay bookkeeping reads through a stack two levels deeper than beamer built.

That is the whole beamer regression on this branch, in a document small enough to step through: 2 618 group events, 6 unmatched, all in the last 250. It is where the next pass starts.

[tannevaled]

Found what leaks in the four-line case, and measured the obvious fix — it is a dead end. Recording both.

What leaks. Tracing every environment open/close, the first \item<2-> is balanced:

BEGIN actionenv → BEGIN uncoverenv → BEGIN altenv → END altenv → END uncoverenv → END actionenv

the second is not:

BEGIN actionenv (6)
 BEGIN uncoverenv (7)
  BEGIN altenv (8)
  END   itemize (prof=8)          <- \end{itemize} arrives with all three still open
 END   beamer@framepauses (7)

beamer closes an item's three overlay wrappers when the NEXT \item comes — so the LAST item's are still open at \end{itemize}, and LaTeX's single \endgroup there is what unwinds them. This engine closes one group, leaves two, and every \end after that lands one frame too high: the stack grows by two per slide.

The fix that suggests itself, measured. Make \end{name} unwind to the group its own \begin{name} opened, closing whatever an environment left behind. It does exactly what it says — the four-line case goes from 6 groups open to 0, the whole suite stays green, and 02369dc5 recovers from 17 to 24 pages. But over the corpora:

	pages	glyph paths	up	down
beamer	1131 → 1081 (−50)	180 959 → 180 996	14	13
arXiv	4051 → 3786 (−265)	9 479 347 → 8 976 736	21	5

−265 pages and half a million glyphs on arXiv. Unwinding closes too much: an environment legitimately still open inside another goes with it. Reverted; the branch stays at 0191091 (beamer −43, arXiv +108).

So the rule is neither "close the innermost" (leaves beamer's wrappers behind) nor "unwind to my own" (takes honest nesting with it). What LaTeX actually has is \endgroup plus environments that close what they opened — beamer's overlay wrappers do, through \aftergroup and \beamer@eoenv. That machinery is what this engine has to run, and it is the next thing to look at rather than another rule at \end.

[tannevaled]

Found the missing link, and it is one line of the kernel.

beamer closes an \item's three overlay wrappers when the NEXT \item comes; the LAST item's are closed by \endtrivlist, which beamer patches:

\def\endtrivlist{%
  \beamer@closeitem
  … restore …
  \beamer@origendtrivlist}

and \endtrivlist is reached through \endlist — ltlists.dtx:

\def\endlist{\global\advance\@listdepth\m@ne \endtrivlist}

In this engine \endlist was \par, so the chain stopped there: \beamer@closeitem never ran, the last item's actionenv/uncoverenv/altenv stayed open past \end{itemize}, and every \end after them closed one group too high. \endlist now ends the trivlist, and \list opens one (it was \par too, so the pair stayed balanced).

The four-line case goes from 6 groups open to 0, and over the corpora:

against main (#105)	pages	glyph paths	up	down
beamer	1131 → 1121 (−10)	180 959 → 188 933 (+7974)	18	8
arXiv	4051 → 4159 (+108)	9 479 347 → 9 486 650 (+7303)	22	5

beamer was −43 pages with 10 talks down; it is now −10 with 8, while drawing 7 974 more glyphs. Both corpora now gain content.

Pushed as bf3a93e with a kernel-level test (\list … \endlist scopes what it contains — it fails on main). Still a draft: 8 talks lose pages and several of them draw MORE glyphs while doing it (02dbc62: 1 499 → 1 592 glyphs but 15 → 8 pages), so the next question is pagination, not lost material.

[tannevaled]

Measured against the FOREIGN JUDGE — the 16 talks with a tectonic PDF beside them, so there is a ground truth to be right or wrong about:

	pages	absolute error	exact
reference (tectonic)	93	—	—
main	67	26	11/16
this branch	79	14	13/16

No talk moves away from the reference. Two move closer, and one of them (talk 9) goes from rendering 0 pages to exactly the 10 tectonic produces.

So the −10 pages over the 200-talk corpus is not a regression, it is convergence: main was producing pages the reference does not have, and this branch stops. The +7 974 glyph paths on the same corpus say the same thing from the other side — more material, on fewer pages, closer to LaTeX.

That is the bar I was holding out for, and it is met:

• against ground truth: 26 → 14 pages of error, 11 → 13 exact, nothing worse
• beamer, 200 talks: 1131 → 1121 pages, 180 959 → 188 933 glyphs, 18 up / 8 down
• arXiv, 200 papers: 4051 → 4159 pages, +7 303 glyphs, 22 up / 5 down
• suite green, 14 new tests, every piece backed by the reference (tex.web §537/§791, ltmiscen, ltplain, ltcounts, ltthm, ltlists, ltfiles)

Taking it out of draft.