on_source: a module is told about every source the lexer pushes (coop patch)

[schivei]

on_source(&fn) — a module is told about every source the lexer pushes. Requested by the
teko/ngen consumer, approved by the owner on 2026-09-06. A patch: no release label.

Why

A module can already see the files it opens itself (it calls lex_include from its own
handler). What it cannot see is a #include the core resolved: do_directive is internal
and pushes without telling anybody. ngen gets free declaration order from a lexical pre-scan, so
it was blind to exactly the files the source asks for.

void on_source(uptr fn);          // handler: void f(uptr name, uptr src, i64 len)

road	who pushes it
the entry file	lex_init
#include "x.mc"	do_directive → lex_include — internal to the core
#include <name>	the bundle or a locked package — also internal
a replayed source	p_push_source

name is what lex_file() prints for that frame; src/len are the whole buffer. The handler
answers nothing.

What was decided

The entry file IS announced. lex_init pushes it before user_init() runs (src/cli.mc,
src/driver.mc and src/limits.mc are all in that order), so the natural call has already
happened when a handler can first exist. on_source closes that by replaying — at registration
time, to the newly registered handler alone, in push order — every source already open. The
rule is one sentence: a handler sees every source pushed after it registers, plus the ones
already open when it registers, and it holds for a registration made during parsing too. That
is what OF_SRC/OF_LEN are for (OF_SIZE 32 → 48): a frame keeps the buffer it was pushed
with untouched while cp walks it, so a replay hands over the whole source and not the bytes
still unread. The doc says a module need not scan the entry by hand any more.

The table is in src/hooks.mc, the call site cannot be. src/lexdump.mc includes the lexer
with arena.mc and nothing else, so the lexer must not name a symbol of hooks.mc. It is M15's
bundle shape exactly: lex_set_source_hook(&run_on_source), one function pointer, stored by the
first registration. With nothing registered the pointer is 0 and there is not even a callp.

The guard was measured before it was written. A handler runs with the frame it is being told
about already on the stack. A push from inside the callback interleaves the announcement of one
source with the opening of the next, and a handler that pushes unconditionally recursed until the
process stack was gone — reproduced with lib/user_srcpush.mc: SIGSEGV, exit 139, no
diagnostic. A frame-depth comparison after the call cannot see it, because the call never
returns; the guard is a re-entrancy flag tested at the head of lex_push_mem, so the push itself
is the error, on the first one:

mc: on_source handler pushed a source: srcpush runtime      (exit 1)

Arena tag T_ONSOURCE inserted after T_ONJUMP, T_COUNT 40 → 41, the eleven tags after it
renumbered and lim_names/lim_seeds reconciled by name (the M42 lesson: the 16 that belongs
to backends travelled with it). mc limits gains an on_source row.

Cost

133 added lines in src/, 66 of them neither comment nor blank: lex.mc +60/31,
hooks.mc +52/19, arena.mc +21/16 (twelve of those last being the renumbered tags and the two
seed rows). Five new globals — the seed's MAXGLOBALS goes from 444/512 to 449/512 (87%).

Proofs

• lib/user_syntax_demo.mc (+55) counts the sources and joins their names
  (syntax_expr("srccount"), syntax_expr("srcnames"), the latter building an N_STR node).
  scripts/check-surface.sh (+115) compiles one program that reads both back and prints them:

  /tmp/…/onsrc.mc|box runtime|/tmp/…/inc.mc|prelude        exit 42   (4 + 40 - 2)
  

  the entry (replay), the source the demo module itself pushes at the end of user_init, a
  relative #include the core resolved and a bundled #include <prelude> the core resolved —
  in push order. The default compiler refuses the same source (onsrc.mc:6: unknown name).

• The guard asserted with its exact message through lib/mc_srcpush.mc.

• Inert by construction: lib/user_source_nop.mc + lib/mc_source_nop.mc — a module whose
  only registration is on_source and whose handler does nothing — produce byte-identical
  --dump-ast and objects over the whole tests/ corpus. The four new lib/ fixtures are
  not in tools/bundle.list (the M41 precedent for check-script-only modules).

Gates

make bundle first (93 files, raw 1178425 → LZ 551503, blob 552665 B), then:

make check RC 0, zero FAIL (8m14s): budget 2848/3000, test 32/32, check-lex 149/149
(3 skipped), check-ast 150/150, check-asm 150/150, check-obj 32/32 identical to the
frozen seed, check-bundle, bootstrap at a fixed point (mc2.o == mc3.o, 1268832 B; the
--dump-asm diff between mc1 and mc2 is empty), check-surface 32/32 + the four new
cases, test-exe 32/32, check-mc 15/15, check-standalone, check-parts, check-toml 10/10,
check-build 53/53, check-pkg 85/85, check-stubs 9/9, check-sysroots (13 rows),
check-limits 17/17 under 90%, check-minimal, test-linux 41/41, test-linux-x86_64
39/39, test-linux-exe 44/44 musl + 44/44 gnu, test-linux-x86_64-exe 42/42 + 42/42,
test-windows 42/42 and test-windows-x86_64 40/40 objects cross-compiled, check-examples,
check-lang, check-conc, check-desktop, check-float, check-wide, check-kernel
(QEMU 11.0.1), check-avr, test-sandbox 55 ok / 0 failed / 1 skipped, check-docs
(198 symbols, 36 flags, 27 TOML keys, 10 directives, 51 samples, 361 links), site 89 pages

• check-site (0 link problems) + check-site-linux 11/11.

make check-linux-host RC 0 over all four cells (aarch64 musl 41/41 and gnu 42/42, x86_64
musl 39/39 and gnu 40/40), each after its own mc2l.o == mc3l.o and with the cross proof green.

scripts/check-inert.sh build/mc1.pre build/mc1 (pre = a mc1 built from origin/main):
33 objects identical (tests/*.mc and src/mc.mc) plus byte-identical artefacts for
examples/api, lang, conc, desktop and kernel — check-inert: everything identical.

The five goldens, each after its own criterion

bbf735bd3bdb58948221c945d7652be2e21980f1dc57a169e217f72f7fd6326b  build/mc2.o
f71dda4d84313e40b1a2d205a0c0314cfd61747b4dfda2fe09a76461304f5868  build/mc2l.o   (linux-arm64)
24ed180c0538715675ac1c35332ebc51506d39c9beaa21dac9c4bb659551b1c9  build/mc2l.o   (linux-x86_64)
a39a9f4d363882c58ba80b406c586f9838bf9b683443cdbaddcec71bfd2ffe2d  build/mc2w.obj (windows-arm64,   1295230 B)
54a3b2536ee550154add7e1f4c869c16861dfb6522e778fdfb8ea80fc5ffd208  build/mc2w.obj (windows-x86_64, 1330894 B)

mc2.sha256 after the empty --dump-asm diff and cmp build/mc2.o build/mc3.o; the Linux pair
deleted and re-recorded by make check-linux-host; the Windows pair cross-computed per
tests/golden/README.md, and build/mc2 writes both byte for byte as build/mc1 does.

Docs

docs/reference/hooks.md (§ 1's pipeline, § 3 is now six word registrations + six hooks that
claim none, and an on_source section with the four roads, the entry-file rule and the guard),
docs/reference/diagnostics.md (one row in § 5), docs/surface.md (the ten registrations and a
§ "Every source the lexer pushes"), CLAUDE.md § State.

stage0/ untouched: 2848/3000, git diff origin/main -- stage0/ empty.

🤖 Generated with Claude Code

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[schivei]

Merging: CI 14/14 green on cbe0811, mergeable clean, patch (no release label). Copilot answered encountered an error twice (the regenerated 1 MB src/bundle_data.mc in the diff, as on #30); per the owner's rule of 2026-09-06 the architect's read stands in: the lexer holds one function pointer (lex_set_source_hook) and a busy flag; lex_push_mem refuses a push from inside a handler (on_source handler pushed a source) before touching the frame stack, then records OF_SRC/OF_LEN and announces; on_source (hooks.mc) grows the table, installs the pointer and replays the frames already open to the new handler alone. Inertness measured (33 objects + five taught examples identical), check-obj 32/32 against the seed, four Linux cells green.