Skip to content

bench: 8 more real functions per project — close the size/feature blind spot (192 → 240 rows) - #14

Closed
macabeus wants to merge 4 commits into
mainfrom
bench/expand-real-tier
Closed

bench: 8 more real functions per project — close the size/feature blind spot (192 → 240 rows)#14
macabeus wants to merge 4 commits into
mainfrom
bench/expand-real-tier

Conversation

@macabeus

@macabeus macabeus commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Adds 8 real-tier functions per project — 48 rows, 192 → 240 real, 723 total. Board: asmlift 350 match, m2c 342; no pre-existing row moved.

Why

PR #13 shipped an unconditional L3 pass and exactly one benchmark row changed. Instrumenting the pass over all 675 rows showed why: it fires on 1. The same instrumentation over a whole game's worth of klonoa functions showed it fires on 11, seven of which change the emitted C. The difference was size — the shape needs an if whose arms each carry several statements, and the real tier's medians were 9–15 source lines with maxima of 17–71. A capability could be right or wrong and the suite would report the same number either way.

What the new rows add

project new rows (loc) prior max asmlift m2c
af 36–122 31 8 declined 7 nonmatch, 1 declined
kleod 50–136 58 2 nonmatch, 4 declined, 2 noncompile 3 nonmatch, 2 declined, 3 failed
marioparty3 24–50 17 8 declined 7 nonmatch, 1 declined
pokeemerald 29–105 25 4 nonmatch, 4 declined 4 nonmatch, 3 declined, 1 failed
sa3 30–117 71 3 nonmatch, 3 declined, 2 noncompile 2 match, 5 nonmatch, 1 failed
snowboardkids2 79–125 52 8 declined 3 nonmatch, 5 declined

Median new row: 64 lines. Features the previous 192 rows never exercised: jump-table switches, do-while, goto, unions, bitfields, 64-bit arithmetic, varargs, function pointers, struct-by-value arguments, static-local .byte tables, 2-D array indexing.

They are hard for both decompilers — 37 asmlift declines against 11 m2c declines + 21 noncompiles + 5 failures — so this is not a set that flatters either side.

How they were chosen

Every candidate was mined from the checkout, then compiled standalone and run through the raising tower before selection, so each entry's build status and outcome were known before it landed. Where a project had more compiling candidates than slots, the picks were balanced deliberately: 4 that asmlift emits (so gap size can move) and 4 that expose distinct capability gaps. The decline reasons on the new rows are 27 distinct messages, dominated by jump tables (5), MIPS calls (11 across three projects), branch-likely (7), sp-as-data (2) and unions (2).

Every prependC is verbatim project text — a source file's own #include block, its declaration preamble, or a struct definition copied from where the project declares it. No authored types, per the dataset's own boundary rule.

GBA rows are canonical-agbcc only: pokeemerald's libc.c/m4a.c (old_agbcc) and librfu_intr.c (ARM mode), sa3's lib/m4a and lib/agb_flash (-O1), and kleod's m4a/eeprom are excluded.

Two things this immediately found

1. asmlift emits calls that do not type-check against the project's own declarations. Four new rows are noncompile — a first for asmlift on this board:

  • sa3:sub_803213Ctoo many arguments to function VramMalloc
  • kleod:SetupBG3WindowOverlaytoo many arguments to function m4aSoundVSyncOff
  • sa3:sub_806132Cassignment makes integer from pointer without a cast (VramMalloc returns void *)
  • kleod:EntityDeathAnimation — appeared only after the context fix below gave the row its project's real u8-returning callee declarations

asmlift recovers call arity from register usage; where the project declares the real prototype and the recovered arity disagrees, the candidate fails to compile in the project's own context. The previous 192 rows never surfaced this.

2. Every GBA row now verified against the project's own build — and 11 pre-existing rows fail it. The strongest available fidelity check is not comparing flag sets, it is comparing the harness's object against the project's own build object for that symbol (masking relocations and Thumb bl operands, since a call to a same-TU symbol is already resolved in the project's object but is a relocation in the standalone one). Over all 120 agbcc rows:

project equal differ
kleod 40 0
pokeemerald 32 8
sa3 37 3

All 11 are pre-existing rows, and the cause is the flag delta: 10 differ by -fprologue-bugfix (the canonical set has it; pokeemerald never enables it, and sa3 only for BUILD_NAME == sa1), and sa3:VerifyFlashSector_Core differs because the project builds lib/agb_flash at -O1. For those rows "match" means reproducing our canonical re-compile, not what the project's build emits. That is the documented design, so changing the canonical flags — which would move rows board-wide — is left as a decision rather than folded in here.

Three of the rows this PR adds failed the same check on the first pass and were fixed by giving each the context its real translation unit has (all verbatim project text): data/trig.h for UpdateCameraScroll, where SIN/COS are macros over gSineTable and without them agbcc emitted calls instead of the inlined table lookup; the u8-returning callee declarations at code_1.c:426-428 for EntityDeathAnimation; and bike.c's own static prototype block for AcroBikeHandleInputTurning. All 24 new GBA rows now reproduce their project's bytes exactly.

On old_agbcc specifically: m4aSoundVSyncOff is an old_agbcc function (defined in src/m4a.c, which kleod's Makefile compiles with old_agbcc -mthumb-interwork -O2), but its caller SetupBG3WindowOverlay is byte-identical to kleod's own build — the callee's compiler never reaches the caller, which only emits a bl. The row stands.

Verification

pnpm bench run + merge → 723 rows, meta.asmlift.dirty: false, stale-check green. pnpm bench fidelity: 1446 script runs — 1446 ok, 0 warn, 0 fail. Every new GBA row byte-verified against its project's own build object. Only one pre-existing row differs from the pre-#13 snapshot, and that is #13's own effect. pnpm typecheck, format:check, lint clean; 139 benchmark tests, 34 web tests (including the summary-consistency provenance gate) pass.

🤖 Generated with Claude Code

macabeus and others added 4 commits August 3, 2026 23:44
The real tier's blind spot was SIZE: median 9-15 source lines per project,
maxima 17-71. A pass that only fires on a multi-statement if/else had exactly
one inhabitant in 675 rows, so a whole capability moved one number.

Eight functions per project, chosen for features the 192 existing rows never
exercised — jump-table switches, do-while, goto, unions, bitfields, 64-bit
arithmetic, varargs, function pointers, struct-by-value, static-local data
tables — with a deliberate mix of sizes (24-136 lines, median 64).

Selection was screened, not guessed: every candidate was compiled standalone
against its own project context and run through the raising tower, so each
entry is known to build and its outcome is known before it lands.

Every prependC is VERBATIM project text — a source file's own #include block,
its declaration preamble, or a struct definition copied from where the project
declares it. No authored types.

GBA rows are canonical-agbcc only: pokeemerald's libc.c/m4a.c (old_agbcc),
librfu_intr.c (ARM mode), sa3's lib/m4a + agb_flash (-O1) and kleod's
m4a/eeprom are excluded.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
723 rows (483 synthetic + 240 real). asmlift 350 match, m2c 342.

The 48 new rows land as 0 match / 8 nonmatch / 37 declined / 3 noncompile for
asmlift, and 2 / 8 / 12 / 21 noncompile + 5 failed for m2c — hard for both, and
the decline reasons are the point: jump tables, sp-as-data, unions, MIPS calls,
branch-likely, sub-word data tables.

No pre-existing row moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Verified every new GBA row against the project's OWN build object (mask
relocations and Thumb BL operands, then compare the symbol's bytes). Three did
not reproduce their project's codegen, all because the standalone TU was
missing context the real translation unit has:

  kleod:UpdateCameraScroll        SIN/COS are macros over gSineTable in
                                  data/trig.h; without it agbcc emitted calls
                                  where the project inlines a table lookup
                                  (804 → 840 bytes, now equal)
  kleod:EntityDeathAnimation      the u8-returning callees declared at
                                  code_1.c:426-428 were absent, so their
                                  results were not truncated (376 → 384)
  pokeemerald:AcroBike…Turning    bike.c's own static prototype block, same
                                  class of missing truncation (116 → 120)

engine.c's preamble carries INCLUDE_ASM(...) lines, which only compile with
include_asm.h — a header this manifest deliberately excludes because it pulls
raw .s files into the TU. They are dropped from the copied preamble.

All 24 new GBA rows now reproduce their project's own bytes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
asmlift 350 match, m2c 342, 723 rows. kleod:EntityDeathAnimation moves
nonmatch → noncompile: with the project's real callee declarations now in its
context, asmlift's recovered call signature no longer type-checks — the same
defect the other three noncompile rows show, surfaced by MORE faithful context
rather than less.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@macabeus

macabeus commented Aug 4, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #15 — same content, clean two-commit history, and the description now lists every new row with its outcome and why it was picked.

@macabeus macabeus closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant