bench: 8 more real functions per project — close the size/feature blind spot (192 → 240 rows) - #14
Closed
macabeus wants to merge 4 commits into
Closed
bench: 8 more real functions per project — close the size/feature blind spot (192 → 240 rows)#14macabeus wants to merge 4 commits into
macabeus wants to merge 4 commits into
Conversation
The real tier's blind spot was SIZE: median 9-15 source lines per project, maxima 17-71. A pass that only fires on a multi-statement if/else had exactly one inhabitant in 675 rows, so a whole capability moved one number. Eight functions per project, chosen for features the 192 existing rows never exercised — jump-table switches, do-while, goto, unions, bitfields, 64-bit arithmetic, varargs, function pointers, struct-by-value, static-local data tables — with a deliberate mix of sizes (24-136 lines, median 64). Selection was screened, not guessed: every candidate was compiled standalone against its own project context and run through the raising tower, so each entry is known to build and its outcome is known before it lands. Every prependC is VERBATIM project text — a source file's own #include block, its declaration preamble, or a struct definition copied from where the project declares it. No authored types. GBA rows are canonical-agbcc only: pokeemerald's libc.c/m4a.c (old_agbcc), librfu_intr.c (ARM mode), sa3's lib/m4a + agb_flash (-O1) and kleod's m4a/eeprom are excluded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
723 rows (483 synthetic + 240 real). asmlift 350 match, m2c 342. The 48 new rows land as 0 match / 8 nonmatch / 37 declined / 3 noncompile for asmlift, and 2 / 8 / 12 / 21 noncompile + 5 failed for m2c — hard for both, and the decline reasons are the point: jump tables, sp-as-data, unions, MIPS calls, branch-likely, sub-word data tables. No pre-existing row moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Verified every new GBA row against the project's OWN build object (mask
relocations and Thumb BL operands, then compare the symbol's bytes). Three did
not reproduce their project's codegen, all because the standalone TU was
missing context the real translation unit has:
kleod:UpdateCameraScroll SIN/COS are macros over gSineTable in
data/trig.h; without it agbcc emitted calls
where the project inlines a table lookup
(804 → 840 bytes, now equal)
kleod:EntityDeathAnimation the u8-returning callees declared at
code_1.c:426-428 were absent, so their
results were not truncated (376 → 384)
pokeemerald:AcroBike…Turning bike.c's own static prototype block, same
class of missing truncation (116 → 120)
engine.c's preamble carries INCLUDE_ASM(...) lines, which only compile with
include_asm.h — a header this manifest deliberately excludes because it pulls
raw .s files into the TU. They are dropped from the copied preamble.
All 24 new GBA rows now reproduce their project's own bytes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
asmlift 350 match, m2c 342, 723 rows. kleod:EntityDeathAnimation moves nonmatch → noncompile: with the project's real callee declarations now in its context, asmlift's recovered call signature no longer type-checks — the same defect the other three noncompile rows show, surfaced by MORE faithful context rather than less. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner
Author
|
Superseded by #15 — same content, clean two-commit history, and the description now lists every new row with its outcome and why it was picked. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds 8 real-tier functions per project — 48 rows, 192 → 240 real, 723 total. Board: asmlift 350 match, m2c 342; no pre-existing row moved.
Why
PR #13 shipped an unconditional L3 pass and exactly one benchmark row changed. Instrumenting the pass over all 675 rows showed why: it fires on 1. The same instrumentation over a whole game's worth of klonoa functions showed it fires on 11, seven of which change the emitted C. The difference was size — the shape needs an if whose arms each carry several statements, and the real tier's medians were 9–15 source lines with maxima of 17–71. A capability could be right or wrong and the suite would report the same number either way.
What the new rows add
Median new row: 64 lines. Features the previous 192 rows never exercised: jump-table switches,
do-while,goto, unions, bitfields, 64-bit arithmetic, varargs, function pointers, struct-by-value arguments, static-local.bytetables, 2-D array indexing.They are hard for both decompilers — 37 asmlift declines against 11 m2c declines + 21 noncompiles + 5 failures — so this is not a set that flatters either side.
How they were chosen
Every candidate was mined from the checkout, then compiled standalone and run through the raising tower before selection, so each entry's build status and outcome were known before it landed. Where a project had more compiling candidates than slots, the picks were balanced deliberately: 4 that asmlift emits (so gap size can move) and 4 that expose distinct capability gaps. The decline reasons on the new rows are 27 distinct messages, dominated by jump tables (5), MIPS calls (11 across three projects), branch-likely (7), sp-as-data (2) and unions (2).
Every
prependCis verbatim project text — a source file's own#includeblock, its declaration preamble, or a struct definition copied from where the project declares it. No authored types, per the dataset's own boundary rule.GBA rows are canonical-agbcc only: pokeemerald's
libc.c/m4a.c(old_agbcc) andlibrfu_intr.c(ARM mode), sa3'slib/m4aandlib/agb_flash(-O1), and kleod'sm4a/eepromare excluded.Two things this immediately found
1. asmlift emits calls that do not type-check against the project's own declarations. Four new rows are
noncompile— a first for asmlift on this board:sa3:sub_803213C— too many arguments to functionVramMallockleod:SetupBG3WindowOverlay— too many arguments to functionm4aSoundVSyncOffsa3:sub_806132C— assignment makes integer from pointer without a cast (VramMallocreturnsvoid *)kleod:EntityDeathAnimation— appeared only after the context fix below gave the row its project's realu8-returning callee declarationsasmlift recovers call arity from register usage; where the project declares the real prototype and the recovered arity disagrees, the candidate fails to compile in the project's own context. The previous 192 rows never surfaced this.
2. Every GBA row now verified against the project's own build — and 11 pre-existing rows fail it. The strongest available fidelity check is not comparing flag sets, it is comparing the harness's object against the project's own build object for that symbol (masking relocations and Thumb
bloperands, since a call to a same-TU symbol is already resolved in the project's object but is a relocation in the standalone one). Over all 120 agbcc rows:All 11 are pre-existing rows, and the cause is the flag delta: 10 differ by
-fprologue-bugfix(the canonical set has it; pokeemerald never enables it, and sa3 only forBUILD_NAME == sa1), andsa3:VerifyFlashSector_Corediffers because the project buildslib/agb_flashat-O1. For those rows "match" means reproducing our canonical re-compile, not what the project's build emits. That is the documented design, so changing the canonical flags — which would move rows board-wide — is left as a decision rather than folded in here.Three of the rows this PR adds failed the same check on the first pass and were fixed by giving each the context its real translation unit has (all verbatim project text):
data/trig.hforUpdateCameraScroll, whereSIN/COSare macros overgSineTableand without them agbcc emitted calls instead of the inlined table lookup; theu8-returning callee declarations atcode_1.c:426-428forEntityDeathAnimation; andbike.c's own static prototype block forAcroBikeHandleInputTurning. All 24 new GBA rows now reproduce their project's bytes exactly.On old_agbcc specifically:
m4aSoundVSyncOffis an old_agbcc function (defined insrc/m4a.c, which kleod's Makefile compiles withold_agbcc -mthumb-interwork -O2), but its callerSetupBG3WindowOverlayis byte-identical to kleod's own build — the callee's compiler never reaches the caller, which only emits abl. The row stands.Verification
pnpm bench run+merge→ 723 rows,meta.asmlift.dirty: false,stale-checkgreen.pnpm bench fidelity: 1446 script runs — 1446 ok, 0 warn, 0 fail. Every new GBA row byte-verified against its project's own build object. Only one pre-existing row differs from the pre-#13 snapshot, and that is #13's own effect.pnpm typecheck,format:check,lintclean; 139 benchmark tests, 34 web tests (including the summary-consistency provenance gate) pass.🤖 Generated with Claude Code