bench: 8 more real functions per project — close the size/feature blind spot (192 → 240 rows) - #15
Merged
Merged
Conversation
The real tier's blind spot was SIZE: median 9-15 source lines per project, maxima 17-71. A pass that only fires on a multi-statement if/else had exactly one inhabitant in 675 rows, so a whole capability moved one number. Eight functions per project, chosen for features the 192 existing rows never exercised — jump-table switches, do-while, goto, unions, bitfields, 64-bit arithmetic, varargs, function pointers, static-local data tables, 2-D array indexing — with a deliberate mix of sizes (24-136 lines, median 64). Selection was screened, not guessed: every candidate was compiled standalone against its own project context and run through the raising tower, so each entry is known to build and its outcome is known before it lands. Where a project had more compiling candidates than slots, the picks are balanced between functions asmlift emits (so gap size can move) and functions that expose distinct capability gaps. Every prependC is VERBATIM project text — a source file's own #include block, its declaration preamble, or a struct definition copied from where the project declares it. No authored types. Three rows needed context their real translation unit has and the manifest headers do not: data/trig.h for UpdateCameraScroll (SIN/COS are macros over gSineTable — without them agbcc emits calls where the project inlines a table lookup), the u8-returning callee declarations at code_1.c:426-428 for EntityDeathAnimation, and bike.c's own static prototype block for AcroBikeHandleInputTurning. kleod's per-file #include blocks cannot be copied wholesale: include_asm.h pulls raw .s files into the TU, and engine.c's preamble carries INCLUDE_ASM(...) lines. GBA rows are canonical-agbcc only: pokeemerald's libc.c/m4a.c (old_agbcc), librfu_intr.c (ARM mode), sa3's lib/m4a + agb_flash (-O1) and kleod's m4a/eeprom are excluded. All 24 new GBA rows were verified to reproduce their project's OWN build object byte-for-byte (relocations and Thumb bl operands masked). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
723 rows (483 synthetic + 240 real). asmlift 350 match, m2c 342. The 48 new rows land as 0 match / 7 nonmatch / 37 declined / 4 noncompile for asmlift, and 2 / 9 / 11 declined / 21 noncompile / 5 failed for m2c — hard for both, and the decline reasons are the point: MIPS calls, branch-likely, jump tables, sp-as-data, unions, sub-word data tables. No pre-existing row moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 'bitfield' tag on three new rows came from a selection-time detector whose
regex matched TERNARIES (`? 6 : 2;`), not bitfield declarations. None of the
three touches a C bitfield:
pokeemerald:AnimTask_FlashHealthboxOnLevelUp_Step only reaches
gTasks[].data[] — struct Task is all scalars
sa3:sub_802DFC8, sa3:sub_803213C SPRITE_FLAG is
`((value) << SF_SHIFT(name))` (sprite.h:399) — a shift/OR into a u32
Retagged to what they actually exercise. Conversely kleod:EntityItemDrop reads
a genuine bitfield (`u32 unk_204_2 : 2` in its preamble) and was NOT tagged;
it is now, and it is the only real bitfield row in the set.
The rows stand on their other merits — sub_802DFC8 is still the only
branch-target-is-a-data-label decline — but the dataset should not claim
coverage it does not have.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tags only — every outcome, score and gap size is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…TU has The row's prependC copied code_1.c:404-428 (the EntityDeathStruct typedef plus the three callee prototypes just above the function), but `m4aSongNumStart` is declared at code_1.c:30 — outside that span. So the row compiled with an implicit declaration, and agbcc's warning about it became the FIRST diagnostic line, which is the one that survives to the published marker. That made the row's failure read as `warning: implicit declaration of function m4aSongNumStart`, blaming a benign warning the reference build also emits (and exits 0 on). With the prototype present the marker names the actual error: `too many arguments to function m4aSongNumStart` — the same call-arity defect the other three noncompile rows show. Only the one prototype this row calls: the sibling m4a declarations reference `struct MP2KPlayerState`, which this context does not declare, and would trade one spurious warning for another. Byte-verified unchanged against the project's own build (384 bytes, equal). m2c moves declined -> noncompile on the same context. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only that row moves: its marker now names the real error, and m2c goes declined -> noncompile on the same richer context (m2c declined 33 -> 32, noncompile 144 -> 145). asmlift's board is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds 8 real-tier functions per project — 48 rows, 192 → 240 real, 723 total. Board: asmlift 350 match, m2c 342; no pre-existing row moved.
Why
PR #13 shipped an unconditional L3 pass and exactly one benchmark row changed. Instrumenting the pass over all 675 rows showed why: it fires on 1. The same instrumentation over a whole game's worth of klonoa functions showed it fires on 11, seven of which change the emitted C. The difference was size — the shape needs an
ifwhose arms each carry several statements, and the real tier's medians were 9–15 source lines with maxima of 17–71. A capability could be right or wrong and the suite would report the same number either way.How the rows were chosen
Every candidate was mined from the checkout, then compiled standalone and run through the raising tower before selection, so each entry's build status and outcome were known before it landed. Hit rates were low enough that this mattered — roughly 60/93 for kleod, 44/392 for sa3, 33/150 for pokeemerald, 15/160 for af, 16/70 for marioparty3, 13/70 for snowboardkids2. These counts come from a throwaway screening script that is not in the repo, so take them as reported, not verified; what is verifiable is every row's landed outcome, below.
Where a project had more compiling candidates than slots, the picks are balanced between functions asmlift emits (so gap size can move) and functions that expose distinct capability gaps. On the three MIPS projects that balance was not available: screening ~300 candidates produced zero that asmlift lifts, because essentially every real MIPS function contains a call or a branch-likely. (Same caveat — the screen is not committed. The consequence is visible in the tables: all 24 MIPS rows decline, and every decline is a
jal/jalror a branch-likely.)Every
prependCis verbatim project text — a source file's own#includeblock, its declaration preamble, or a struct copied from where the project declares it. No authored types.All 24 new GBA rows were verified to reproduce their project's own build object byte-for-byte (relocations and Thumb
bloperands masked, since a call to a same-TU symbol is already resolved in the project's object).The 48 new rows
Outcome key:
diff:N= compiles, objdiff distance N.declined= explicit incompleteness markers.noncompile= marker-free output that fails to build.failed= no usable output.kleod — agbcc (prior rows: max 58 lines)
ProcessInputAndUpdateEntitiesUpdateCameraScrollScrollBGLayer, a shape no row had, and it is the only row that declines on a symbol+offset pool word. (It was also picked as a/coalesceinhabitant, but it declines at lift, so it never reaches L3 — that motivation did not pan out.)ConfigureEntityBehaviormov pc, r0jump-table switch at 94 lines. kleod's one prior switch row (TransformSingleEntityToScreen) is also a jump table with the same decline, so this adds size, not a new gap.CountCollectedGemsUpdateWorldMapNodeAnimEntityItemDropu32 unk_204_2 : 2) — the only real bitfield row in the set.SetupBG3WindowOverlay/coalesceinhabitant from the klonoa mining (candidate counts not recorded in the row), plus array writes into VRAM-shaped memory. Lands asnoncompile— see the findings below.EntityDeathAnimationStrCpy), so this is depth, not a new feature — and thenoncompileit produces is the useful part.pokeemerald — agbcc (prior rows: max 25 lines)
TrySetCantSelectMoveBattleScriptGetMoveTargetSetMauvilleOldManLanguageunion OldMan *and switches on its tag — no prior row on any project (real or synthetic) mentionsunion. Two other new rows reach unions too (af:_MtxF_to_Mtx,sa3:sub_804DC38); this is the one that both compiles and scores.Cmd_tryconversiontypechangeUpdateShoalTideFlag.bytetable.bytetable; the only row of its kind.AcroBikeHandleInputTurningu8-returning static callees whose return truncations are load-bearing; also the closest head-to-head on the new rows.AnimTask_FlashHealthboxOnLevelUp_Steps16 data[]instruct Task), which its union modelling declines. Nothing in the source is a bitfield.DoForcedMovementvoid (*moveFunc)(u8)), compiled tobl _call_via_r9.sa3 — agbcc (prior rows: no switch at all, max 71 lines)
PackSaveSectorsub_804DC38Sio32MultiLoadMainsub_802DFC8sub_806132Cnoncompileinstead, which is the finding below.sub_804D360Sio32MultiLoadIntrsub_803213Cnoncompile. (Also selected on a false bitfield reading — see below.)af — ido7.1 (prior rows: max 31 lines)
Every af pick declines, with six distinct reasons. af's asm is FP-heavy and call-heavy, which is exactly what asmlift's MIPS frontend does not model yet.
Skin_Matrix_MulMatrixsdc1store_MtxF_to_Mtxlwc1loadMtxFandMtxare both unions, indexed as 2-D arrays, with float↔fixed bit twiddling.suMtxMakeSRTjaladd_calc_abc1tlget_percent_forAccelBrakebc1flSkin_Matrix_SetQuaternionsdc1sAdo_Get_KokeruLabeljalatans_tablebc1fbc1frather than a branch-likely. Despite the name it reads no table — it callsU_GetAtanTable.marioparty3 — gcc2.7.2 (prior rows: max 17 lines, six features total)
The narrowest project in the suite: its 32 rows only ever exercised struct, pointer, branch, global, arithmetic and loop.
ContDataUpdateblezlfunc_80012B14_13714jalHu3DCam3DToScreenjalfunc_800230F8_23CF8beqlHuSprGrpCreatejalDecodeDatajaljr v0), a shape this project never had — but asmlift declines earlier, on thejalat 0x40, so the table is not what this row currently exercises.FileSeekbeqzlHuMemAllocTagjalrjaldeclines.snowboardkids2 — gcc2.7.2kmc (prior rows: max 52 lines)
func_8005DF10_5EB10jalfunc_8001E104_1ED04bnezlfunc_8005AE8C_5BA8Cjalfunc_800B2C18_A2AC8jalfunc_8005AB58_5B758jalfunc_800B3F2C_A3DDCjalfunc_800B3784_A3634beqzlfunc_80038090_38C90jalCorrections after an adversarial review of this description
A hostile fact-check of the table above (48 rows, every claim checked against the manifests, the project checkouts and
results.json) found six false rationales. They are fixed above, and one of them was a dataset bug, not just prose:bitfieldand are not. The selection-time detector's regex matched ternaries (? 6 : 2;).struct Taskis all scalars, and sa3'sSPRITE_FLAGis((value) << SF_SHIFT(name))— a shift into a plainu32. Retagged in a separate commit; converselykleod:EntityItemDropreads a genuineu32 unk_204_2 : 2and was not tagged, so it now is. Outcomes and scores are unchanged.kleodalready had a do-while row (StrCpy) and its prior switch row (TransformSingleEntityToScreen) is already amov pc, r0jump table with the identical decline marker. Those two picks add size and depth, not new gaps.marioparty3:DecodeDatadeclines onjal, not on its jump table — the table is there, but asmlift never reaches it.pokeemerald:DoForcedMovementdispatches through a function-pointer parameter, not a table.kleod:UpdateCameraScrollcannot exercise PR feat(l3): common-tail merge, scoped base clusters, and ranked live-range coalescing #13's coalescing lever — it declines at lift, so it never reaches L3.sp-as-datarows were described as having address-taken locals; neither takes an address. The marker's parenthetical is a disjunction and the real cause is spill slots.What this immediately found
asmlift emits calls that do not type-check against the project's own declarations. Four new rows are
noncompile— a first for asmlift on this board:sa3:sub_803213C— too many arguments to functionVramMallockleod:SetupBG3WindowOverlay— too many arguments to functionm4aSoundVSyncOffsa3:sub_806132C— assignment makes integer from pointer without a cast (VramMallocreturnsvoid *)kleod:EntityDeathAnimation— surfaced only after the row was given its project's realu8-returning callee declarationsasmlift recovers call arity from register usage; where the project declares the real prototype and the recovered arity disagrees, the candidate fails to compile in the project's own context. The previous 192 rows never surfaced this.
11 pre-existing rows do not reproduce their project's own build. Auditing all 120 agbcc rows against the projects' build objects: kleod 40/40 equal, pokeemerald 32 equal / 8 differ, sa3 37 equal / 3 differ. Ten differ by
-fprologue-bugfix(the canonical flag set has it; pokeemerald never enables it, and sa3 only forBUILD_NAME == sa1), andsa3:VerifyFlashSector_Corebecause the project buildslib/agb_flashat-O1. For those rows "match" means reproducing our canonical re-compile rather than what the project's build emits. That is the documented design, so changing the canonical flags — which would move rows board-wide — is left as a decision rather than folded in here.On old_agbcc:
m4aSoundVSyncOffis an old_agbcc function (defined insrc/m4a.c, which kleod builds withold_agbcc -mthumb-interwork -O2), but its callerSetupBG3WindowOverlayis byte-identical to kleod's own build — the callee's compiler never reaches the caller, which only emits abl. Only the TU a function is defined in matters for eligibility.Verification
pnpm bench run+merge→ 723 rows,meta.asmlift.dirty: false,stale-checkgreen.pnpm bench fidelity: 1446 script runs — 1446 ok, 0 warn, 0 fail. Only one pre-existing row differs from the pre-#13 snapshot, and that is #13's own effect.pnpm typecheck,format:check,lintclean; 139 benchmark tests and 34 web tests (including the summary-consistency provenance gate) pass. A second full run on this branch reproduced every row measurement-identically.Supersedes #14 (same content, clean two-commit history).
🤖 Generated with Claude Code