bench: hold judgement feature tags to a checkable floor - #19
Merged
Conversation
macabeus
force-pushed
the
bench/judgement-tags
branch
from
August 4, 2026 13:37
68ff8a7 to
97ff9d0
Compare
macabeus
force-pushed
the
bench/judgement-tags
branch
from
August 4, 2026 13:46
97ff9d0 to
93bc899
Compare
macabeus
force-pushed
the
bench/judgement-tags
branch
from
August 4, 2026 13:56
93bc899 to
379ef94
Compare
#16 fixed the machine-checked half of the vocabulary and left the judgement half alone, so the audits' findings there survived: `table` on functions with no array, `arithmetic` on bodies with no operator, `branch` on straight-line code. A judgement tag cannot be decided mechanically — "is this BULK memory movement?" is a human call — but most of them have a FLOOR that can be, and the fabrications all sat below it. `JUDGEMENT_FLOOR` checks that floor and `features.test.ts` asserts it; the sufficient condition stays with the reviewer. Removed (each verified against the body or the compiled reference): table af:atans_table reads no array — it calls U_GetAtanTable, and the tag came from the NAME; pokeemerald:DoForcedMovement has no `[` at all; kleod:ConfigureEntityBehavior indexes only with literals; marioparty3:DecodeData's table is the jump table, which is now a derived tag arithmetic 11 marioparty3 functions with no `+ - * / %` in the body — GWMgNoSet's whole compiled form is `lui/jr/sb/nop`; af:lbRk_IsLeapMonth is one equality test; pokeemerald:FeebasRandom's arithmetic is inside ISO_RANDOMIZE2, so it is tagged `macro` instead branch pokeemerald:StopScript is two assignments; sa3:sub_806132C has zero `cmp` across 177 lines cast kleod:ReadUnalignedU32 and snowboardkids2:loadAssetGroupResources contain no cast operator array pokeemerald:AcroBikeHandleInputTurning has no `[` struct snowboardkids2:func_8005DF10_5EB10 has no `.` or `->` global three snowboardkids2 rows and SetMauvilleOldManLanguage, each disproved by the relocation/pool evidence Added, each checked against the project's headers: `union` + `bitfield` on sa3:OamMalloc and ProcessOamBuffers (OamData is a typedef union of bitfield structs, gba/types.h:125-155), `union` on three more rows, `bitfield` on pokeemerald:TrySetCantSelectMoveBattleScript (gProtectStructs' `u32 x:1`), `fnptr` on marioparty3:HuMemAllocTag — the dataset's only `jalr`. `nested` is retired: it meant nested LOOPS in sa3 and nested IFS in af, and collided with the machine-checked `nested-loop`. `global`, `memory`, `pointer` and the type-ish tags get NO floor on purpose. kleod spells several globals as address macros, which emit a raw `.word 0x3004d84` rather than a symbol; af:gfxopen is `return gfxHead + 1;`, pointer arithmetic with no `*`, `[` or `->` anywhere. A floor that flags those would be worse than none. Three of my own floors were too strict and were caught by the corpus before they removed a correct tag: `cast` missed a lowercase project typedef (`(s_xyz*)`), `struct` demanded member access where `sizeof(struct MemBlock)` should qualify, and `pointer` missed `->` entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tags only — every outcome, score and gap size is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
macabeus
force-pushed
the
bench/judgement-tags
branch
from
August 4, 2026 15:35
379ef94 to
2b9461d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #18. #16 fixed the machine-checked half of the feature vocabulary and left the judgement half alone — so the audits' findings there survived. This closes them, and adds a floor so they cannot come back.
The idea
A judgement tag cannot be decided mechanically: is this bulk memory movement? is this arithmetic the point of the function? is a human call. But nearly every fabricated tag the audits found sat below a floor that is decidable —
tableon a function with no array,arithmeticon a body with no operator,branchon two straight-line assignments.So
JUDGEMENT_FLOORchecks the floor andfeatures.test.tsasserts it. The sufficient condition stays with the reviewer; only the indefensible is rejected.Removed
tableaf:atans_tableU_GetAtanTable(). The tag came from the name.pokeemerald:DoForcedMovement[in the body at all; it dispatches through a function-pointer parameterkleod:ConfigureEntityBehaviorgEntityInfo[0x23])marioparty3:DecodeDataarithmetic+ - * / %in the body.GWMgNoSet's entire compiled form islui/jr/sb/nopaf:lbRk_IsLeapMonthpokeemerald:FeebasRandomISO_RANDOMIZE2— retaggedmacrobranchpokeemerald:StopScriptsa3:sub_806132Ccmpacross 177 lines of assemblycastkleod:ReadUnalignedU32,snowboardkids2:loadAssetGroupResourcesarraypokeemerald:AcroBikeHandleInputTurning[structsnowboardkids2:func_8005DF10_5EB10.or->globalSetMauvilleOldManLanguageAdded
Each checked against the project's own headers:
union+bitfieldonsa3:OamMallocandsa3:ProcessOamBuffers—OamDatais atypedef unionof bitfield structs (gba/types.h:125-155), and both write.split.fractional. The dataset's two most obviously-union rows had neither tag.unionon three more rows;bitfieldonpokeemerald:TrySetCantSelectMoveBattleScript(gProtectStructs'u32 palaceUnableToUseMove:1, written five times).fnptronmarioparty3:HuMemAllocTag— the dataset's onlyjalr.nestedis retired: it meant nested loops in sa3 and nested ifs in af, and collided with the machine-checkednested-loop.What deliberately gets no floor
global,memory,pointerand the type-ish tags. kleod spells several globals as address macros (#define gStreamPtr (*(u8**)0x03004D84)), which emit a raw.word 0x3004d84rather than a symbol;af:gfxopenisreturn gfxHead + 1;— pointer arithmetic with no*,[or->in the body. A floor that flagged those would be worse than none.Three of my own floors were wrong
Caught by running them over the corpus before applying anything, not by inspection:
castmissed a lowercase project typedef —(s_xyz*)inaf:ValueSet__s_xyz.structdemanded member access, sosizeof(struct MemBlock)inpokeemerald:PutFirstMemBlockHeaderfailed it.pointermissed->entirely, flagging every struct-pointer function in snowboardkids2.All three would have removed correct tags. That is the third time in this stack a detector of mine was wrong in the same direction, which is the argument for the corpus check being part of the workflow rather than an afterthought.
Verification
743 rows,
dirty: false,stale-checkgreen, board unchanged (asmlift 354 / m2c 342) — tags only. 157 benchmark tests (2 new) and 34 web tests pass;typecheck,lint,format:checkclean.🤖 Generated with Claude Code