Skip to content

test: re-baseline the evaluation benchmark - #203

Merged
vicmaster merged 1 commit into
masterfrom
benchmark-rebaseline
Aug 17, 2026
Merged

test: re-baseline the evaluation benchmark#203
vicmaster merged 1 commit into
masterfrom
benchmark-rebaseline

Conversation

@vicmaster

Copy link
Copy Markdown
Owner

npx tsx benchmark/run.ts has been failing on clean master since the baselines were last regenerated in Phase 12. Everything from Phases 13 through 29 was unaccounted for, so this guard has been dark for a long stretch — including the whole of Phase 29, where I hit it repeatedly and each time had to re-verify it was pre-existing rather than mine.

Every delta, attributed

The point of a baseline is to catch unintended evaluator drift, so each changed number had to be explainable by a known change rather than accepted in bulk:

delta cause
new category usability Phase 27 slice C
new category coverage Phase 24 slice C
8 APCA advisories across the corpus Phase 25 — info-only, never gates
4 × "Display text at 28px with default tracking" Phase 25 tracking advisory
radius census 5 → 4 distinct Phase 27 slice B excluded pill radii; the hero's cornerRadius: 999 pill stopped counting
eyebrow census 4 → 3 Phase 28 slice C's detectTable three-row rule now finds a table in the hero, and a table's header is excluded from the census by design — the same detection produces the two new coverage findings
spacing message reworded Phase 29 slice D — variety is measured against the declared scale, and corpus canvases declare none, so they take the no-scale branch

Two of these took actual digging. The eyebrow count dropping by exactly one looked arbitrary until the coverage findings gave it away: the same table detection explains both. I checked it rather than waving at "Phase 28 changed the tells".

The corpus still discriminates

98 / 82 / 72 across minimal, bad-contrast and hero — the spread that makes the benchmark useful is intact.

One finding worth more than the re-baseline

bad-contrast rose 77 → 82 without improving. Its color score is still 0:

  color           0 ->   0        ← the fixture's actual defect, unchanged
  usability       — -> 100        ← new category
  coverage        — -> 100        ← new category

The rise is dilution. Two new categories scoring 100 lifted the weighted average while the fixture's real problem was untouched.

That generalises: adding an evaluator category raises the score of every screen that doesn't exercise it. So scores are not comparable across phases that changed the category set, and the > 95 bar has quietly become easier to reach as categories accumulated — eight now, against five when that bar was set. Not a bug, and not something to fix in this PR, but it is the kind of thing a dark benchmark hides. Flagging it rather than burying it in a baselines diff.

Verification

benchmark/run.ts passes. test-evaluate 20, test-discoverability 117/117 — the change is data-only, but those are the two that would notice.

benchmark/run.ts has been failing on clean master since the baselines
were last regenerated in Phase 12. Everything from Phases 13–29 was
unaccounted for, so the guard has been dark for a long stretch —
including the whole of Phase 29.

Every delta traces to a deliberate change, checked one at a time rather
than accepted in bulk:

  new category: usability          Phase 27 slice C
  new category: coverage           Phase 24 slice C
  APCA advisories (8 across the    Phase 25 (info-only, never gates)
    corpus)
  "Display text at 28px with       Phase 25 tracking advisory
    default tracking" (4)
  radius census 5 -> 4 distinct    Phase 27 slice B excluded pill radii;
                                   the hero's cornerRadius: 999 pill
                                   stopped counting
  eyebrow census 4 -> 3            Phase 28 slice C's detectTable
                                   three-row rule now finds a table in
                                   the hero, whose header is excluded
                                   from the census by design (the same
                                   detection produces the two new
                                   coverage findings)
  spacing message reworded         Phase 29 slice D — variety is measured
                                   against the DECLARED scale, and corpus
                                   canvases declare none, so they take
                                   the no-scale branch

The corpus keeps its discriminating power: 98 / 82 / 72 across minimal,
bad-contrast and hero.

WORTH KNOWING, and not a bug: bad-contrast rose 77 -> 82 without
improving. Its color score is still 0. The rise is dilution — two new
categories scoring 100 lifted the weighted average while the fixture's
actual defect was untouched. Adding categories makes every screen that
does not exercise them score higher, so scores are not comparable across
phases that changed the category set.
@vicmaster
vicmaster merged commit 8ff503a into master Aug 17, 2026
@vicmaster
vicmaster deleted the benchmark-rebaseline branch August 17, 2026 19:34
@vicmaster vicmaster mentioned this pull request Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant