preflight: report the quality column from the check that measures it - #266
Merged
Conversation
The "Output quality" column watched for the check names {"text_baseline",
"quality"}. The quality verdict the harness records is "extraction_quality"
(checks.check_quality), and nothing has ever emitted "quality" -- so the
column never saw the check at all. What it did see was "text_baseline",
which is not a verdict: preflight.py records it PASS for every arch that
gets past the prefix probe, and its only other status (ERROR) returns before
any later check runs.
So the column reported green for a run whose quality arm was never
requested, and would have reported green for one where check_quality failed
a recall floor. A reader of the fold's headline artifact -- the matrix is
embedded in README.md and attached to the release -- was told the extraction
scores had been measured and cleared their floors, on the evidence of a
token-prefix calibration that cannot fail.
Both tracked run artifacts show it: neither ran the quality arm, and both
render "Output quality: green" before this change and "not run" after.
text_baseline is dropped rather than kept beside extraction_quality. It is
not in the documented check table at all, and a group is reported at its
weakest check -- a check that cannot fail can only inflate the group it sits
in. A run with no extraction_quality result now falls through to the
generator's existing rule for an unmeasured cell and reads "not run"
(--quality is opt-in, so that is most runs); a SKIP -- no thresholds
recorded for the arch, or a lineage carrying preflight/ without
vision_suite.py -- reads "skipped". Never PASS.
README.md is NOT regenerated here; that happens at fold time from the real
run. Its cuda row's "Output quality: green" is wrong either way:
runs/full-0332-final.json is gitignored and not in the tree, but the version
it was generated from (0.33.2-dynres-5-g2b95b4a) resolves to profile
cuda-dynres-903, and expectations.toml records [quality.*] thresholds only
for cuda-dynres-005. With no thresholds for any of that profile's three
arches check_quality can only SKIP, so the cell can never have been green:
"not run" if the fold run did not pass --quality, "skipped" if it did. The
next regeneration will say so.
Six tests go in test_verdicts.py, the no-GPU gate CI already runs twice
(once on a tree carrying preflight/ alone, which they do not depend on).
Five of them fail on the old mapping: a FAIL quality result must render
**FAIL**, an absent one "not run", a SKIP "skipped", a PASS green,
text_baseline must appear in no column, and every name in GROUPS must be one
checks.py/preflight.py emits or a recorded artifact carries -- the last is
what would have caught "quality" the day it was typed.
Also closes the artifact file handle main() leaked, which running it
in-process from the test surfaced as a ResourceWarning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uality column The checked-in matrix said "Output quality: green" for cuda. That run (runs/full-0332-final.json, PASS 20/1) recorded no extraction_quality result at all -- the profile's arches have no [quality.*] thresholds -- and the old generator read the always-PASS text_baseline calibration as the column's verdict. Regenerated from the same artifact with the fixed generator: the cell now reads what was measured. Every other cell is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
glennneuber
added a commit
that referenced
this pull request
Sep 4, 2026
…results Criteria 3 and 4 ticked with the facts. The Conflicts table gains a "resolved as" column recording what was actually done per file rather than what was planned, including the two things that differed from the plan: - 17 of the 18 mlxtest.Setup sites were in upstream-owned files and upstream ported them itself; the 18th (constrain_bench_test.go) cannot be ported because the new API takes only *testing.T, so it and constrain_test.go are deleted, bringing forward part of ADR 0033's follow-up. - server/images.go's suppressAudioCapability had silently lost its gemma4 branch in a clean auto-merge — a fifth hidden break, alongside the four the plan predicted — which would have advertised gemma4 audio this fork cannot serve. Restored. New "Excluded under D1-A" section lists the three upstream gemma4 files the merge removes (audio.go, audio_test.go, process_image.go) with what each contains, so the D1-B spike starts from it, and records the two that were kept but are dead (process_audio.go, process_audio_test.go), plus TestMLXOperationsSurviveRescheduling going with ClaimOSThread. Gate 2's numbers are recorded with their controls: 4230 pass / 0 fail / 237 skip, 188 of the skips being "MLX not available" — a skip is not a pass, and the vision goldens are among them. The one go vet finding and the one gofumpt finding are both shown to be pre-existing by a control run of origin/main in the same image. Also records that this branch is 9 commits behind origin/main (#266, #213, #268 landed after it was cut), that none of the six files they touch is touched by this merge, and that Gate 2 therefore ran without #266's six new test_verdicts.py cases (94 here vs 100 on main). ADR 0021 gets a status note: D1-A upholds it, and its decision 6 / consequence naming mlx.ClaimOSThread and mlxtest.Setup now point at mlxthreadtest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
release_matrix.pymapped the "Output quality" column to{"text_baseline", "quality"}. Nothing has ever emitted a check namedquality(the harness recordsextraction_quality,checks.py:942), andtext_baselineis the ladder's prefix calibration — recorded PASS for every arch that gets past the prefix probe, never a verdict. A column reported at its weakest check therefore read green for every run, including runs made without--qualityand runs wherecheck_qualityfailed a recall floor. Both tracked artifacts (runs/mlx-baseline-first.json,runs/preflight-mlx-metal-0330-first.json) show it:text_baselinePASS everywhere, noextraction_qualityresult, "Output quality: green".extraction_qualityonly; a run with no quality result readsnot run, a SKIP readsskipped(the generator's existing rules).test_verdicts.py(TestReleaseMatrixColumns), placed beforeunittest.main(); 5 of 6 fail against the old mapping.python3 test_verdicts.py: 100 OK.full-0332-final.json(on this host): the cuda Output quality cell now reads what was measured (no quality thresholds exist forcuda-dynres-903's arches); every other cell unchanged.Follow-up worth a decision (not changed here):
aspect_ladder,mlx_payload_pin,model_presentand the lookup checks belong to no column, so anaspect_ladderFAIL is invisible while "Image size ladder" reads green fromtoken_ladder.🤖 Generated with Claude Code