fix(ci): the nightly Qwen story died after six lines and every path meant to report it was broken - #2347
Merged
Merged
Conversation
…eant to report it was broken The 2026-07-31 qwen-story-daily run went red with "could not add label: 'qwen-story-daily' not found". That message is about the notifier. The actual story had failed, and four independent defects conspired to make that unsayable. 1. errexit leak (REGRESSION FROM #2344, mine). scripts/apr_bin.sh opened with `set -euo pipefail`. It is SOURCED, and `set` in a sourced file mutates the CALLER. qwen-story.sh had chosen `set -uo pipefail` WITHOUT -e on purpose: its whole design is to run all eight beats and tally failures (emit_fail / FAILED_BEATS / exit 2). Under the leaked errexit the first non-zero command aborted the run. The story died after SIX LINES inside Beat 1's ADVISORY pmat hunt. Reproduced at 12551dd (exit 5, 6 lines); with this fix the same tree runs all eight beats, 153 lines, 13 PASS / 1 FAIL / 1 SKIP in 215s. apr_bin.sh now fails by return status, callers use `|| exit 1`. 2. jq shape. `pmat query --format json` returns an ARRAY of function records, but on a semantic miss it falls back to a document search and returns {"documents":[...]}. The filter ran `.[] | \(.function)` unconditionally, so the fallback shape raised "Cannot index array with function" - jq exit 5, propagated by pipefail. That was the specific non-zero that #1 turned fatal. Now shape-guarded, and `|| true` because an advisory manifest must never fail the story it annotates. 3. manifest grep never matched. The extractor looked for '^ gap' - FOUR spaces - while the script emits EIGHT. pmat-manifest.txt was 0 lines on every run, including nights whose logs plainly contained ` churn ...` rows, which pinned `growth` at 0 forever and made the "manifest grew by >5" alert branch unreachable. 4. the verdict was gated behind the notifier. Neither `qwen-story-daily` nor `regression` existed as repo labels, so `gh issue create --label` had never once succeeded. Worse, a step `if:` without a status function carries an implicit `success()`, so that failure SKIPPED "Fail the job if the story failed". The job went red for the wrong reason and by accident. Labels are now created idempotently, the notifier is continue-on-error, the verdict step is `always() &&`, and a new step fails closed when no verdict was produced at all. Ratchet: scripts/check_sourced_libs_option_neutral.sh. Both files in #1 read correctly in isolation - only the combination is wrong, which is why this is mechanical rather than a review note. Mutation-verified against the real pre-fix header: RED with it, GREEN without. Wired into ci.yml's text-only guard job with --self-test. An earlier draft of that guard also scanned workflows and flagged check_format_sovereignty.sh, because ci.yml contains the comment "# catch dev-dep cycles. See scripts/check_format_sovereignty.sh" and the sentence-ending period parsed as the source builtin. Scope is now script-to-script only; a workflow `run:` block is its own throwaway shell. Blast radius: one nightly. #2344 landed after the 07-30 run (green at e514cc5); 07-31 was the first run to source apr_bin.sh. The revived story surfaces one real finding not addressed here: B2 `apr qa` Golden Output now fails on the 1.5B model where it passed on 07-30. No commit in that range touches inference, so it is filed separately rather than assumed to be a regression. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
noahgift
enabled auto-merge
July 31, 2026 14:42
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 31, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 2026-07-31
qwen-story-dailyrun went red withcould not add label: 'qwen-story-daily' not found. That message is about the notifier. The story itself had failed, and four independent defects conspired to make that unsayable.1. errexit leak — a regression from #2344 (mine)
scripts/apr_bin.shopened withset -euo pipefail. It is sourced, andsetin a sourced file mutates the caller's shell.qwen-story.shhad chosenset -uo pipefailwithout-edeliberately — its entire design is to run all eight beats and tally failures (emit_fail/FAILED_BEATS/exit 2). Under the leaked errexit, the first non-zero command aborted the run.Reproduced at
12551dd19: exit 5, six lines, dead inside Beat 1's advisory pmat hunt. With this fix, the same tree runs all eight beats — 153 lines, 13 PASS / 1 FAIL / 1 SKIP, 215s.apr_bin.shnow fails by return status; callers use. scripts/apr_bin.sh || exit 1, which is fail-closed in exactly the same way without seizing the caller's shell.2. jq shape mismatch
pmat query --format jsonreturns an array of function records — but on a semantic miss it falls back to a document search and returns{"documents":[...]}. The filter ran.[] | \(.function)unconditionally, so the fallback shape raised "Cannot index array with function" → jq exit 5, propagated bypipefail. That was the specific non-zero that defect 1 turned fatal.Now shape-guarded, and
|| true— an advisory manifest must never be able to fail the story it annotates.3. The manifest grep never matched anything
The extractor looked for
^ gap(four spaces); the script emits eight.pmat-manifest.txtwas 0 lines on every run — including nights whose logs plainly containedchurn ...rows. That pinnedgrowthat 0 forever and made the "manifest grew by >5" alert branch unreachable.4. The verdict was gated behind the notifier
Neither
qwen-story-dailynorregressionexisted as repo labels, sogh issue create --labelhad never once succeeded. Worse: a stepif:without a status function carries an implicitsuccess(), so that failure skippedFail the job if the story failed. The job went red for the wrong reason, by accident.Labels are now created idempotently, the notifier is
continue-on-error, the verdict step isalways() && ..., and a new step fails closed when no verdict was produced at all.Ratchet
scripts/check_sourced_libs_option_neutral.sh. Both files in defect 1 read correctly in isolation — only the combination is wrong, which is why this is mechanical rather than a review note.Mutation-verified against the real pre-fix header: RED with it, GREEN without. Wired into ci.yml's text-only guard job with
--self-test.An earlier draft also scanned workflows and flagged
check_format_sovereignty.sh, because ci.yml contains the comment# catch dev-dep cycles. See scripts/check_format_sovereignty.sh— the sentence-ending period parsed as the source builtin. Scope is now script-to-script only; a workflowrun:block is its own throwaway shell.Blast radius
One nightly. #2344 landed after the 07-30 run (green at
e514cc5ed); 07-31 was the first run to sourceapr_bin.sh.Not fixed here
The revived story surfaces a real finding: B2
apr qaGolden Output now fails on the 1.5B model where it passed on 07-30. No commit in that range touches inference, so it is filed separately rather than assumed to be a regression.🤖 Generated with Claude Code