Skip to content

fix(ci): the nightly Qwen story died after six lines and every path meant to report it was broken - #2347

Merged
noahgift merged 1 commit into
mainfrom
fix/qwen-story-errexit-leak
Jul 31, 2026
Merged

fix(ci): the nightly Qwen story died after six lines and every path meant to report it was broken#2347
noahgift merged 1 commit into
mainfrom
fix/qwen-story-errexit-leak

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

The 2026-07-31 qwen-story-daily run went red with could not add label: 'qwen-story-daily' not found. That message is about the notifier. The story itself had failed, and four independent defects conspired to make that unsayable.

1. errexit leak — a regression from #2344 (mine)

scripts/apr_bin.sh opened with set -euo pipefail. It is sourced, and set in a sourced file mutates the caller's shell. qwen-story.sh had chosen set -uo pipefail without -e deliberately — its entire design is to run all eight beats and tally failures (emit_fail / FAILED_BEATS / exit 2). Under the leaked errexit, the first non-zero command aborted the run.

Reproduced at 12551dd19: exit 5, six lines, dead inside Beat 1's advisory pmat hunt. With this fix, the same tree runs all eight beats — 153 lines, 13 PASS / 1 FAIL / 1 SKIP, 215s.

apr_bin.sh now fails by return status; callers use . scripts/apr_bin.sh || exit 1, which is fail-closed in exactly the same way without seizing the caller's shell.

2. jq shape mismatch

pmat query --format json returns an array of function records — but on a semantic miss it falls back to a document search and returns {"documents":[...]}. The filter ran .[] | \(.function) unconditionally, so the fallback shape raised "Cannot index array with function" → jq exit 5, propagated by pipefail. That was the specific non-zero that defect 1 turned fatal.

Now shape-guarded, and || true — an advisory manifest must never be able to fail the story it annotates.

3. The manifest grep never matched anything

The extractor looked for ^ gap (four spaces); the script emits eight. pmat-manifest.txt was 0 lines on every run — including nights whose logs plainly contained churn ... rows. That pinned growth at 0 forever and made the "manifest grew by >5" alert branch unreachable.

4. The verdict was gated behind the notifier

Neither qwen-story-daily nor regression existed as repo labels, so gh issue create --label had never once succeeded. Worse: a step if: without a status function carries an implicit success(), so that failure skipped Fail the job if the story failed. The job went red for the wrong reason, by accident.

Labels are now created idempotently, the notifier is continue-on-error, the verdict step is always() && ..., and a new step fails closed when no verdict was produced at all.

Ratchet

scripts/check_sourced_libs_option_neutral.sh. Both files in defect 1 read correctly in isolation — only the combination is wrong, which is why this is mechanical rather than a review note.

Mutation-verified against the real pre-fix header: RED with it, GREEN without. Wired into ci.yml's text-only guard job with --self-test.

An earlier draft also scanned workflows and flagged check_format_sovereignty.sh, because ci.yml contains the comment # catch dev-dep cycles. See scripts/check_format_sovereignty.sh — the sentence-ending period parsed as the source builtin. Scope is now script-to-script only; a workflow run: block is its own throwaway shell.

Blast radius

One nightly. #2344 landed after the 07-30 run (green at e514cc5ed); 07-31 was the first run to source apr_bin.sh.

Not fixed here

The revived story surfaces a real finding: B2 apr qa Golden Output now fails on the 1.5B model where it passed on 07-30. No commit in that range touches inference, so it is filed separately rather than assumed to be a regression.

🤖 Generated with Claude Code

…eant to report it was broken

The 2026-07-31 qwen-story-daily run went red with "could not add label:
'qwen-story-daily' not found". That message is about the notifier. The
actual story had failed, and four independent defects conspired to make
that unsayable.

1. errexit leak (REGRESSION FROM #2344, mine). scripts/apr_bin.sh opened
   with `set -euo pipefail`. It is SOURCED, and `set` in a sourced file
   mutates the CALLER. qwen-story.sh had chosen `set -uo pipefail`
   WITHOUT -e on purpose: its whole design is to run all eight beats and
   tally failures (emit_fail / FAILED_BEATS / exit 2). Under the leaked
   errexit the first non-zero command aborted the run. The story died
   after SIX LINES inside Beat 1's ADVISORY pmat hunt. Reproduced at
   12551dd (exit 5, 6 lines); with this fix the same tree runs all
   eight beats, 153 lines, 13 PASS / 1 FAIL / 1 SKIP in 215s.
   apr_bin.sh now fails by return status, callers use `|| exit 1`.

2. jq shape. `pmat query --format json` returns an ARRAY of function
   records, but on a semantic miss it falls back to a document search and
   returns {"documents":[...]}. The filter ran `.[] | \(.function)`
   unconditionally, so the fallback shape raised "Cannot index array with
   function" - jq exit 5, propagated by pipefail. That was the specific
   non-zero that #1 turned fatal. Now shape-guarded, and `|| true`
   because an advisory manifest must never fail the story it annotates.

3. manifest grep never matched. The extractor looked for '^    gap' -
   FOUR spaces - while the script emits EIGHT. pmat-manifest.txt was 0
   lines on every run, including nights whose logs plainly contained
   `        churn ...` rows, which pinned `growth` at 0 forever and made
   the "manifest grew by >5" alert branch unreachable.

4. the verdict was gated behind the notifier. Neither `qwen-story-daily`
   nor `regression` existed as repo labels, so `gh issue create --label`
   had never once succeeded. Worse, a step `if:` without a status
   function carries an implicit `success()`, so that failure SKIPPED
   "Fail the job if the story failed". The job went red for the wrong
   reason and by accident. Labels are now created idempotently, the
   notifier is continue-on-error, the verdict step is `always() &&`, and
   a new step fails closed when no verdict was produced at all.

Ratchet: scripts/check_sourced_libs_option_neutral.sh. Both files in #1
read correctly in isolation - only the combination is wrong, which is why
this is mechanical rather than a review note. Mutation-verified against
the real pre-fix header: RED with it, GREEN without. Wired into ci.yml's
text-only guard job with --self-test.

An earlier draft of that guard also scanned workflows and flagged
check_format_sovereignty.sh, because ci.yml contains the comment
"# catch dev-dep cycles. See scripts/check_format_sovereignty.sh" and the
sentence-ending period parsed as the source builtin. Scope is now
script-to-script only; a workflow `run:` block is its own throwaway shell.

Blast radius: one nightly. #2344 landed after the 07-30 run (green at
e514cc5); 07-31 was the first run to source apr_bin.sh.

The revived story surfaces one real finding not addressed here: B2
`apr qa` Golden Output now fails on the 1.5B model where it passed on
07-30. No commit in that range touches inference, so it is filed
separately rather than assumed to be a regression.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@noahgift
noahgift enabled auto-merge July 31, 2026 14:42
@noahgift
noahgift added this pull request to the merge queue Jul 31, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 31, 2026
@noahgift
noahgift added this pull request to the merge queue Jul 31, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 31, 2026
@noahgift
noahgift added this pull request to the merge queue Jul 31, 2026
Merged via the queue into main with commit 3a686ce Jul 31, 2026
21 of 23 checks passed
@noahgift
noahgift deleted the fix/qwen-story-errexit-leak branch July 31, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant