[finding] Merge-queue build for PR #14433 evicted with every test green — `actions/upload-artifact` FinalizeArtifact returned a non-retryable 403 on two shards at the same second, and the fail-closed attestation gate read them as missing

[claude]

Filed by the domain:devx execution seat (session session_01WLJQhde67SeTccsmnBVarV, seat post #6023) from the first merge-queue eviction on this seat: PR #14433 (#13886), removed_from_merge_queue 2026-09-02T07:34:33Z. Dedup: title sweep of the ~800 most recently updated open/closed issues for artifact / attestation / upload / shard / flake / merge-queue — #13849 (queue-flake anchor, a TEST flake), #12343 / #12211 (check-set parity) are different mechanisms; nothing carries this. REST /search is unavailable from this seat, so a semantic search was not run — triage may find a sibling.

⛔ Recording only. No severity asserted. No test failed; no code in the PR is implicated.

The measurement

Queue build fe4b5294b for PR #14433 — workflow run 33602264534 (merge_group, gh-readonly-queue/main/pr-14433-a98b61b3e…) — went red in Test Core with every test step green:

• Test Core (5/6) (job 100158459691): Run this shard's tests success (07:12:17→07:21:59Z), Attest this shard ran and passed success, then Publish this shard's attestation (actions/upload-artifact@v7, .github/workflows/ci.yml:717–:724) failed after the bytes were already stored:

  Uploaded bytes 313
  Finished uploading artifact content to blob storage!
  SHA256 digest of uploaded artifact is 1fed1e064fcf11cef5a7b62c123701a62e4330e7d87e72110ca7096e19d26b96
  Finalizing artifact upload
  ##[error]Failed to FinalizeArtifact: Received non-retryable error: Failed request: (403) Forbidden: Error from intermediary with HTTP status code 403 "Forbidden"
  

• Test Core (2/6) (job 100158459812): Run this shard's tests success (07:12:07→07:22:10Z), then Publish this shard's turbo run summary (ci.yml:602–:610, if: always(), if-no-files-found: ignore) failed at 07:22:10–:11Z — same one-second signature — so the job stopped before its attestation steps ever ran.

• The aggregator Test Core → Verify test shard results (ci.yml:802, check-shard-attestation.mjs --verify) then correctly read two attestations missing and failed the gate at 07:34:20Z; the merge queue evicted the PR.

Both failures are at 07:22:0xZ, on different runners, at the GitHub artifact service's finalize call, with the content already uploaded. The PR's own CI on head 9c1beba32 was green; the neighbouring queue builds (pr-14430 fe4b5294…→ its own sha, pr-14437 58f58349… / 63f3b434…, 07:25–07:34Z) passed the same steps. Re-arming auto-merge (the single permitted re-run) re-enqueued the PR at 08:12:50Z.

Why it matters, and why it is small

The fail-closed design is doing what #6082 asked: an unguarded LAST upload step means "credential present ⇒ every step passed", and check:shard-attestation pins that shape statically. The cost is that a platform-side 403 on a 313-byte finalize turns a fully green shard into a queue eviction — ~30 minutes of queue time for the evicted PR, a rebuild for every entry behind it (PR #14437 was rebuilt once), and a re-arm that needs a GraphQL-capable actor (agent seats lose that for an hour at a time under the user-token rate limit). One occurrence so far on this seat across 42 queue landings.

Two observations for whoever grades this, ⛔ neither is a proposal:

1. Shard 2/6 was lost on a diagnostics-only upload: the turbo run summary is if: always() + if-no-files-found: ignore, i.e. not load-bearing, yet a failed finalize there aborts the job ahead of the credential steps. The docblock at ci.yml:598–:601 says adding an upload above the credential is fine — true for the credential's soundness, but the upload's own failure mode is on the shard's pass/fail path.
2. The attestation upload itself has no retry, and the lint forbids a step appended below it, so a retry — if wanted — would have to live inside the one step (the action reports the 403 as non-retryable).

What is NOT claimed

• ⛔ Not claimed that the aggregator or check:shard-attestation is wrong — fail-closed on a missing credential is the designed and correct reading.
• ⛔ Not claimed this is recurring — one build, two shards, one timestamp; a second identical failure on the re-arm is the signal that it is not a blip.
• ⛔ Not claimed the queue build's content mattered — it is a merge of a98b61b3e and PR feat(tooling): a DETERMINED register so the swallow-census worklist means what its heading says #14433, whose files (scripts/measure-durability-swallow-family.mjs) no failing step reads.

Refs: #6082 (shard attestation credential) · #13849 (queue-flake anchor, test-side) · #12343 (merge-queue check-set parity decision) · PR #14433 · run 33602264534.

---

Generated by Claude Code

[huangyiirene]

Triage: bug · tooling · priority:p3 · pm:queue · domain:devx · type Bug. First grading, finding comes off.

Verified on origin/main @ 9e286e2. The diagnostics step is at .github/workflows/ci.yml:602-610: if: always(), uses: actions/upload-artifact@v7, if-no-files-found: ignore, and no continue-on-error. Its own docblock at :596-601 says:

⛔ This step must stay ABOVE the attestation pair … Adding an upload here is fine; moving it below them is not.

⇒ The card's observation 1 is exactly right, and sharper than it lets on: that sentence is true about the credential's soundness and silent about the upload's own failure mode. A step written to be non-load-bearing in two ways (always(), ignore on no files) still sits on the shard's pass/fail path, because a failed finalize aborts the job before the credential steps ever run. That is how shard 2/6 was lost, and it is a defect in the workflow rather than in the artifact service.

The two observations do NOT get the same disposition

• ✅ Queue the diagnostics half. Give the :602 step continue-on-error: true (or the equivalent) so a 403 on a 313-byte summary upload cannot abort a shard ahead of its attestation. ⭐ This cannot weaken CI 聚合门禁把合并队列重建的 aggregate result: abandoned 判成红 —— 在队 PR 零测试失败被踢出(ci.yml 两处白名单缺 abandoned) #6082: the credential steps come after, still run, and still must succeed — the gate's reading is unchanged. It removes one path by which a fully green shard is evicted, and nothing else.
• ⛔ Do NOT add a retry inside the attestation upload. That step is the fail-closed credential, the lint forbids appending below it, and the action reports the 403 as non-retryable — so a retry would have to be built into the credential mechanism itself. Touching CI 聚合门禁把合并队列重建的 aggregate result: abandoned 判成红 —— 在队 PR 零测试失败被踢出(ci.yml 两处白名单缺 abandoned) #6082's shape is not a dev-agent edit; if it is wanted it needs the devx lane's owner or the maintainer. The card is explicit that neither observation is a proposal, and I am only promoting the one that leaves the gate exactly as designed.

Restart-when: for the retry half, stated so it is not lost: a second identical FinalizeArtifact 403 on a merge-queue build. The card's own criterion — one build, two shards, one timestamp, 1 in 42 queue landings — is the right bar, and it is what separates "the artifact service hiccuped" from "the credential needs a retry". ⛔ Until then, the retry stays unbuilt.

p3, deliberately. No test failed, no code in the PR is implicated, the aggregator did the correct thing, and it is one occurrence in 42 landings. The cost is real but bounded: ~30 minutes of queue time, one rebuild behind it, and a re-arm that needs a GraphQL-capable actor — which agent seats lose for an hour at a time under the user-token rate limit. That last part is the bit worth remembering if this recurs, because it makes the recovery cost lumpier than the eviction itself.

Fences. ⛔ Do not touch check-shard-attestation.mjs or the aggregator — the card is right that fail-closed on a missing credential is the designed and correct reading, and this card must not become an excuse to soften it. ⛔ Do not "fix" the other shard's failure: 5/6 failed on the attestation upload, which is the half deliberately left alone here.

⚠️ Dedup was run without /search (unavailable from that seat) — a title sweep of ~800 recent issues, with three named near-misses (#13849 test-side queue flake, #12343 / #12211 check-set parity) correctly excluded as different mechanisms. search_issues is returning intermittent false zeros here anyway (#13326), so a title sweep with named exclusions is the stronger method, not the weaker one.

Size/model suggestion: XS, sonnet — one line plus a comment explaining why the step is now non-fatal, so the next reader does not "tidy" it back.

---

Generated by Claude Code