fix(conformance): classify silently-collapsed packaging/toolchain failures - #102
Merged
Conversation
…ilures
resolveRustToolchain's three realpathSync calls (cargo/rustc/rustdoc) were
unwrapped, so a missing or unreadable binary collapsed into the generic
phase1.stage.toolchain.failed diagnostic instead of the already-public
phase1.environment.rustup-{cargo,rustc,rustdoc}.failed identifiers that
exist in the allowlist but were never thrown. observeReleaseToolVersions
also called resolveRustToolchain unwrapped, unlike safeEnvironment's own
caller, which already wraps it in phase1.environment.rust-toolchain.failed.
Separately, runPhase1Conformance's schema-v2 (--platform) branch returns
runSchemaV2Conformance() directly, bypassing the activeStage-tracked
try/catch/finally that the rest of the function uses to attribute
unrecognized failures to a specific stage. Any error thrown inside that
branch that main().catch() couldn't classify fell all the way through to
the fully generic "Phase 1 conformance failed." literal with zero stage
context. Wrap the dispatch in phase1.stage.schema-v2-production.failed
(new, added to the allowlist) so schema-v2 evidence-production failures
are at least attributable to that stage.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ueGDYGKGaMzNq5mTE6ogG
assertProductionAdapterAtRevision was the only unwrapped call inside packageLockedArtifacts (chat-79's static walk of the whole 225-line stage confirmed no other bare throw sites remain). Sitting between the chat-web-build and chat-native-build stages with no diagnostic wrap, any failure there collapsed into the generic phase1.stage.packaging.failed that main().catch() alone can't tell apart from the ~20 other things that stage covers. Wrap it in a dedicated phase1.packaging.production-adapter.failed (new, added to the allowlist) so a future authority mismatch here is attributable on sight instead of indistinguishable from everything else packaging can fail on. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ueGDYGKGaMzNq5mTE6ogG
…se fix Repins harness.revision and harnessAuthority.revision/tree to 4afae50 (the two preceding fix commits: rustup/schema-v2 stage classification, then the production-adapter authority wrap) and updates the scripts/phase1-conformance.mjs file-hash pin to match. harness.revision/tree still need a follow-up rebind to the eventual merge commit once this lands on main, per the repo's usual two-step pattern. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ueGDYGKGaMzNq5mTE6ogG
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The changes are narrowly-scoped error-wrapping plus consistent repinning, and the new/used diagnostic IDs are verified to be in the runtime allowlist.
Pull request overview
This PR improves Phase 1 conformance failure attribution by wrapping previously-unclassified toolchain/packaging/schema-v2 failures in specific allowlisted diagnostic IDs, avoiding opaque “stage-only” or fully-generic failures while preserving the existing redaction policy.
Changes:
- Wrap Rust toolchain resolution (
realpathSyncfor cargo/rustc/rustdoc and the overall toolchain probe) with existing allowlisted diagnostics. - Wrap schema-v2 (
--platform) conformance execution with a new stage diagnostic to prevent fallback-to-generic failure classification. - Wrap the production adapter revision assertion in packaging with a new specific packaging diagnostic, and repin all hash/lock references accordingly.
File summaries
| File | Description |
|---|---|
scripts/phase1-conformance.mjs |
Adds new public diagnostic IDs and wraps previously-unwrapped failure sites to preserve specific classification. |
src/phase1-conformance-lock.test.ts |
Updates expected harness authority revision/tree and script hash pins. |
phase1-conformance.lock.json |
Repins harness/harnessAuthority revision/tree and scripts/phase1-conformance.mjs hash. |
docs/phase1-conformance.md |
Updates the documented scripts/phase1-conformance.mjs byte-size and sha256 pin. |
.github/workflows/client-v1-conformance.yml |
Updates the workflow’s embedded file-size/sha256 pins for scripts/phase1-conformance.mjs in both validation tables. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The previous repin commit edited .github/workflows/client-v1-conformance.yml (updating its embedded pin table for scripts/phase1-conformance.mjs) but missed that the workflow file is itself one of the tracked harnessAuthority.files entries. Editing it changed its own size/hash, which the docs-guide consistency test (client-v1-conformance-workflow.test.ts) caught immediately since that test hashes the live file directly rather than trusting the lock. Following the repo's usual sequencing, this pin-table edit belongs in the same commit as the fix it describes, not the repin commit — so this also advances harness.revision/harnessAuthority.revision+tree to the repin commit itself (which already has the fix, the workflow edit, and every other pin in their final state) rather than leaving them one commit short. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ueGDYGKGaMzNq5mTE6ogG
3 tasks
BunsDev
added a commit
that referenced
this pull request
Sep 5, 2026
chore(conformance): repin harness authority to the merged commit #102 was squash-merged, so its branch commits are not in main's history and the harness authority named a revision that was only reachable through a surviving branch ref. Deleting that branch would have broken the pin outright rather than leaving it stale. Repins harness.revision and harnessAuthority.revision/tree to 12f143d, which is main's own tip and therefore durably reachable. Tree content is unchanged. Also records two things that were not discoverable until a test failed: that .github/workflows/ci.yml is harness-pinned alongside client-v1-conformance.yml, and that conformance pull requests must be merged with a merge commit rather than squashed so the pinned revision stays an ancestor of main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The last ~30 hours of cross-repo conformance churn (Chat #95-#100, SDK #109-#119) has been a series of blind retries: the harness collapses unclassified failures down to a handful of fully generic stage IDs (or, in one path, a literal
'Phase 1 conformance failed.'string), so nobody chasing a red run from CI logs alone could tell which of ~20 things inside a stage actually broke. This PR closes the three unwrapped call sites currently capable of producing that collapse, without loosening the harness's deliberate redaction policy anywhere.resolveRustToolchain's three barerealpathSynccalls (cargo/rustc/rustdoc) now wrap in the already-allocatedphase1.environment.rustup-{cargo,rustc,rustdoc}.failedidentifiers, which existed in the allowlist but were never thrown.observeReleaseToolVersionsnow wraps its call toresolveRustToolchaininphase1.environment.rust-toolchain.failed, matchingsafeEnvironment's existing caller.runPhase1Conformance's schema-v2 (--platform) branch returnedrunSchemaV2Conformance()directly, bypassing theactiveStage-tracked try/catch/finally the rest of the function uses to attribute unrecognized failures to a stage. This is what actually produced the fully opaquemain().catch()literal on a real Windows conformance run — traced end to end against that run's logs. Now wrapped in a newphase1.stage.schema-v2-production.failed.assertProductionAdapterAtRevisionwas the only unwrapped call in all 225 lines ofpackageLockedArtifacts(confirmed by a full static walk of the stage) — the only site capable of producing a barephase1.stage.packaging.failedinstead of one of the ~15 more specificphase1.packaging.*diagnostics already covering that stage. Now wrapped in a newphase1.packaging.production-adapter.failed.Two findings from the same investigation were deliberately not changed here:
main().catch()'s three-way fallback is working as designed — loosening it to print raw error/cause text would let any future unwrapped throw leak arbitrary internals into a public CI log, which is exactly what the curated allowlist exists to prevent. Andwindows-conformance-child.ps1's exit-code-only throw doesn't actually discard anything —Invoke-Checkednever redirects the child's stdout/stderr, so the Node process's own diagnostic line is inherited straight to the console already; it was just uninformative because of the schema-v2 bug above, not because PowerShell ate it.Standard second-step follow-up: repins
harness.revision/harnessAuthority.revision+treeto the second fix commit, and updates thescripts/phase1-conformance.mjsfile-hash pin in all four required places (both tables inclient-v1-conformance.yml, the row indocs/phase1-conformance.md, andphase1-conformance.lock.json), plus the hardcoded mirror inphase1-conformance-lock.test.ts.Test plan
pnpm typecheck/pnpm lint/pnpm format:check— cleanphase1.stage.schema-v2-production.failed,phase1.packaging.production-adapter.failed) and the threerustup-*IDs are real members of the assembled runtime allowlist (dynamic import +publicPhase1FailureDiagnosticprobe), not just present in the source literal — the allowlist is also built from generated entries a plain grep won't seevitest run --config vitest.heavy.config.ts src/phase1-conformance-lock.test.ts src/phase1-conformance.test.ts— 227 passed, 1 pre-existing skip, 0 failed🤖 Generated with Claude Code
https://claude.ai/code/session_016ueGDYGKGaMzNq5mTE6ogG