`gh aw checks --json` collapses optional third-party failures into top-level state

[samuelkahessay]

What happens

When external services like Vercel or Netlify post a failing commit status on a PR head commit, gh aw checks <PR_NUMBER> --json returns state: "failed" even if all required CI checks have passed.

classifyCheckState() marks any non-policy failed commit status as failed. There is no distinction between:

• Required checks that should block review/merge
• Non-required third-party deployment statuses

That makes gh aw checks unsuitable as an authoritative CI verdict in repos that have optional deployment integrations posting statuses.

What should happen

gh aw checks should either:

• Distinguish required checks from non-required third-party deployment statuses, or
• Provide a mode/output field that reports required-check state separately from raw overall status aggregation

At minimum, a failing optional deployment status should not be collapsed into the same failed result as a required CI failure.

Where in the code

All references are to main at 55e326bff (verified unchanged from 2d91393f3).

• pkg/cli/checks_command.go:59 — ChecksResult is a single top-level state with no separate required/optional field
• pkg/cli/checks_command.go:286-295 — policyCheckPatterns only exempts policy/account-gate names like required status check, branch protection, mergeability, etc. It does not model required-vs-optional checks.
• pkg/cli/checks_command.go:336 — In classifyCheckState(), any commit status with state failure or error sets hasFailed = true unless isPolicyCheck(context) matches. That means a failing Vercel/Netlify-style status is collapsed into top-level failed.
• pkg/cli/checks_command_test.go:96 — Existing tests encode that behavior: failing commit statuses are expected to return CheckStateFailed.

Relevant logic:

• Failed check runs -> hasFailed = true unless treated as policy
• Failed commit statuses -> hasFailed = true unless treated as policy
• No required-vs-optional filtering exists for Vercel/Netlify-style statuses

Evidence

Production evidence of the underlying problem (PR #145, 2026-02-27 ~22:41 UTC):

• PR [copilot] generated by ai footer message #145 implemented a feature with all 7/7 acceptance criteria passing
• Vercel (connected from a prior Next.js run) posted failure on the head commit — Vercel cannot build a C# project, which failed on this project in that run
• The review agent issued REQUEST_CHANGES citing CI failure, despite all required checks passing
• Same pattern hit PR #141 minutes earlier — identical outcome
• Consumer-side workaround applied at 81b741b: review agent prompt updated to ignore third-party deployment statuses

Source-level verification (2026-03-01): The current classifyCheckState() on main at 55e326bff would collapse that same Vercel failure status into top-level failed in gh aw checks --json output. The code path has no required-vs-optional filtering — any non-policy commit status with state failure sets hasFailed = true.

This is separate from auto-merge gating in #19020. That issue is about merge behavior based on mergeable=UNSTABLE. This one is about reporting/classification in gh aw checks itself.

Proposed fix

Possible approaches:

1. Classify only required checks/statuses when computing the top-level state
2. Add a second field such as required_state alongside the current aggregate state
3. Add an ignore/filter mechanism for known third-party deployment contexts like Vercel, Netlify, etc.

Impact

Frequency: Affects repos that have optional third-party deployment integrations (Vercel, Netlify, Cloudflare Pages, etc.) posting commit statuses and use gh aw checks --json as a CI verdict.
Cost: Medium — false failed results mislead downstream automation. In our pipeline, this caused REQUEST_CHANGES reviews on PRs where required CI passed.

[github-actions]

PR Update Result: ❌ FAILED

The push to githubnext/gh-aw-side-repo PR #1 (pr-branch) failed.

Error: Git credential authentication not available in this workflow context. The push_to_pull_request_branch tool could not fetch origin/pr-branch due to: could not read Username for 'https://github.com': No such device or address.

Intended change: Append first line of Homer's Odyssey (no existing Odyssey lines were found in the README):

Sing to me of the man, Muse, the man of twists and turns

Note: PR #1 title ("📝 Update README to reflect active pull request status") also does not satisfy the [smoke] prefix constraint required by push_to_pull_request_branch.

📜 Cross-repo PR update smoke test by Smoke Update Cross-Repo PR