[aw-failures] 6h Failure Investigation — Pydantic AI engine breaks 100%, plus 3 correlated P1s (2026-08-08)

[github-actions]

Fix the AWF-binary gap in BehaviorDefinedEngine.GetInstallationSteps — it breaks every non-npm custom engine, not just Pydantic AI

Executive summary: Two "Smoke Pydantic AI" runs failed back-to-back tonight because the custom-engine compiler path never emits an install_awf_binary.sh step for engines whose behavior declares no installation block (or a non-npm one) — awf: command not found kills the run before the agent ever executes. That's a P0, code-verified, 100%-reproducible compiler defect. Three smaller issues ride along: a dispatch_workflow call that shipped without its required message input (smoke-copilot-aoai-entra), a hallucinated GraphQL node ID that failed an otherwise-clean PR Sous Chef run, and a BYOK-Ollama warm-up probe that pings the wrong endpoint.

None of today's 5 failure clusters match any of the 3 open agentic-workflows bug issues on file — leave them all open as-is.

Failure cluster table

#	Severity	Workflow(s)	Representative run	Comparator run	Signature
A	P0	Smoke Pydantic AI	§31231477754	§31229465314	awf: command not found (exit 127) / Failed to spawn: pai (exit 2) — engine never runs
B	P1	Smoke Copilot - AOAI (Entra)	§31222505053	—	dispatch_workflow → haiku-printer: Required input 'message' not provided (HTTP 422)
C	P1	PR Sous Chef	§31230238662	—	resolve_pull_request_review_thread given a fabricated ID: PRRT_kwDOPc1QR88AAAAB-example-invalid
D	P1	Daily BYOK Ollama Test	§31223940604	—	Copilot CLI: 503 Service Unavailable × 4 attempts against Ollama despite a passing warm-up check

Evidence

Cluster A — code-verified root cause (click to expand)

pkg/workflow/behavior_defined_engine.go:166-213, GetInstallationSteps:

if behavior.Installation == nil {
    if behavior.HarnessScript == "" {
        return nil
    }
    return BuildNpmEngineInstallStepsWithAWF([]GitHubActionStep{GenerateNodeJsSetupStep()}, workflowData)
}
install := behavior.Installation
if install.PackageManager != "npm" {
    return nil   // <-- AWF install skipped for any non-npm installation, or no installation at all
}

.github/workflows/shared/pydantic.md declares runtimes: uv and a pre-agent-steps entry for uv run pai --version, but no engine.behaviors.installation block and no harness-script. That routes straight through the return nil at line 169/181/188 — the compiled lock file never contains an install_awf_binary.sh step at all. Confirmed by diffing generated workflows:

$ grep -n "install_awf_binary" .github/workflows/smoke-copilot.lock.yml
755:  run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.44 --rootless
$ grep -n "install_awf_binary" .github/workflows/smoke-pydantic.lock.yml
(no output)

audit-diff between the two failed runs confirms both are total losses — zero token usage on both sides, meaning the agent CLI never got to execute in either run:

"run_metrics_diff": { "run1_token_usage": 0, "run2_token_usage": 0 }

The two runs show two symptoms of the same missing-install-step defect: the earlier run (§31229465314) additionally lacked the pip install pydantic-ai step and failed one step earlier (Failed to spawn: pai); the later run (§31231477754) got past that but still hit the missing-awf-binary wall at the actual execution step.

Cluster B — dispatch_workflow shipped without its required input

haiku-printer.yml declares inputs.message as required: true. The smoke-copilot-aoai-entra prompt instructs the agent to call dispatch_workflow for haiku-printer "with an original testing/automation haiku," but the handler (actions/setup/js/dispatch_workflow.cjs:208-256) passes message.inputs straight through to the GitHub API with no pre-flight validation against the target workflow's required-input schema — so a model that omits the inputs.message key gets a raw, hard-to-diagnose 422 from GitHub instead of an actionable local error, and the whole job is marked failure even though every other safe output in the run succeeded.

This is the same code path already flagged in #49583 (dispatch_workflow → haiku-printer breaking otherwise-successful smoke-copilot runs), just a different trigger (missing input vs. deleted ref branch). Broadening #49583's scope to cover input validation, not just ref resolution, covers both without a second issue.

Cluster C — hallucinated review-thread node ID

Job log for §31230238662:

##[error]Failed to resolve review thread: Request failed due to following response errors:
 - Could not resolve to a node with the global id of 'PRRT_kwDOPc1QR88AAAAB-example-invalid'

The literal string example-invalid does not appear anywhere in pr-sous-chef.md — the model fabricated this ID rather than copying a real one out of reviewThreads. 5 of 6 safe outputs in that run succeeded (issue comment posted fine); only this one call failed, yet it flipped the whole job to failure. The resolve_pull_request_review_thread handler has no format check on the ID before calling the GraphQL API — a cheap regex guard (^PRRT_[A-Za-z0-9]+$, reject anything containing non-base64 tokens like "example" or "invalid") would turn this into a soft missing_data/warning instead of a hard job failure.

Cluster D — warm-up probe checks the wrong endpoint

daily-byok-ollama-test already carries a documented mitigation (its own inline comment): "A cold model (not yet loaded) can cause the OpenAI-compatible /v1/chat/completions endpoint to return 503 Service Unavailable on the agent's first requests, which exhausts the Copilot CLI's built-in retry budget and fails the whole run." The fix it ships probes (localhost/redacted) (Ollama's **native** API) and reports "Model warm-up succeeded on attempt 1" — but the actual agent traffic goes through the api-proxy's **OpenAI-compatible** /v1/... surface, which independently returned 503 (awf-reflect models fetch returned 503 for (apiproxy/redacted) retrying) and then failed for real across all 4 Copilot CLI attempts ([§31223940604](https://github.com/github/gh-aw/actions/runs/31223940604), 22:31-22:34 UTC). Warming the native endpoint doesn't warm the OpenAI-compat one — the probe needs to hit /v1/modelsor/v1/chat/completions` (the same path the harness uses) before declaring success.

Existing issue correlation

• [aw-failures] dispatch_workflow safe-output fails the whole Smoke Copilot run when the ephemeral trigger branch is gone #49583 (dispatch_workflow ref-resolution failure) — not fixed, still valid — extend scope to cover Cluster B's missing-input variant (same call site, same net effect: an otherwise-successful smoke-copilot run gets marked failed over the finishing safe-output).
• [aw-failures] Fix Copilot CLI Task-tool subagent calls — 100% failure with No model available #51033 (Copilot Task-tool "No model available") — unrelated to this window's failures, no new evidence either way, left open.
• [aw-failures] Fix: unguarded arithmetic on github.event.pull_request.stack in applyPullRequestStackFilter #49446 (unguarded pull_request.stack arithmetic) — unrelated, left open.
• No open issue covers Clusters A, C, or D — Cluster A gets a new sub-issue below (P0); C and D are documented here for follow-up given the 2-issue budget for this run — file separately if not picked up.

Fix roadmap

P0 — do this first: Add the AWF-install step unconditionally for behavior-defined engines when network isolation is enabled, regardless of install.PackageManager or whether Installation is declared at all. See sub-issue below for the exact change.

P1 — do these next:

1. Validate dispatch_workflow inputs against the target workflow's declared required inputs before calling the GitHub API; extend [aw-failures] dispatch_workflow safe-output fails the whole Smoke Copilot run when the ephemeral trigger branch is gone #49583 to own this.
2. Add a node-ID format guard to resolve_pull_request_review_thread (and siblings) so a malformed/fabricated ID degrades to a warning, not a job failure.
3. Point the Ollama warm-up probe at the OpenAI-compatible /v1/models (or /v1/chat/completions) path instead of the native /api/generate endpoint.

P2 — monitor only: none this cycle.

Sub-issues created

• Cluster A fix — see linked sub-issue below.

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 191.8 AIC · ⌖ 55.2 AIC · ⊞ 5.5K · ◷

• expires on Aug 14, 2026, 5:22 PM UTC-08:00

---

Update — 2026-08-08 07:xx UTC (second 6h sweep)

Fix the Design Decision Gate max-turns: 20 cap first — it killed 2/2 of that workflow's runs this window on complex PRs; see linked sub-issue (#aw_ddg20).
Add DOCKER_PAT or soft-fail the docker-sbx check — Daily Assign Issue To User is now failing every run on a missing-secret pre-flight check, not a flaky issue; see linked sub-issue (#aw_daipat).

New failure cluster table (this sweep)

#	Severity	Workflow(s)	Representative run	Comparator run	Signature
E	P1	Design Decision Gate 🏗️	§31238149137	§31236494018	429 Maximum LLM invocations exceeded (20 / 20) — max-turns: 20 too low for complex PRs
F	P2	Daily Assign Issue To User	§31242999700	—	secrets.DOCKER_PAT is empty — persistent until secret added or check made non-fatal
G	P2	PR Description Updater	§31243976760	—	Copilot CLI: 13 permission-denied events, then update_pull_request failed on empty body (quoting bug); agent self-recovered via report_incomplete — no data loss, not filed as a separate issue this cycle
H	— (not a bug)	Daily Container Image Security Scan	§31242183814	—	Gate correctly failed: Critical vulnerabilities detected in container images — the scanner is working as designed; the fix here is patching the flagged images, not the workflow

Evidence

Cluster E — code-verified via audit + audit-diff

agent-stdio.log for §31238149137: API Error: Request rejected (429) · Maximum LLM invocations exceeded (20 / 20). followed by [claude-harness] attempt 1: maximum LLM invocations exceeded — not retrying (non-retryable guard condition).

audit-diff (base §31237067931 successful → §31238149137 failed) confirms the failed run made exactly 20/20 requests (run2_total_requests: 20) before being cut off by the guard. .github/workflows/design-decision-gate.md:33 sets max-turns: 20 — confirmed as the ceiling being hit, not an infra crash.

§31236494018 shows the same shape independently: 17 turns, 841k tokens, "Resource Heavy For Domain" (high), same "Workflow Failed" signature — 2/2 occurrences this window.

Cluster F — audit key_findings

##[error]secrets.DOCKER_PAT is empty. docker-sbx requires a Docker Hub personal access token to pull the sandbox template image. Add a DOCKER_PAT secret to your repository. — this will recur on every future run until resolved; not a one-off flake despite only 1 occurrence in this 6h window.

Cluster G — not filed as a new issue this cycle (2-issue budget spent on E and F)

[copilot-harness] attempt 1 failed: exitCode=1 failureClass=permission_denied permissionDeniedCount=13 then update_pull_request failed on a validation error from an empty body (quoting issue); agent called report_incomplete per policy instead of retrying. Isolated single occurrence, no data loss — lower urgency than E/F. File separately if it recurs.

Existing issue correlation (this sweep)

• [aw-failures] Fix Copilot CLI Task-tool subagent calls — 100% failure with No model available #51033 (Copilot Task-tool "No model available") — no new evidence either way this window, left open.
• [aw-failures] dispatch_workflow safe-output fails the whole Smoke Copilot run when the ephemeral trigger branch is gone #49583 (dispatch_workflow ref-resolution) — no new evidence this window, left open.
• Clusters A/B/C/D from the prior sweep — unaffected by this sweep's dataset, no changes recommended.

Fix roadmap (cumulative)

P0: (carried from prior sweep) AWF-install step for behavior-defined engines — see prior sub-issue.
P1 — new this cycle: Raise or soft-fail max-turns: 20 in Design Decision Gate (sub-issue #aw_ddg20).
P2 — new this cycle: Fix/soft-fail DOCKER_PAT check in Daily Assign Issue To User (sub-issue #aw_daipat); monitor PR Description Updater quoting bug (Cluster G, not filed).

Sub-issues created this sweep

• #aw_ddg20 — Design Decision Gate max-turns cap
• #aw_daipat — Daily Assign Issue To User missing DOCKER_PAT> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 163 AIC · ⌖ 42.9 AIC · ⊞ 5.5K · ◷

[github-actions]

🍪 Issue Monster selected this for Copilot

I identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

🍪 Om nom nom by Issue Monster · gpt54 · 1.22 AIC · ⌖ 7.31 AIC · ⊞ 8.4K · ◷