[aw-failures] Fix false authentication_failed classification when watchdog kills an idle-but-completed agent (daily-experiment-r [Content truncated due to length]

[github-actions]

Fix the false authentication_failed classification that discards completed daily-experiment-report runs

Fix copilot_harness.cjs's watchdog-completion rescue so it also covers authentication_failed — right now it only rescues partial_execution/long_run_exit/no_output, so a real report gets thrown away every single day. Reason: the classifier hits authentication_failed before it ever reaches the branches that are already rescued.

Affected workflows and runs

daily-experiment-report (.github/workflows/daily-experiment-report.lock.yml), agent job, 4 consecutive days:

Date	Run	Job	Session duration before idle
2026-07-30	30527752090	90822762281	6m19s
2026-07-31	30617399212	91113870137	19m23s
2026-08-01	30692043079	91348411264	17m42s
2026-08-02	30740108209	91475924149	12m38s

Every run has agent_job_conclusion: failure even though the agent finished its analysis, uploaded charts, computed recommendations, and built the report body before the job died.

Probable root cause

Two bugs compound:

1. Over-broad regex scope. AUTHENTICATION_FAILED_PATTERNS in actions/setup/js/harness_retry_guard.cjs:28 includes /not logged in/i, matched by isAuthenticationFailedError() against the entire combined stdout+stderr of the whole session — not just the CLI's own provider-auth handshake. In run 30740108209, at 2026-08-02T08:40:19Z, the agent itself ran a bash tool call (checking whether gh was usable) and got back the benign, expected result "You are not logged into any GitHub hosts. To log in, run: gh auth login". That substring alone flips isAuthenticationFailedError to true for the whole run. Confirmed identical pattern (or an equivalent benign substring) across all 4 runs — grep -c "Authentication failed with provider" returns 0 in every one, proving no real provider-auth error ever occurred.
2. Missing rescue branch. classifyCopilotFailure() (copilot_harness.cjs:554-568) checks isAuthenticationFailed (line 562) before it would ever fall through to long_run_exit/partial_execution, so it returns "authentication_failed". The watchdog-completion rescue at copilot_harness.cjs:1275 only checks for failureClass === "partial_execution" || failureClass === "long_run_exit" || (failureClass === "no_output" && result.watchdogFired) — "authentication_failed" is absent from this list, even though (per the existing comment at lines 1271-1274) the post-result watchdog is only armed after hasTerminalSafeOutput is already true, meaning a watchdogFired=true run has, by construction, already produced its final deliverable.

Net effect: any copilot-engine workflow whose agent incidentally triggers an unauthenticated gh call (or anything else matching /not logged in/i or the other broad patterns) and then goes idle after completing will hit this same false failure — not unique to daily-experiment-report.

Specific proposed remediation

In actions/setup/js/copilot_harness.cjs, extend the rescue condition at line 1275 to also cover the watchdog-fired case for authentication_failed:

if (
  (failureClass === "partial_execution" ||
    failureClass === "long_run_exit" ||
    (failureClass === "no_output" && result.watchdogFired) ||
    (failureClass === "authentication_failed" && result.watchdogFired)) &&
  safeOutputsPath &&
  hasTerminalSafeOutput(safeOutputsPath)
) {

This reuses the same safety invariant already relied on for the no_output case (watchdog only arms post-hasTerminalSafeOutput), so it doesn't loosen retry behavior for genuine auth failures that occur before a terminal safe-output exists (those still fail normally, no watchdogFired). As a secondary hardening, consider scoping isAuthenticationFailedError() to the CLI's own log lines (excluding embedded tool.execution_complete JSON payloads from agent-run bash commands) so benign agent-side "not logged in" text can't trip the classifier at all — but the watchdog rescue alone fixes the observed data loss.

Success criteria / verification

• Add a unit test in actions/setup/js/copilot_harness.test.cjs reproducing this exact shape: result.watchdogFired = true, result.output containing a benign "not logged in" substring from a tool-execution payload (not a real provider auth failure), and hasTerminalSafeOutput(...) returning true — assert the harness treats it as success (lastExitCode = 0), not a retry/failure.
• Re-run daily-experiment-report after the fix lands and confirm agent_job_conclusion: success even when the agent's session goes idle post-completion.
• Confirm genuine authentication failures (e.g. Cluster D in the parent report, Authentication failed with provider ... HTTP 403, which occurs without watchdogFired and without a terminal safe-output) are unaffected and still fail/retry as before.
  Related to [aw-failures] [aw] Failure Investigator Report — 2026-08-02 (6h) #49788

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 173.6 AIC · ⌖ 36.4 AIC · ⊞ 6.8K · ◷

• expires on Aug 9, 2026, 5:24 AM UTC-08:00

[github-actions]

🍪 Issue Monster selected this for Copilot

I've identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

🍪 Om nom nom by Issue Monster · gpt54 · 7.02 AIC · ⌖ 6.1 AIC · ⊞ 10.3K · ◷