CI Failure DoctorCI Failure Investigation: Daily Perf/Test Improver - setup_threat_detection Regression (22+ days)

[github-actions[bot]]

Summary

The Daily Perf Improver and Daily Test Improver agentic workflows have been failing consistently for 22 consecutive days (since March 2, 2026). The failure occurs in the detection job at the "Setup threat detection" step, which blocks all agent outputs from being published.

This is not a Rust code issue — all Rust CI jobs (clippy, tests, doc, deny, msrv) pass normally.

---

Failure Details

• Affected Run: Daily Perf Improver #152
• Commit: 448b4e6f8cfb28665b2697538d802f6dfee66cc5
• Trigger: schedule (daily cron)
• First Failure: 2026-03-02 (run 22565153541)
• Last Success: 2026-03-01 (run 22538146335)
• Failure Duration: 22+ consecutive days and counting
• Affected Workflows: Daily Perf Improver AND Daily Test Improver (both fail identically)

---

Root Cause Analysis

The Trigger: SHA Change in github/gh-aw/actions/setup

PR #84 ("ci: bump github/gh-aw from 0.47.1 to 0.51.2", merged March 2, 2026) changed the pinned SHA for github/gh-aw/actions/setup in all .lock.yml workflow files:

SHA	Status	Version Tag	Introduced By
c94abee3e98890c5ddf4e8f41a74ed3302c56cec	✅ Working	# v0.47.1	original
5fa65dd15a71cec00f2d14c664467154d343d875	❌ Failing	# v0.47.1	PR #84
c2ce8fd41aa47d7471b14b1014c4c7560b7070f2	❌ Failing	# v0.47.1	PR #95
0ce8adde9abb3d0841cfb16ede313b9f99301642	❌ Failing	# v0.47.1	intermediate
1b847e3d0c6d8ebc44cb538c55198da42baa8a78	❌ Failing	# v0.47.1	PR #140 (current)

All SHAs carry the version comment # v0.47.1, suggesting the v0.47.1 tag on the github/gh-aw repo has been force-pushed multiple times. Each updated SHA introduced (or retained) the breaking change in setup_threat_detection.cjs.

What Fails

The setup_threat_detection step runs a JavaScript action:

const { main } = require('/opt/gh-aw/actions/setup_threat_detection.cjs');
await main();
```

This step fails almost immediately (completes within 0 seconds at 2026-03-23T07:15:13Z). All downstream steps are skipped, including:
- `Validate COPILOT_GITHUB_TOKEN secret`
- `Execute GitHub Copilot CLI` (the actual threat scanner)
- `Parse threat detection results`
- Safe outputs publishing

### Impact

- **22+ days of suppressed agent outputs**: Neither Daily Perf Improver nor Daily Test Improver has been able to create PRs, issues, or other outputs
- **Subsequent gh-aw bumps didn't help**: PRs #84, #95, and #140 all kept the broken SHA
- **Agent job itself still succeeds**: The Copilot CLI agent runs, analyzes the codebase, generates proposals — but they're all silently blocked

---

## Failed Jobs and Errors

```
Job: pre_activation   → success
Job: activation       → success
Job: agent            → success ✓ (agent runs fine!)
Job: detection        → FAILURE
  Step: Setup threat detection → failure ← ROOT CAUSE
  Step: Ensure threat-detection directory → skipped
  Step: Validate COPILOT_GITHUB_TOKEN → skipped
  Step: Execute GitHub Copilot CLI → skipped
  Step: Parse threat detection results → skipped
Job: safe_outputs     → skipped (blocked by detection failure)
Job: push_repo_memory → skipped
```

---

## Investigation Findings

### Pattern Confirmation

Checked all scheduled runs over the entire failure window:

| Date | Daily Perf | Daily Test |
|------|-----------|------------|
| Mar 1 | ✅ | ✅ |
| Mar 2–Mar 23 | ❌ (22 days) | ❌ (22 days) |

### Key Observations

1. The failure is 100% reproducible — occurs every single day, no flakiness
2. The agent execution succeeds — the problem is exclusively in the post-processing `detection` job
3. The failure duration (22 days) means the codebase has missed ~22 potential performance improvement PRs and ~22 test improvement PRs
4. The `actions-lock.json` still records the old SHA (`c94abee3...`) as v0.47.1, while the lock files now use `1b847e3d...` — indicating a desync between the actions lock and the compiled workflow files

---

## Recommended Actions

- [ ] **Immediate**: Manually revert `github/gh-aw/actions/setup` SHA in all `.lock.yml` files back to `c94abee3e98890c5ddf4e8f41a74ed3302c56cec` to restore the last known working state
- [ ] **OR recompile**: Run `gh aw compile` with a verified working version of the gh-aw CLI to regenerate the lock files
- [ ] **Contact gh-aw team**: Report the regression in `setup_threat_detection.cjs` that was introduced after SHA `c94abee3`
- [ ] **Add monitoring**: Create a workflow health check that alerts when both Perf Improver AND Test Improver fail for 3+ consecutive days

---

## Prevention Strategies

1. **Test SHA changes before merging**: When dependabot bumps `github/gh-aw`, verify a test run of the affected workflows before merging
2. **Add a workflow health check**: A dedicated scheduled workflow that checks the last N runs of agent workflows and creates an alert issue if they're all failing
3. **Update the `ci-doctor` monitored workflows**: Add "Daily Perf Improver" and "Daily Test Improver" to the CI Doctor's monitored workflows list so failures trigger automated investigation immediately
4. **Pin to stable SHA**: Instead of following floating version tags, pin to a specific SHA that has been validated

---

## AI Team Self-Improvement

For AI coding agents (`CLAUDE.md` / `copilot-instructions.md` additions):

```
## Agentic Workflow Health Monitoring

When `github/gh-aw/actions/setup` SHA changes occur (via dependabot PRs targeting 
`daily-perf-improver.lock.yml`, `daily-test-improver.lock.yml`, etc.):
- Always check if the SHA change could affect `setup_threat_detection.cjs` behavior
- After merging such PRs, verify the next scheduled run of affected agent workflows succeeds
- If "Setup threat detection" starts failing after a SHA bump, immediately revert the 
  lock files to the previous known-working SHA

SHA history for `github/gh-aw/actions/setup`:
- WORKING: c94abee3e98890c5ddf4e8f41a74ed3302c56cec (# v0.47.1)
- BROKEN: all subsequent SHAs from 2026-03-02 onwards

---

Historical Context

• First occurrence: March 2, 2026 (triggered by PR ci: bump github/gh-aw from 0.47.1 to 0.51.2 #84 merging)
• Previous issues: Issue [aw] CI Failure Doctor failed #124 tracks a separate CI Failure Doctor failure (March 11) — different failure mode
• No prior investigations of this specific Daily Perf/Test Improver threat detection failure exist
• This is a new, long-running regression not previously tracked or resolved

AI generated by CI Failure Doctor · history

• expires on Mar 30, 2026, 7:27 AM UTC