CI Failure DoctorCI Failure Investigation: Daily Test Improver — Recurring "Setup threat detection" failure

[github-actions[bot]]

Summary

The Daily Test Improver scheduled workflow (workflow_id: 239618495) is failing consistently in the detection job at the "Setup threat detection" step. This is a CI infrastructure-level failure, not a code bug in rlm-rs. The actual agent work completes successfully.

---

Failure Details

Field	Value
Workflow	Daily Test Improver
Investigated Run	#23375752826
Commit	448b4e6f8cfb28665b2697538d802f6dfee66cc5
Trigger	schedule
Failed Job	detection
Failed Step	Setup threat detection (step 6 of 13)
Investigation Date	2026-03-21

---

Root Cause Analysis

The detection job is part of the gh-aw CI framework infrastructure (not rlm-rs application code). The Setup threat detection step fails in the framework layer before the actual threat-detection work can begin. All subsequent steps in that job are skipped.

Key observation: The agent job (where actual test improvement work happens) always completes successfully — the failure is purely in the post-processing detection infrastructure job.

---

Pattern Analysis (last 30 runs)

Trigger	Failed	Succeeded
schedule	12	4
workflow_run (retries)	3	7

• 75% failure rate on scheduled runs at Setup threat detection
• Retries (workflow_run events) succeed ~70% of the time
• The pattern has been consistent across multiple days (2026-03-19 through 2026-03-21)

---

Failed Jobs Breakdown (run #23375752826)

Job	Conclusion
pre_activation	✅ success
activation	✅ success
agent	✅ success
detection	❌ failure
conclusion	✅ success
push_repo_memory	⏭ skipped
safe_outputs	⏭ skipped

---

Investigation Findings

1. Infrastructure failure, not code failure: The rlm-rs codebase, tests, and builds are unaffected. This is a failure in the gh-aw agentic workflow framework's internal threat detection step.

2. Recurring pattern: Same failure in runs 23374387133 (07:00 UTC), 23375752826 (08:28 UTC), 23334966255 (2026-03-20), 23292567345 (2026-03-19), and many more.

3. Auto-retry partially mitigates: Workflow retries (workflow_run trigger) succeed about 70% of the time, suggesting the issue is intermittent at the infrastructure level but systematically triggered by schedule events.

4. Safe outputs skipped: Because the detection job fails, the safe_outputs job is also skipped — meaning results from successful agent runs may not be posted on some days.

---

Recommended Actions

• Monitor the safe_outputs skip pattern to determine if Daily Test Improver agent results are being lost
• Check gh-aw version: The repo recently bumped github/gh-aw from 0.56.2 to 0.58.3 (PR ci: bump github/gh-aw from 0.56.2 to 0.58.3 #140, 2026-03-16); verify if this version has known issues with the detection job on scheduled runs
• Consider gh-aw issue report: This appears to be a framework-level bug in github/gh-aw affecting Setup threat detection — report upstream if not already known
• Workflow configuration: Review whether the detection job's failure should block the overall workflow conclusion

---

Prevention Strategies

1. Update to latest github/gh-aw if a fix is available for the Setup threat detection failure
2. Consider making the detection job non-blocking (continue-on-error: true) if threat detection failures should not fail the overall workflow
3. Track safe_outputs skip rate to detect when agent results are being silently lost

---

AI Team Self-Improvement

Add to instructions.md for AI coding agents:

When investigating CI failures in this repository:
- Check if the failure is in the `detection` job at "Setup threat detection" — 
  this is a known recurring infrastructure issue in the gh-aw framework, 
  NOT a code bug in rlm-rs. The agent job completing successfully means 
  test improvement work succeeded even when detection fails.
- Always distinguish between gh-aw framework failures (detection, safe_outputs jobs)
  and actual code failures (ci.yml build/test/clippy jobs).

---

Historical Context

• Run 23374387133 — same failure pattern 90 minutes before this run
• Run 23334966255 (2026-03-20) — same pattern
• Run 23292567345 (2026-03-19) — same pattern
• This appears to be a systemic issue since at least 2026-03-19, coinciding approximately with the gh-aw bump in ci: bump github/gh-aw from 0.56.2 to 0.58.3 #140 (merged 2026-03-16)

AI generated by CI Failure Doctor · history

• expires on Mar 28, 2026, 8:40 AM UTC