Automated nightly codebase scanner

[derekmisler]

Summary

Adds an automated nightly codebase scanner that uses a multi-agent architecture to detect code quality issues and automatically create GitHub issues for findings.

Key features:

• Multi-agent system with 5 specialized agents: orchestrator (Claude Sonnet), security analyzer (OpenAI o3-mini), bug detector (Gemini Flash), documentation checker (Claude
  Haiku), and issue reporter (Claude Haiku)
• Persistent memory via SQLite database cached between runs - learns from previous scans to avoid false positives
• Automated issue creation using gh CLI with duplicate detection and a strict 2-issue-per-run limit
• Multi-provider support leveraging different models' strengths (reasoning models for security, fast models for bugs)

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Root Orchestrator                        │
│                   (claude-sonnet-4-5)                       │
│  • Loads memory from previous scans                         │
│  • Delegates to sub-agents                                  │
│  • Filters/prioritizes findings                             │
│  • Adds CATEGORY field before forwarding to reporter        │
└─────────────────────────────────────────────────────────────┘
          │              │                │              │
          ▼              ▼                ▼              ▼
    ┌──────────┐  ┌──────────┐  ┌───────────────┐  ┌──────────┐
    │ Security │  │   Bugs   │  │ Documentation │  │ Reporter │
    │ (o3-mini)│  │ (gemini) │  │    (haiku)    │  │ (haiku)  │
    └──────────┘  └──────────┘  └───────────────┘  └──────────┘

Files Changed

• .github/agents/nightly-scanner.yaml - Multi-agent configuration with instructions, toolsets, and permissions
• .github/workflows/nightly-scan.yml - GitHub Actions workflow (runs daily at 6am UTC, supports dry-run mode)

Test plan

• Trigger workflow manually with dry-run: true to verify agents run without creating issues
• Verify the scanner memory database is created and cached
• Trigger without dry-run: true on a branch with a known issue to verify issue creation
• Verify duplicate detection prevents recreating the same issue

[krissetto]

Might be good to try out some alloy models in this and other actions, could potentially give deeper perspective to the agents and better overall results

[derekmisler]

/review

fixed important issues

[derekmisler]

Might be good to try out some alloy models in this and other actions, could potentially give deeper perspective to the agents and better overall results

sure! what did you have in mind? i'm using 4 different models right now (1 root, 3 sub-agents). is that what you mean? or, like, two sub-agents with different models that both do kind of the same thing, and then the root compares their output?

edit: i just saw that you can combine models like model: claude,gpt-4o. how did i miss that?

changes have been made

[derekmisler]

/review

[github-actions]

Review Summary

I've analyzed the PR and found 2 issues worth noting:

1. Logic flaw in documentation agent execution - Documentation issues will only be detected when there are NO security or bug issues, which contradicts the stated goal of comprehensive nightly scanning.

2. Incomplete error handling for reporter agent failures - The orchestrator has error handling for analysis agents but not for the final reporter agent, which could lead to unreported failures when creating GitHub issues.

Both issues are in the agent configuration and represent design flaws rather than critical bugs. The workflow will function but may not behave as comprehensively as intended.