Skip to content

Evaluate whether note label issues provide value or should be eliminated #4

@markgar

Description

@markgar

Context

The current review pipeline uses a two-tier severity system for GitHub Issues:

  • finding (bug/security) — filed by the commit watcher, acted on immediately by the builder
  • note (cleanup/robustness) — filed by the commit watcher as observations, only promoted to finding by the milestone reviewer if a pattern recurs in 2+ locations

The intent was to reduce noise for the builder by deferring low-severity issues until the milestone reviewer confirms they represent a real pattern. In practice, it's unclear whether note issues are providing value or just adding complexity.

Questions to investigate

  1. Are notes ever getting promoted? Do milestone reviews actually promote notes to findings, or do they just accumulate and get ignored?
  2. Should notes just be findings? The commit watcher could file everything as finding and let the builder triage. Simpler pipeline, fewer moving parts.
  3. Should notes be eliminated entirely? If cleanup/robustness issues aren't worth the builder's time at all, maybe the commit watcher should only file bug/security issues and skip the rest.
  4. Is the frequency filter (2+ locations) the right threshold? Maybe one-off cleanup issues are still worth fixing but the current system filters them out.
  5. Impact on the issue builder: The dedicated issue builder only picks up bug/finding issues. If notes were findings, the issue builder would handle them too. Is that desirable?
  6. Cost/token overhead: Each note issue costs API calls to create and later evaluate. Is the two-pass approach (file note → milestone reviewer evaluates) worth the token spend vs. just filing or not filing?

Affected code

  • src/agentic_dev/prompts/reviewer.py — commit watcher prompts that distinguish note vs finding
  • src/agentic_dev/prompts/reviewer.py — milestone reviewer prompt that does the frequency filter / promotion
  • src/agentic_dev/milestone_reviewer.py — milestone review orchestration
  • src/agentic_dev/watcher.py — commit watcher orchestration
  • src/agentic_dev/builder.py — issue builder only queries bug/finding
  • src/agentic_dev/utils.pylist_open_issues_for_milestones() filters by severity label

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions