Triage agent: flag issues that reference open PRs before auto-labeling ready-to-code

[fullsend-ai-retro]

What happened

Issue #1639 was filed by ifireball as a follow-up from reviewing ADR 0041 on PR #1578. The issue body explicitly referenced PR #1578 as the source context. The triage agent labeled it ready-to-code within 5 minutes, the code agent created PR #1642, and the review agent approved it. 24 minutes later, the human closed both the issue and PR, applying the changes directly on PR #1578 instead. The full agent pipeline (triage + code + review) produced work that was discarded.

What could go better

The triage agent could detect when an issue body references an open PR (e.g., 'found during review on PR #1578') and flag this before auto-labeling ready-to-code. In such cases, the changes may belong on the referenced PR rather than a new one. This is a modest improvement — the waste here was small (doc-only changes, ~15 min pipeline), and it's hard to distinguish legitimate follow-up issues from ones the human intends to handle inline. Confidence: medium-low. This pattern may not be common enough to warrant added triage complexity. Existing issues #1647 (verify issue open), #576 (superseded close reason), and #1414 (companion PR detection) cover adjacent but distinct scenarios — none specifically address detecting open-PR references in issue body at triage time.

Proposed change

In the triage agent's instructions (agent definition or skill), add a heuristic: when the issue body contains references to open PRs (e.g., 'PR #NNN', 'pull/NNN'), the triage agent should check if those PRs are still open. If so, include a note in the triage comment suggesting the issue author consider whether changes should be applied directly to the referenced PR, and optionally use a label like needs-decision instead of ready-to-code to pause auto-coding. This should be a soft signal (comment note) rather than a hard gate, since many follow-up issues legitimately warrant separate PRs.

Validation criteria

Over the next 10 issues where the body references an open PR, the triage agent should include a note about the referenced PR in at least 80% of cases. Track whether any of those issues are subsequently closed as superseded — if the rate drops compared to the current baseline, the heuristic is adding value.

---

Generated by retro agent from #1642

[fullsend-ai-triage]

Triage Summary

Severity: Low | Category: Feature

This is a well-scoped feature request to add an open-PR detection heuristic to the triage agent. The motivation is clear: when an issue body references an open PR (e.g., "found during review on PR #1578"), the triage agent should flag this before auto-labeling ready-to-code, since the changes may belong on the referenced PR rather than a new one.

Proposed Approach

1. Scan issue body for PR reference patterns (PR #NNN, pull/NNN)
2. Check if matched PRs are still open
3. If open PRs found, add a note in the triage comment and consider using a soft label (e.g., needs-decision) instead of ready-to-code
4. This should be a soft signal, not a hard gate — many follow-up issues legitimately warrant separate PRs

Related Issues

• Triage agent should not label ready-to-code when it identifies an existing PR that addresses the issue #1321 — Don't label ready-to-code when triage identifies an existing PR (distinct: that's about triage's own PR search, not issue body references)
• Expand ISSUE_HAS_PR check to detect PRs that mention (not just formally link) an issue #1320 — Expand ISSUE_HAS_PR to detect mentions, not just formal links (complementary but different mechanism)
• Triage agent: support "superseded" close reason for issues decomposed into multiple successors #576 — Superseded close reason (adjacent workflow concern)

Proposed Test Case

Scenario: Issue body references an open PR
  Given an issue with body containing "found during review on PR #42"
  And PR #42 is currently open
  When the triage agent processes the issue
  Then the triage comment includes a note about the open PR reference
  And the agent considers a soft label instead of ready-to-code

Scenario: Issue body references a closed PR
  Given an issue with body containing "found during review on PR #42"
  And PR #42 is closed/merged
  When the triage agent processes the issue
  Then no open-PR note is included

Edge cases:
  - Multiple PR references (mix of open/closed)
  - PR references inside code blocks (should be ignored)
  - Cross-repo PR references

The validation criteria in the issue (80% detection rate over 10 issues referencing open PRs) provide a good acceptance bar.

---

Labels: This is a triage agent feature request for improving the dispatch pipeline, warranting agent/triage and type/feature labels with low priority given the author's own medium-low confidence assessment.

[fullsend-ai-prioritize]

RICE Priority Score: 0.15

Score breakdown

Dimension	Score	Reasoning
Reach	0.5	This affects the internal dev team using the fullsend agent pipeline. The pattern of issues referencing open PRs is acknowledged by the author as potentially uncommon ('medium-low confidence' that it warrants added complexity). Only a few team members would benefit from reduced wasted pipeline runs.
Impact	0.5	Low impact — the waste described was small (~15 min pipeline on doc-only changes) and the human workaround is trivial (close the unnecessary PR). This is a soft optimization to the triage workflow, not fixing a blocking problem.
Confidence	0.6	The issue is well-described with clear proposed changes, test cases, and validation criteria, which raises confidence. However, the author's own 'medium-low' confidence that this pattern is common enough, plus uncertainty about false-positive rates on the heuristic, keeps confidence at medium.
Effort	1	Requires adding a PR-reference detection heuristic to the triage agent: regex scanning for PR patterns, API calls to check PR state, conditional logic for labeling and comment notes, plus handling edge cases (code blocks, cross-repo refs, mixed open/closed). Medium complexity touching the triage agent definition.

Formula: (0.5 x 0.5 x 0.6) / 1 = 0.15