Skip to content

fix: harden auto-pilot stall handoffs - #2826

Merged
stranske merged 3 commits into
mainfrom
fix/autopilot-stall-handoff-safeguards
Aug 1, 2026
Merged

fix: harden auto-pilot stall handoffs#2826
stranske merged 3 commits into
mainfrom
fix/autopilot-stall-handoff-safeguards

Conversation

@stranske

@stranske stranske commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Source: Issue #2864

Closes #2864

Automated Status Summary

Scope

Stalled PR recovery must route to a viable registered agent, dispatch an available handoff workflow, and avoid permanently setting agent:auto when dispatch fails.

Context for Agent

Related Issues/PRs

Tasks

  • Use registry-derived required-secret availability.
  • Allow a single viable alternate agent.
  • Use the available root or consumer handoff workflow safely.
  • Apply routing changes only after a successful dispatch and keep the template aligned.
  • Resolve all current review feedback.

Acceptance criteria

  • Workflows and consumer templates follow the same safe handoff semantics.
  • Failed dispatch leaves retryable routing state.
  • Focused tests and workflow validation pass.

Summary by CodeRabbit

  • Bug Fixes
    • Improved stalled work rotation by preserving trial history and preventing conflicting routing labels.
    • Improved detection of available credentials and eligible fallback agents.
    • Added reliable fallback handling when handing off stalled pull request monitoring.
    • Clear errors are now reported when no handoff workflow can be started.
  • Chores
    • Updated internal attempt tracking metadata.

@cursor

cursor Bot commented Jul 24, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 5 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 00f484d9-90f2-4464-aec7-45d7a5b18b0d

📥 Commits

Reviewing files that changed from the base of the PR and between 109ed14 and 0d50588.

📒 Files selected for processing (2)
  • .github/workflows/agents-auto-pilot.yml
  • templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
📝 Walkthrough

Walkthrough

The auto-pilot workflows now preserve stall-rotation state, assess keepalive agents against available credentials and current assignment, and explicitly dispatch ordered keepalive workflows. The worker attempt metadata also advances to PR 2826.

Changes

Stall rotation and keepalive handoff

Layer / File(s) Summary
Belt rotation state handling
.github/workflows/agents-auto-pilot.yml, templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
Belt rotation removes agent:auto before explicit routing and creates missing tried-marker labels.
Credential-aware eligibility
.github/workflows/agents-auto-pilot.yml, templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
Monitor-pr exports credential flags, checks repository secrets, filters keepalive agents by required secrets, and compares eligible agents with the current labeled agent.
Explicit keepalive workflow dispatch
.github/workflows/agents-auto-pilot.yml, templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
Monitor stalls dispatch ordered keepalive workflows with fallback handling and log the dispatched workflow identifier.

Worker attempt metadata

Layer / File(s) Summary
Attempt record update
langsmith-fleet-worker-attempt.json
The emitted timestamp and PR number change to the attempt for PR 2826. Other attempt settings remain unchanged.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant monitor-pr
  participant agent_stall_rotation.js
  participant GitHubSecrets
  participant KeepaliveWorkflows
  monitor-pr->>agent_stall_rotation.js: load registry and current agent
  monitor-pr->>GitHubSecrets: verify available repository secrets
  GitHubSecrets-->>monitor-pr: secret names
  monitor-pr->>agent_stall_rotation.js: compute eligible keepalive agents
  agent_stall_rotation.js-->>monitor-pr: eligible alternatives
  monitor-pr->>KeepaliveWorkflows: dispatch ordered handoff workflow
  KeepaliveWorkflows-->>monitor-pr: dispatch result
Loading

Possibly related PRs

Suggested labels: verify:compare

Suggested reviewers: copilot

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning The langsmith-fleet-worker-attempt.json metadata update changes a PR number and timestamp without supporting the linked handoff-routing objectives. Remove the unrelated metadata update or provide a requirement that directly connects it to the stalled-PR handoff changes.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: strengthening auto-pilot stall handoff behavior.
Linked Issues check ✅ Passed The workflow and consumer template implement registry-based secret checks, alternate-agent handling, safe dispatch fallback, and retryable stall routing [#2864].
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/autopilot-stall-handoff-safeguards

Comment @coderabbitai help to get the list of available commands.

@stranske
stranske temporarily deployed to agent-high-privilege July 24, 2026 22:28 — with GitHub Actions Inactive
@stranske-keepalive

stranske-keepalive Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Workflow source detected

PR #2826 now has valid workflow source context (origin=github_issue ref=#2864).

A linked GitHub issue is present for this PR.

@agents-workflows-bot

agents-workflows-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Automated Status Summary

Head SHA: 1e671dc
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / guard
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job Result Logs
(no jobs reported) ⏳ pending

Coverage Overview

  • Coverage history entries: 0

Updated automatically; will refresh on subsequent CI/Docker completions.


Keepalive checklist

Scope

No scope information available

Tasks

  • No tasks defined

Acceptance criteria

  • No acceptance criteria defined

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bdeee7d91e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/agents-auto-pilot.yml Outdated
Comment thread .github/workflows/agents-auto-pilot.yml
Comment thread .github/workflows/agents-auto-pilot.yml Outdated
@stranske
stranske force-pushed the fix/autopilot-stall-handoff-safeguards branch from bdeee7d to cde185e Compare July 25, 2026 21:58
@stranske
stranske temporarily deployed to agent-high-privilege July 25, 2026 21:58 — with GitHub Actions Inactive

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/agents-auto-pilot.yml:
- Around line 3370-3404: Extract the duplicated ordered workflow-dispatch
fallback logic from the current block and the local
dispatchFirstAvailableWorkflow implementation into a shared helper under
.github/scripts/*.js. Update both github-script steps to import and reuse that
helper, preserving ordered attempts, 404-only fallback behavior, failure
collection, and the existing no-dispatch error semantics.
- Around line 3370-3404: Ensure the agent:auto label update occurs only after
the dispatch loop successfully sets dispatchedWorkflow. Move the unconditional
label-add logic associated with the handoff below the successful dispatch
confirmation, or remove the label when both workflow dispatch attempts fail, so
the outer catch cannot leave agent:auto blocking future retries.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 7b098871-9dde-44fd-a747-f822c7b62d2d

📥 Commits

Reviewing files that changed from the base of the PR and between 9aa4404 and cde185e.

📒 Files selected for processing (2)
  • .github/workflows/agents-auto-pilot.yml
  • templates/consumer-repo/.github/workflows/agents-auto-pilot.yml

Comment thread .github/workflows/agents-auto-pilot.yml
@stranske stranske added agent:codex Agent-created issues from Codex agents:keepalive Use to initiate keepalive functionality with agents autofix Opt-in automated formatting & lint remediation agent:retry Add to trigger agent retry after rate limit or pause labels Aug 1, 2026
@stranske
stranske temporarily deployed to agent-high-privilege August 1, 2026 00:13 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Runner dispatch state for codex on PR #2826. Do not edit.

Copilot AI review requested due to automatic review settings August 1, 2026 00:15
@agents-workflows-bot

agents-workflows-bot Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

🤖 Bot Comment Handler

  • Agent: codex
  • Bot comments to address: 5

The agent has been assigned to this PR to address the bot review comments.

Instructions for agent

  1. Implement suggested fixes that improve the code
  2. Skip suggestions that don't apply (note why in your response)

The bot comment handler workflow has prepared context in the artifacts.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens auto-pilot’s stalled-PR recovery by improving agent handoff routing, making secret-availability checks registry-driven, and ensuring the workflow can safely dispatch an available keepalive handoff workflow from either the consumer surface or the repo root.

Changes:

  • Remove stale agent:auto before assigning an explicit next agent during belt stall rotation.
  • Derive keepalive eligibility from the agent registry’s required_secrets, and attempt dispatch via either agents-81-gate-followups.yml (consumer) or agents-keepalive-loop.yml (root) to avoid dead-end handoffs.
  • Update LangSmith worker attempt metadata for this PR run.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/agents-auto-pilot.yml Adds stall-handoff hardening: clears conflicting routing labels, computes registry-based secret availability, and dispatches an available keepalive handoff workflow.
templates/consumer-repo/.github/workflows/agents-auto-pilot.yml Mirrors the same hardened handoff semantics for consumer templates to keep behavior aligned.
langsmith-fleet-worker-attempt.json Updates emitted metadata (timestamp / PR number) for the recorded worker attempt.
Suppressed comments (2)

.github/workflows/agents-auto-pilot.yml:3404

  • If no handoff workflow can be dispatched, this path currently leaves agent:auto on the issue/PR. That contradicts the stated goal of keeping routing state retryable when dispatch fails (a later run will see hasAuto and skip delegation). Consider cleaning up agent:auto before throwing so the next monitor attempt can retry delegation once dispatch becomes available.
                  if (!dispatchedWorkflow) {
                    throw new Error(
                      `No keepalive handoff workflow could be dispatched: ${dispatchFailures.join('; ')}`
                    );
                  }

templates/consumer-repo/.github/workflows/agents-auto-pilot.yml:3404

  • If no handoff workflow can be dispatched, this path currently leaves agent:auto on the issue/PR. That contradicts the stated goal of keeping routing state retryable when dispatch fails (a later run will see hasAuto and skip delegation). Consider cleaning up agent:auto before throwing so the next monitor attempt can retry delegation once dispatch becomes available.
                  if (!dispatchedWorkflow) {
                    throw new Error(
                      `No keepalive handoff workflow could be dispatched: ${dispatchFailures.join('; ')}`
                    );
                  }

Comment thread templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
Comment thread .github/workflows/agents-auto-pilot.yml
@stranske

stranske commented Aug 1, 2026

Copy link
Copy Markdown
Owner Author

Mergeability restored — conflict resolved, out-of-scope artifact dropped

Merged main (dbce5dd3) into this branch as 3bd36a77. The PR is now MERGEABLE.

The conflict was almost entirely artificial. Diffing the merge base (9aa4404d) against main, the only change main made to agents-auto-pilot.yml since this branch diverged was the actions/setup-python v6 -> v7 SHA bump (ece7cb06 -> 5fda3b95), one line. Both the root and consumer-template copies three-way merged with zero conflicts, and the merged copies remain byte-identical to each other.

The one true conflict was langsmith-fleet-worker-attempt.json, a per-run artifact this branch modified and main untracked in #2856 ("chore: untrack the per-run langsmith worker-attempt artifact"). Took main's deletion, so that artifact is no longer part of this PR. The changed-file list is now exactly the two workflow copies, which is what #2864 scopes.

Verified after the merge: no conflict markers, the v7 SHA is present and the v6 SHA is gone, and both copies parse as YAML.

What remains is the review feedback in #2864's task list — six unresolved threads on the stalled-PR handoff routing (chatgpt-codex-connector x3, coderabbitai, copilot-pull-request-reviewer x2). That is substantive routing-logic review, not a mechanical fix, so it is recorded as the next closer complex lane rather than attempted in the same pass as the conflict.

The Actions-approval outage that previously blocked this PR is no longer the gating factor: the conflict is resolved without keepalive, and the remaining review threads can be addressed directly by the closer lane the same way #2867's were. The scoped blocker on #2864 is cleared accordingly.

@stranske
stranske force-pushed the fix/autopilot-stall-handoff-safeguards branch from 3bd36a7 to 0d50588 Compare August 1, 2026 03:25
@stranske
stranske temporarily deployed to agent-high-privilege August 1, 2026 03:25 — with GitHub Actions Inactive
@stranske
stranske merged commit 31339e1 into main Aug 1, 2026
45 checks passed
@stranske
stranske deleted the fix/autopilot-stall-handoff-safeguards branch August 1, 2026 03:39
@stranske stranske added the verify:compare Compare multiple LLM evaluations label Aug 1, 2026
@stranske
stranske temporarily deployed to agent-high-privilege August 1, 2026 03:39 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Provider Comparison Report

Provider Summary

Provider Model Verdict Confidence Summary
openai gpt-5.6-terra PASS 84% The root workflow and consumer template receive matching changes, preserving consistent handoff behavior. The implementation derives required credentials from keepalive-capable registry entries, ev...
anthropic claude-sonnet-5 PASS 62% The visible portion of the diff shows concrete, sensible hardening: registry-derived required-secret availability (mapping HAS_ env vars filtered by agents with pr_keepalive capability), sa...
📋 Full Provider Details (click to expand)

openai

  • Model: gpt-5.6-terra
  • Verdict: PASS
  • Confidence: 84%
  • Scores:
    • Correctness: 9.0/10
    • Completeness: 9.0/10
    • Quality: 8.0/10
    • Testing: 6.0/10
    • Risks: 8.0/10
  • Summary: The root workflow and consumer template receive matching changes, preserving consistent handoff behavior. The implementation derives required credentials from keepalive-capable registry entries, evaluates alternate-agent viability, and uses available workflow discovery/dispatch behavior rather than assuming one fixed handoff file. Routing-label mutation is structured to occur only after a successful dispatch path, so a dispatch failure does not prematurely commit the issue to agent:auto or an alternate routing state. The added label handling also removes stale agent:auto state before applying explicit routing and records tried agents. The main residual weakness is the absence of targeted automated regression tests in this change; this does not undermine the functional implementation.
  • Concerns:
    • No focused test files were added or updated alongside the workflow logic. Existing validation may cover syntax/imports, but direct regression coverage for dispatch failure, exactly-one-alternate selection, and root-versus-consumer workflow discovery would improve confidence.

anthropic

  • Model: claude-sonnet-5
  • Verdict: PASS
  • Confidence: 62%
  • Scores:
    • Correctness: 7.0/10
    • Completeness: 6.0/10
    • Quality: 7.0/10
    • Testing: 6.0/10
    • Risks: 6.0/10
  • Summary: The visible portion of the diff shows concrete, sensible hardening: registry-derived required-secret availability (mapping HAS_ env vars filtered by agents with pr_keepalive capability), safe removal of agent:auto before assigning a new agent, and idempotent creation of a 'tried' marker label. This aligns with tasks 1 and partially 4/5. However, the diff was truncated right before the section implementing the actual handoff dispatch and the gating of label/routing changes on dispatch success — the most safety-critical part of the acceptance criteria (avoiding permanent agent:auto on failed dispatch, retryable state) — so full correctness of that logic cannot be independently verified from the given diff. CI (pr-00-gate, pr-11-ci-smoke, selftest-ci) all passed on the merge commit, including Python/JS test suites and YAML validation, which supports functional correctness and template consistency. Given the visible code aligns well with the stated intent and CI confirms no regressions, but full inspection of the dispatch-then-route ordering was not possible, this is a PASS with moderate confidence rather than high confidence.
  • Concerns:
    • The full diff was truncated before the acceptance-critical section that dispatches the handoff workflow and gates routing changes on dispatch success, so 'apply routing changes only after successful dispatch' and 'failed dispatch leaves retryable routing state' cannot be fully confirmed from the available code alone.
    • The 'allow a single viable alternate agent' logic (eligibleAgents/currentAgentFromLabels usage) is referenced but its concrete decision logic is not visible in the truncated diff.
    • Both root workflow and consumer template are modified with identical diff stats (+119/-4 each), which is a good sign of template parity, but exact byte-for-byte alignment cannot be verified from the summary alone.
    • No new test files are listed in the file-changes summary; if 'focused tests' were added/updated they may live in already-existing test files not shown in the diff, making test coverage for the acceptance criteria hard to confirm from this view alone.

Agreement

  • Verdict: PASS (all providers)
  • Quality: scores within 1 point (avg 7.5/10, range 7.0-8.0)
  • Testing: scores within 1 point (avg 6.0/10, range 6.0-6.0)

Disagreement

Dimension openai anthropic
Correctness 9.0/10 7.0/10
Completeness 9.0/10 6.0/10
Risks 8.0/10 6.0/10

Unique Insights

  • openai: No focused test files were added or updated alongside the workflow logic. Existing validation may cover syntax/imports, but direct regression coverage for dispatch failure, exactly-one-alternate selection, and root-versus-consumer workflow discovery would improve confidence.
  • anthropic: The full diff was truncated before the acceptance-critical section that dispatches the handoff workflow and gates routing changes on dispatch success, so 'apply routing changes only after successful dispatch' and 'failed dispatch leaves retryable routing state' cannot be fully confirmed from the available code alone.; The 'allow a single viable alternate agent' logic (eligibleAgents/currentAgentFromLabels usage) is referenced but its concrete decision logic is not visible in the truncated diff.; Both root workflow and consumer template are modified with identical diff stats (+119/-4 each), which is a good sign of template parity, but exact byte-for-byte alignment cannot be verified from the summary alone.; No new test files are listed in the file-changes summary; if 'focused tests' were added/updated they may live in already-existing test files not shown in the diff, making test coverage for the acceptance criteria hard to confirm from this view alone.

🔍 LangSmith Traces

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:codex Agent-created issues from Codex agent:retry Add to trigger agent retry after rate limit or pause agents:keepalive Use to initiate keepalive functionality with agents autofix Opt-in automated formatting & lint remediation verify:compare Compare multiple LLM evaluations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harden auto-pilot stalled-PR handoff routing

3 participants