Skip to content

Group failure handling + crash recovery #82

Description

@hsyntax

Parent PRD

#76

What to build

Ensure groups handle failures gracefully and can recover from crashes mid-group.

Failure handling (orchestrator):

  • If the implement phase fails for a sub-issue (timeout, agent error, etc.), retry using existing timeout/retry logic (agent_timeout_retries).
  • Retries apply to the individual sub-issue, NOT the entire group. Other completed sub-issues are preserved.
  • If all retries are exhausted, leave the group in its current state for manual intervention. The group state persists with completed_sub_issues intact, and the failed sub-issue remains the current_task.
  • On the next rlph iteration: the in-progress group is detected and the failed sub-issue is retried again (or the user can manually intervene by removing the state).

Crash recovery (orchestrator + state):

  • On startup, if StateManager has a current_group with incomplete sub-issues:
    • Load the group state (parent ID, completed sub-issues, worktree path, branch).
    • Verify the worktree still exists. If not, recreate it by checking out the existing branch.
    • Determine the next eligible sub-issue (first uncompleted one whose dependencies are satisfied).
    • Resume from that sub-issue.
  • The worktree and branch are the source of truth for completed work. Group state tracks which sub-issues are done.

Edge cases:

  • Crash after agent commits but before state is updated: on recovery, detect the commit in the worktree (by checking git log for the sub-issue reference) and update state accordingly.
  • Crash after PR submission but before review: resume directly into the review pipeline.

Acceptance criteria

  • Failed sub-issue is retried (existing retry logic) without affecting completed sub-issues
  • After retry exhaustion, group state persists with completed sub-issues intact
  • On next iteration, the in-progress group is detected and the failed sub-issue is retried
  • Crash recovery: group resumes from the last completed sub-issue
  • Crash recovery: worktree is verified/recreated if missing
  • Crash recovery: commit detection reconciles state with actual git history
  • Crash after PR submission resumes into review pipeline
  • Integration test: sub-issue fails → retry succeeds → group continues
  • Integration test: simulate crash mid-group → restart → group resumes correctly
  • cargo clippy passes with zero warnings

Blocked by

User stories addressed

  • User story 19: Group state survives crashes and allows resumption
  • User story 20: Failed sub-issues retried before manual intervention

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions