Skip to content

feat(jobs): graceful restart handling for push/resume (milestone 4 / A) - #88

Merged
shoom1 merged 1 commit into
developfrom
feature/job-resume-restart-handling
Jun 18, 2026
Merged

feat(jobs): graceful restart handling for push/resume (milestone 4 / A)#88
shoom1 merged 1 commit into
developfrom
feature/job-resume-restart-handling

Conversation

@shoom1

@shoom1 shoom1 commented Jun 18, 2026

Copy link
Copy Markdown
Owner

Summary

Milestone 4 (Option A — graceful restart handling) of phase-2 push/resume. Builds on the coordinator (#87). Makes the "job finished but its conversation is gone" case correct instead of broken — most relevant after a CLI restart, since ADK's default InMemorySessionService means the conversation (and the pending call the resume must match) is gone on the next run.

The problem

The job and its resume association survive a restart (subprocess sentinel + persisted JobRecord), but the ADK conversation does not. Before this, draining such a job called process_resume, which posted "↻ resuming…" and then streamed nothing (the session/call were gone) — an orphaned, confusing turn.

What it adds

  • BaseWorkflowManager.can_resume(record) — default False (no resume support). GoogleADKWorkflowManager overrides it to require the resume ids and a live session that still holds the pending call.
  • MessageProcessor.process_resume now gates on hasattr(resume_with_job_result) and await workflow.can_resume(record):
    • resumable → resume turn, exactly as before;
    • not resumable → posts ✗/✓ Background job 'x' finished (…) while its conversation was unavailable — fetch the result with /jobs <id> and returns. The coordinator already marks it resumed, so it notifies once; the result stays reachable via /jobs.

Restart UX (no new startup turn)

Deliberately no spontaneous startup resume — running a full turn's output over the live prompt is the same display-unsafe situation that kept the monitor to the status bar. Instead: awaiting_resume is persisted, the monitor's ↻N to resume cue shows it after a restart, and the next turn boundary / explicit /resume drains it — resuming when the conversation survived, posting the notice when it didn't.

Why not durable cross-restart resume (Option B)?

That requires persistent ADK sessions (--session-id + validating/fixing _inject_session_messages for pending long-running calls) — a session-persistence project, highest effort/lowest certainty. Durable resume is more natural on LangGraph's checkpointer (Option C / milestone 5).

Testing

  • tests/workflow/test_adk_job_resume.pycan_resume true when the session is present, false when missing or when ids are absent.
  • tests/cli/test_process_resume.py — resume-vs-notice branching, incl. a backend with no resume support (LangGraph today) → notice.
  • Offline suite: 1588 passed, 26 xfailed. The live full-loop test still passes through the new can_resume gate.

A finished resume-flagged job whose originating conversation is gone (e.g. after
a CLI restart — ADK's default session is in-memory) no longer fires a dead
"resuming" turn with no output. Instead the harness posts a one-line notice and
the result stays reachable by id.

- BaseWorkflowManager.can_resume(record): default False (no resume support).
  GoogleADKWorkflowManager overrides it to require the resume ids AND a live
  session that still holds the pending call.
- MessageProcessor.process_resume gates on hasattr(resume_with_job_result) AND
  await workflow.can_resume(record): resumable → resume turn as before; not
  resumable → "✗/✓ job 'x' finished while its conversation was unavailable —
  fetch with /jobs <id>" + return (coordinator already marks it resumed, so it
  notifies once; the result is reachable via /jobs).

Restart UX needs no spontaneous startup turn (that would repaint the live
prompt): awaiting_resume is persisted, the monitor's "↻N to resume" cue shows it
after a restart, and the next turn boundary / explicit /resume drains it
(resume when the conversation survived, notice when it didn't).

Tests: ADK can_resume true/false on session presence + missing ids
(tests/workflow/test_adk_job_resume.py); process_resume resume-vs-notice
branching incl. a backend with no resume support
(tests/cli/test_process_resume.py). Offline 1588 passed; full-loop live test
still passes through the can_resume gate.
@shoom1
shoom1 merged commit 57b3936 into develop Jun 18, 2026
@shoom1
shoom1 deleted the feature/job-resume-restart-handling branch June 18, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant