feat(jobs): graceful restart handling for push/resume (milestone 4 / A) - #88
Merged
Merged
Conversation
A finished resume-flagged job whose originating conversation is gone (e.g. after a CLI restart — ADK's default session is in-memory) no longer fires a dead "resuming" turn with no output. Instead the harness posts a one-line notice and the result stays reachable by id. - BaseWorkflowManager.can_resume(record): default False (no resume support). GoogleADKWorkflowManager overrides it to require the resume ids AND a live session that still holds the pending call. - MessageProcessor.process_resume gates on hasattr(resume_with_job_result) AND await workflow.can_resume(record): resumable → resume turn as before; not resumable → "✗/✓ job 'x' finished while its conversation was unavailable — fetch with /jobs <id>" + return (coordinator already marks it resumed, so it notifies once; the result is reachable via /jobs). Restart UX needs no spontaneous startup turn (that would repaint the live prompt): awaiting_resume is persisted, the monitor's "↻N to resume" cue shows it after a restart, and the next turn boundary / explicit /resume drains it (resume when the conversation survived, notice when it didn't). Tests: ADK can_resume true/false on session presence + missing ids (tests/workflow/test_adk_job_resume.py); process_resume resume-vs-notice branching incl. a backend with no resume support (tests/cli/test_process_resume.py). Offline 1588 passed; full-loop live test still passes through the can_resume gate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Milestone 4 (Option A — graceful restart handling) of phase-2 push/resume. Builds on the coordinator (#87). Makes the "job finished but its conversation is gone" case correct instead of broken — most relevant after a CLI restart, since ADK's default
InMemorySessionServicemeans the conversation (and the pending call the resume must match) is gone on the next run.The problem
The job and its resume association survive a restart (subprocess sentinel + persisted
JobRecord), but the ADK conversation does not. Before this, draining such a job calledprocess_resume, which posted "↻ resuming…" and then streamed nothing (the session/call were gone) — an orphaned, confusing turn.What it adds
BaseWorkflowManager.can_resume(record)— defaultFalse(no resume support).GoogleADKWorkflowManageroverrides it to require the resume ids and a live session that still holds the pending call.MessageProcessor.process_resumenow gates onhasattr(resume_with_job_result) and await workflow.can_resume(record):✗/✓ Background job 'x' finished (…) while its conversation was unavailable — fetch the result with /jobs <id>and returns. The coordinator already marks it resumed, so it notifies once; the result stays reachable via/jobs.Restart UX (no new startup turn)
Deliberately no spontaneous startup resume — running a full turn's output over the live prompt is the same display-unsafe situation that kept the monitor to the status bar. Instead:
awaiting_resumeis persisted, the monitor's↻N to resumecue shows it after a restart, and the next turn boundary / explicit/resumedrains it — resuming when the conversation survived, posting the notice when it didn't.Why not durable cross-restart resume (Option B)?
That requires persistent ADK sessions (
--session-id+ validating/fixing_inject_session_messagesfor pending long-running calls) — a session-persistence project, highest effort/lowest certainty. Durable resume is more natural on LangGraph's checkpointer (Option C / milestone 5).Testing
tests/workflow/test_adk_job_resume.py—can_resumetrue when the session is present, false when missing or when ids are absent.tests/cli/test_process_resume.py— resume-vs-notice branching, incl. a backend with no resume support (LangGraph today) → notice.can_resumegate.