feat(jobs): push/resume coordinator + /resume (milestone 3) - #87
Merged
Conversation
Closes the push/resume loop end to end (ADK). When a finished long-running job opted in (resume_on_complete), the agent is auto-resumed with its result — no polling. - BaseCLIApp.resume_finished_jobs(): drains JobManager.awaiting_resume() into serialized resume turns (one at a time via a new _turn_lock; marks resumed before running so a crash can't double-fire). Called at turn boundaries when job_auto_resume is on, and by /resume on demand (ungated). - MessageProcessor.process_resume(): streams resume_with_job_result through the exact same rendering as a user turn. Factored the shared turn machinery (events box, HITL callback, Ctrl+C cancel, retry, token accounting) out of process() into _run_turn(source_factory). - job_auto_resume setting (default off); /resume command (ResumeCommand); JobMonitor shows "↻N to resume" when enabled. - examples/jobs_demo.py sets job_auto_resume=True to demo the feature. Tests: coordinator drain/order/guards with fakes (tests/cli/test_resume_ coordinator.py); status-bar resume cue (tests/cli/test_job_monitor.py); and a live full-loop test driving the real coordinator -> process_resume -> resume_with_job_result -> model (tests/integration/test_live_job_resume.py:: test_full_resume_loop_via_coordinator). Offline 1582 passed; 3 live tests pass. The coordinator + association are backend-agnostic; only resume_with_job_result is ADK-specific so far. LangGraph resume is milestone 5.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Milestone 3 of phase-2 push/resume (design:
docs/plans/2026-06-15-job-push-resume-design.md). Builds on the association layer (#85) and the ADK execution primitive (#86). This closes the loop end to end (ADK): when a finished long-running job opted in (resume_on_complete), the agent is automatically resumed with its result — no polling.What it adds
Coordinator (
cli/app.py) —BaseCLIApp.resume_finished_jobs()drainsJobManager.awaiting_resume()into serialized resume turns: one turn at a time via a newself._turn_lock(a user turn and a resume turn never overlap — shared session + thinking boxes), marking eachresumedbefore running so a crash can't double-fire. Called at turn boundaries whenjob_auto_resumeis on, and by/resumeon demand (ungated).Rendering (
cli/message_processor.py) —process_resume()streamsworkflow.resume_with_job_result(record)through the exact same path as a user turn. Factored the shared turn machinery (events box, HITL callback, Ctrl+C cancel, rate-limit retry, token accounting) out ofprocess()into_run_turn(source_factory);processandprocess_resumeare now thin wrappers.Gating / UX —
job_auto_resumesetting (default off);/resumecommand (ResumeCommand);JobMonitorshows↻N to resumein the status bar when enabled.examples/jobs_demo.pysetsjob_auto_resume=Trueto demo it.Why turn-boundary (not spontaneous mid-idle)
Running a full agent turn's output from a background task while the user sits at the live prompt is display-unsafe (the milestone-2 constraint:
thinking_promptboxes are turn-oriented,add_*prints directly). v1 resumes right after the current/next user turn, plus/resume. The status-bar cue tells the user a resume is waiting. (Spontaneous idle wake is a possible gated v2.)Testing
tests/cli/test_resume_coordinator.py— drain resumes each awaiting job once; marks resumed before processing; returns 0 when not ready / no JobManager.tests/cli/test_job_monitor.py—↻N to resumecue shown only whenjob_auto_resumeand there are pending unresumed jobs.tests/integration/test_live_job_resume.py::test_full_resume_loop_via_coordinator,@pytest.mark.llm) — drives the real coordinator: turn 1 starts the job → job finishes →resume_finished_jobs→process_resume→resume_with_job_result→ model reacts, rendered into aRecordingSession. Passes.Scope
The coordinator and association layer are backend-agnostic; only
resume_with_job_resultis ADK-specific so far. LangGraph resume is milestone 5 (update_statewith aToolMessagekeyed bycall_id). Restart recovery (resuming jobs that finished while the CLI was down) is the remaining milestone-4 item.