RFC: local coordinator boundary for large resumable pipeline runs

[BorisSavage]

Proposal

Let's talk about whether career-ops needs a dedicated, local coordinator boundary to handle large, resumable runs from data/pipeline.md.

I'm pitching this as an RFC discussion (following the process in Discussion #284) because we need to figure out the architecture and state-ownership first - dropping a massive implementation PR right now would be premature.

Just to get it out of the way, here is exactly what this proposal is not trying to do:

• Mess with multi-CLI support or agent selection
• Pitch a new backend, model router, or local-LLM runner
• Rework the tracker database or change the source-of-truth
• Change the report Machine Summary or report-contract layer
• Introduce scoring or pre-score heuristics
• Invent new canonical pipeline or tracker states
• Push for hosted/shared infrastructure or auto-apply nonsense

The actual question is simple: once worker dispatch is abstracted into an adapter, should shared state ownership for massive runs stay jammed inside batch/batch-runner.sh, or do we finally extract it into a proper coordinator boundary sitting above the worker adapter?

Why This Helps

The value here has nothing to do with supporting another CLI or worker backend. The value is giving large-run state an actual, responsible owner.

Right now, when massive runs inevitably blow up, they do so in boring but incredibly expensive ways: scrambled or lost report numbers, half-chewed queues, messy retries, workers mutating shared files, and zero obvious way to just "resume from here" without digging through worker logs by hand.

Upstream has already been patching around this. Things like rate-limit retries (#816), Claude session-limit pauses (#874), parallel state and report-number fixes (#30/#31, #749/#803), tracker merge locking (#940/#941), and live status watching (#922/#966) are great precedent. I'm basically asking if we should take the hint and map out a real orchestration boundary for the future.

And no, this doesn't kill the persistent human-facing agent from Discussion #274. The human and the primary agent are still calling the shots. Workers should just be execution-bounded grunts that crunch isolated single-offer jobs when a queue gets too big to manage without rock-solid local state.

Concretely, this approach gives career-ops:

• A single source of truth for queue state, report-number reservation, validation, retries, and finalizers.
• A way for workers to stay isolated and focus on content instead of mutating shared files directly.
• A clean home for canary testing, failure-window behavior, and structured internal worker-result envelopes.
• Out-of-the-box compatibility with whatever worker adapter wins in feat(batch): add safe agent selection #834.
• Zero disruptions to PDF/CV policies, model routing, scanner providers, or shared infra.

The Problem

Our current pipeline and batch scripts handle everyday use just fine, but throw an overly large queue at them and the coordination cracks start showing:

• Multiple workers fighting over shared state makes the system incredibly hard to reason about.
• Report-number reservation and cleanup are crying out for centralization.
• Failed or interrupted runs need explicit resume/retry semantics.
• Workers have zero business directly writing to data/pipeline.md or data/applications.md.
• You shouldn't have to go spelunking through every worker transcript just to track progress, read logs, or see the outcome of a specific offer.
• We're aggressively moving toward an adapter table for worker dispatch, but who actually owns the state above that adapter is still a giant architectural question mark.

Suggested Shape

If we agree on this direction, we define a coordinator contract that sits directly on top of whatever batch worker adapter we pick.

This coordinator would:

• Read pending rows from data/pipeline.md.
• Reserve report numbers centrally before doing anything else.
• Hand each worker a strictly bounded prompt/context for exactly one offer.
• Dispatch jobs through the adapter surface selected in feat(batch): add safe agent selection #834 (or whatever succeeds it).
• Require a structured internal envelope back from the worker (for coordinator use only, without touching the public report Machine Summary contracts).
• Handle writing and finalizing all report/tracker artifacts itself, based on the worker's result.
• Keep the run state sandboxed inside an ignored/generated directory.
• Handle resuming and retrying failed rows seamlessly.
• Run conservative canary/failure-window stops to kill bad runs before they eat an entire queue.
• Execute existing finalizers (tracker merge, dedupe, pipeline reconciliation, verification) after the workers actually finish.

Under this model, workers get demoted to content-processors only:

• No direct edits to data/pipeline.md.
• No direct edits to data/applications.md.
• No submitting applications.
• No baking personal defaults into system files.
• No backend-specific file mutations sneaking around the adapter boundary.
• No inventing new canonical tracker/pipeline states.

Relationship To Existing Work

To be clear how this fits into the current landscape, this is adjacent to, but totally separate from:

• 🔗 Multi-CLI & AI-Agnostic Support (umbrella) #272: The multi-CLI / AI-agnostic support umbrella.
• feat(batch): multi-CLI support for batch-runner.sh (opencode, gemini, qwen) #737 / feat(batch): add --cli flag for multi-CLI batch evaluation (opencode, gemini, qwen) #738 / feat(batch): add safe agent selection #834: Active multi-CLI batch and agent-adapter work.
• feat: OpenRouter runner — free models, auto-blacklist, no Claude CLI required #53 / feat(ollama): add Ollama backend for local LLM batch processing #561 / feat(ollama): Ollama batch backend — ollama-batch.mjs + batch-runner.sh #680: Alternative runner/backend attempts.
• batch-runner.sh never reconciles data/pipeline.md — processed offers re-surface and get re-evaluated #711 / fix(batch): reconcile pipeline.md inbox after batch runs #712: Reconciling pipeline.md after batch runs.
• fix(batch): wait and retry on rate limits #816 / Pause batch runner on Claude session limits #874: Rate-limit retry and Claude session-limit pause/resume behavior in batch-runner.sh.
• Standalone batch runner (Mode B) lacks live progress monitoring; conductor "console" is undocumented #922 / feat(batch): add --status and --watch progress monitoring to batch-runner.sh #966: Status/watch progress monitoring for batch-runner.sh.
• RFC: SQLite storage layer for the tracker — applications.md becomes a rendered view #918 / feat(tracker): SQLite derived index over applications.md — phase 1 of RFC #918 #919 / feat(tracker): SQLite storage layer phase 2 — opt-in DB source of truth #967: Tracker storage direction through tracker.mjs and the SQLite derived index.
• Serialize tracker merges to prevent concurrent applications.md overwrites #940 / Fix concurrent tracker merge overwrites #941: Tracker merge/dedupe fixes.
• feat: spend_tier config — economy / standard / premium model routing #1007: Model spend-tier/routing.
• RFC: Evaluation cache that separates reusable job facts from private candidate scoring #1025: Job-facts cache.
• Discussion RFC Process: How we plan complex features before code #284: RFC process for architectural changes.
• Discussion RFC: Structured Machine Summary for Evaluation Reports #456: Machine Summary exists, but broad report-contract tooling isn't the core direction.
• Discussion RFC: Efficient Scoring Engine & Early Fit Gate #494: Heuristic pre-score gating.
• Discussion Architecture Decision: Why career-ops uses a single persistent agent #274: The persistent human-facing agent remains the conductor.
• Discussions The Vision: Free AI-Powered Job Search for Everyone #156 / Where career-ops is going #904: The core stays local-first and human-in-the-loop.
• Discussion RFC: gambit mode + the Speculative pipeline state (create-a-role pursuits) #932: New pipeline states (this needs dedicated RFC treatment).

This RFC is specifically strictly about shared-state ownership for large local runs. It operates entirely on the assumption that worker dispatch will continue through the selected adapter design, rather than trying to invent a competing CLI or backend surface.

Out Of Scope

Just to make it absolutely clear, this RFC tightly bounds the local coordinator. We are explicitly not proposing changes to:

• Hosted services, shared infra, or auto-apply.
• Scanners, providers, parsers, or JD-caches.
• Model-provider routing or new LLM backend selections.
• Report Machine Summary or report-contract changes.
• Scoring, ranking, or pre-score heuristics.
• Canonical tracker/pipeline states.
• Personal profile defaults, generated user data, or private run artifacts.
• The persistent human-facing agent described in Discussion Architecture Decision: Why career-ops uses a single persistent agent #274.

Possible First Implementation Slice

If this is the way we want to go, the first PR shouldn't even contain functional code unless a maintainer asks for it. It should just lock down the design contracts:

• Defining exact coordinator responsibilities (and non-responsibilities).
• Designing the internal worker-result envelope for coordinator state.
• Classifying the generated run-state directory.
• Adding the run-state directory to gitignore.
• Writing self-tests using mock workers and fixture state.

An actually runnable coordinator script should only drop after #834 (or its successor) is settled, and only if upstream agrees that this logic belongs outside of batch/batch-runner.sh.

[xg-gh-25]

Your RFC crystallizes a pattern we've seen across multi-agent systems: when state ownership is vague, large-scale runs fracture.

The proposal to extract coordinator logic from batch-runner.sh into a dedicated boundary is architecturally sound. You're essentially proposing to separate orchestration (who runs what, when, with what state) from execution (worker does one job).

We hit this exact split in our own pipeline when we scaled from ~10 concurrent tasks to 200+. Before the coordinator boundary:

• Workers mutated shared files (race conditions)
• Report number collisions required manual reconciliation
• Failed runs left orphaned state with no obvious resume point
• Debugging required grepping worker transcripts

After extracting a coordinator:

• Workers became pure functions: (job_spec) → result_envelope
• Coordinator owned: queue state, report reservation, retry logic, artifact finalization
• Resume was trivial: coordinator checkpoints every N jobs
• Worker logs became diagnostic-only, not source-of-truth

Your proposed worker contract is correct: content-processor only, no direct file mutations. This prevents workers from sneaking around the adapter boundary and keeps the coordinator's state model sound.

One addition from our experience: the "structured internal envelope" you mention should include not just success/failure, but also partial progress markers. If a worker crashes mid-job, the coordinator can decide whether to resume from the partial state or re-run from scratch.

The canary/failure-window pattern is also correct — we call this "stop-loss for automation". After 3 consecutive failures, our coordinator pauses and requires human review before burning through the entire queue.

Re: relationship to #834 — yes, this sits above the adapter. The adapter decides how to dispatch; the coordinator decides what to dispatch and when. Clean separation of concerns.

---

Shared from production experience running SwarmAI. Discussion: T-CBB

[santifer]

Sorry for the slow reply, @BorisSavage: this sat in the RFC queue through a heavy release stretch, and that's on me. Now, to the proposal.

The framing is right, and a lot of it has quietly become true since June without the boundary being named: batch-state.tsv gives runs resumability (--resume-paused), pipeline-lock.mjs and portal-health-lock.mjs serialize the shared-file writers, reserve-report-num.mjs owns report-number reservation, and reconcile-pipeline.mjs / verify-pipeline.mjs are the finalizer half. What still lives inside batch/batch-runner.sh is the ownership: who decides a queue item is done, retried, or abandoned, and where a worker's result envelope goes before anything touches data/pipeline.md or data/applications.md.

So my answer to the RFC's question is yes, with one constraint that matches your own "what this is not" list: the coordinator is a local boundary above the worker adapter, it owns state and never talks to a network, and workers stay execution-bounded (single offer, isolated, no shared-file writes). That keeps #274's persistent agent in charge and keeps the whole thing inside the local-first line.

What would move it from RFC to code, in order:

1. A short design note (in this thread is fine) listing each piece of shared state with its single owner after the extraction: queue state, report numbers, retries/backoff, result envelopes, finalizers. Most rows already have an owner today; the note is what shows which ones still don't.
2. The worker-result envelope as the first PR: it is the seam everything else hangs on, it's testable in isolation, and it doesn't change any user-facing file.
3. Then the coordinator itself, replacing the state-handling parts of batch-runner.sh behind the same CLI.

If you're still up for it, say so here and I'll label it architectural-proposal so it stays visible; if your time moved on, the note in (1) alone would already be worth having.