ADR Pipeline Job Model

---

status: accepted decided: 2026-10-07 decision: "Prep as an additive container; independently created and executed jobs writing directly to target tables" related_issues: [56, 41]

ADR: Pipeline Job Model

Status: Accepted Related: Architecture · GitHub #56 (runtime-agnostic pipeline) · #41 (resilient upload)

---

Context

• The pipeline must run from the browser (worker), a backend, or a CLI.
• Users, and later sponsors with their own API key, trigger only the steps they need, and can enrich a prep repeatedly: 5 questions now, 5 more later, cheat sheets or podcasts after that.
• Current storage doesn't allow this: one run per prep, concepts and tasks as JSON, outputs copied to the real tables only at the end, cost tied to an entity that may not exist, authorization baked into repository functions.

Decision

1. Prep is a container; a job is an additive enricher. No pipeline "run". Job types: OCR (per page), concepts, question; later others. A new type needs no core change.
2. Creation is separate from execution. A job is created with its resolved config (never the API key) and its input ids. Anyone allowed can execute it later, in any runtime.
3. Statuses: created → executed | failed(reason). failed is retriable. No leases, no polling. Commit is one transaction: write outputs and flip status with a conditional update. If two executors race, the loser's output is discarded — wasted spend, never duplicates.
4. Dependencies are existing entities only. A job is created only when the entities it reads exist. The caller/planner chains after upstream jobs execute (OCR → concepts → N question jobs). No job-to-job graph. Job inputs are a join table, so the dependency graph stays queryable.
5. One question job = the whole build → review → rewrite loop, bounded by an attempt limit. No intermediate state; failure reruns the loop. Splitting into build and review jobs later is additive.
6. Jobs write straight into target tables — Page, Concept, Question — each carrying producedByJobId. No staging, no copy step. The previous run/slot tables and attempt payloads are dropped, with no data migration.
7. Cost lives on the job. One generation-meta row per LLM call, with a wasted flag, linked to the job (replacing the polymorphic entity link).
   • An entity's own cost: entity → job → meta.
   • Dependency cost: transitive walk over job inputs, shown as "incl. upstream" (not summable across cards).
   • Prep total: sum over jobs. The prep token counter becomes derived.
8. Retention. Executed jobs and their meta are kept for the life of the prep and cascade on prep delete. No pruning.
9. Ports. Core is pure TS: Store, Llm (OpenAI-compatible, so local models work), ProgressSink, AbortSignal; takes one fully resolved config. Adapters: server DB store, API-client store for the browser, file store for the CLI.
10. Actor vs owner. Jobs record who created and who executed them. Initially owner-only; sponsors later. The Store contract carries the actor from the start.

Consequences

• Upload pages become the Page entity; its status is transitional and derives from the OCR job once jobs land.
• Lost: per-attempt resume — a crash costs at most one extra build/review loop.
• Out of scope: leases, pruning, sponsor permissions, the browser-worker runner.