Embedded job system: a JobQueue port, a SQLite adapter, and an in-process spawn dispatcher

[JArmandoAnaya]

Design v0.1.0, grounded in a read-only exploration of fef9c0e. Ships the substrate that turns
"work that outlives its request" from one bespoke thread into a port with an adapter.

Why

server/runner.py is the whole background story today: 77 lines, a one-worker
ThreadPoolExecutor, in memory, queued runs lost on restart, reachable from exactly two lambdas
(routes/sources.py:168, routes/ingest.py:68). Export, verify, publish and promote all answer
inline; runner.py:5-11 says export was meant to adopt launch-and-poll and never did
(routes/releases.py:266-267 states the opposite out loud). There is no JobQueue port, no
progress port, and ingest_service.py:69-72 says so: "What is still not here is a scheduler."

Scope

In (Phase 1):

1. JobQueue and ProgressReporter ports + a generic job domain, transition table and migration.
2. A SQLite-backed queue adapter, and an embedded dispatcher over a spawn ProcessPoolExecutor
   living in the FastAPI lifecycle. No separate worker command.
3. Ingest migrates onto it (zero wire change); IngestRunner is deleted.
4. Release export moves behind the queue — POST /releases/{id}/export becomes 202 + Location.

Out: verify, publish, promote, thumbnail backfill stay synchronous — future job types, not
Phase-1 work. Lease/heartbeat columns (pro/multi-node). Durable event outbox.

Scope correction the exploration forced: lane export is not in scope. formats/lanes/ is a
1-line __init__.py and a README stating the v1 source is not in this repository; there is no
lane_utils and there are no 91 lane tests. cf. #223.

Two naming collisions the design did not catch

Both are real and both are resolved here rather than deferred:

• /jobs is taken. server/routes/jobs.py:36 already serves annotation jobs at that prefix,
  and GET /jobs/{job_id} returns JobOut. The generic router therefore mounts at
  /background-jobs, which also reads consistently with the existing /ingest-jobs.
• JobState / JOB_TRANSITIONS are taken by domain/task.py (annotation jobs). The generic
  domain uses BackgroundJob, BackgroundJobState, BACKGROUND_JOB_TRANSITIONS.

Decisions being locked

Question	Answer	Why
Start method	spawn, pinned via mp_context	fork is disqualified by the always-live QueuePool connection; pinning makes Linux match macOS/Windows and 3.14
Workspace lifetime	per worker, cached by root	per-task open() pays pragmas + migration check + a full schema reflection every time
Pool size	default 1	runner.py:13-19 is right that one writer is a property of the store; >1 permitted, degrades to WorkspaceBusy/503
Queue storage	same visionset.db	one workspace artifact, one backup story; WAL makes the dispatcher's poll a free read
Claim	guarded UPDATE, rowcount is the answer	the set_asset_progress precedent (sqlite_metadata_store.py:276); BEGIN IMMEDIATE stays declined
Crash recovery	no lease; sweep_orphans() at boot	one server process owns all workers, so a running row at startup is an orphan by definition
running -> running	stays illegal	consistent with INGEST_TRANSITIONS' stated rationale
ingest_job	coexists	it is the domain record and the wire contract; collapsing it is a future migration with its own discussion
Events	bus stays per-process	dispatcher publishes JobSucceeded/JobFailed API-side; handler-published events reach the worker only, documented

Deviation from an earlier conversation, flagged: pool default is 1, not 2. The evidence —
per-item progress commits against a single-writer store — overrides the earlier call.

Acceptance

• Two ports, a domain, migration 4 (job created whole — a FK cannot arrive by ALTER)
• SqliteJobQueue + an embedded spawn dispatcher, lazily started, drained in the lifespan
  before the workspace closes
• Handlers registered by import string and resolved in the worker; a test imports and pickles
  every ref
• Ingest on the new executor, server/runner.py deleted, ingest wire byte-still
• Export at 202 + Location, artifact streamed from a new route; openapi.json and the
  generated client regenerated; frontend polls with the existing usePollingQuery
• VISIONSET_JOB_* settings through the repo's first pydantic-settings object
• An import-linter contract keeping handler modules out of visionset.server
• Claim atomicity raced over one file; one real-pool integration test
• bash scripts/check.sh green, all three suites. No new required CI check.

Open items carried into the PR

1. Whether export's 202 break is acceptable now or waits for a minor version.
2. GET /background-jobs/{id}/artifact vs. a generic download route — naming and TTL policy.
3. Whether job should carry a project_id FK. Proposed no: jobs are workspace-scoped
   plumbing and scoping lives in the payload.