feat(router): worker-image build engine (compose, build, pin, smoke-test) [MNG-1723]

[aaight]

Worker Dockerfile 3/5 — router-side build engine

Implements ticket 3/5 of the worker-Dockerfile sequence: a router-side worker-image-build dashboard job that composes an operator's extra layers onto the pinned worker base, builds a local image, pins it by immutable local image ID, runs the existing runtime smoke-test, and records verified/failed fail-closed. It is the first-ever docker.buildImage() in the tree and a strict superset of the spec-022 validation handler.

Ships operator-unreachable-but-tested: no set surface enqueues a build yet (ticket 4/5). Tested by invoking handleWorkerImageBuild directly with injected deps against fabricated dockerfile-sourced rows — Docker sits behind the injectable dep struct, so no real daemon is needed.

Linear: https://linear.app/mongrel/issue/MNG-1723/worker-dockerfile-35-build-engine-compose-build-pin-smoke-test-content

Engine contract (in order)

1. Guard — drop when the DB's worker_image_build_hash no longer equals the job's hash (superseded).
2. Base digest — resolve the global routerConfig.workerImage to an immutable repo@sha256:….
3. Compose — FROM <base>@<digest> · USER root · <operator lines> · USER node · LABEL cascade.managed=true; defensively throws on a self-declared FROM.
4. Content-hash reuse — reuse an intact local cascade-built-<projectId>:latest whose cascade.build_hash label equals the full-hash (a changed base digest ⇒ different full-hash ⇒ rebuild); skips docker build.
5. Build — docker.buildImage() from an in-memory single-file tar, tagged per-project, bounded by a wall-clock WORKER_BUILD_TIMEOUT_MS.
6. Pin — inspect the built image → immutable local image ID.
7. Smoke-test — the SAME buildWorkerImageCheckScript one-shot docker.run the validator uses, run against the pin.
8. Record fail-closed — write the pin + statuses, guarded on the build hash.

Two hashes (kept distinct)

• content-hash = sha256(raw operator content) → the worker_image_build_hash column + job identity + supersede guard (computed by the set surface in ticket 4).
• full-hash = sha256(composed + base digest) → stamped as LABEL cascade.build_hash, used only for the step-4 reuse fast path.

Fail-closed & no-strand

• Every non-verified path (self-declared FROM, build error, wall-clock timeout, smoke-test non-zero, unexpected throw) records failed with a precise reason and never leaves the project in building. Build failures start build failed:; smoke-test failures start a distinct runtime requirement missing:. Unexpected throws are additionally Sentry-captured under tag worker_image_build. The handler never rejects.
• No-strand: a rebuild that fails while a last-good verified image exists keeps worker_image_status = verified on the old pin (keepActive) and records only worker_image_build_status = failed + reason. A first build failure (no prior verified image) sets worker_image_status = failed.

GC (no new loop)

Built images carry LABEL cascade.managed=true + a stable per-project :latest tag. A successful rebuild retags :latest → the superseded digest becomes dangling → reclaimed by the existing 30-min dangling-image-cleanup loop. The active tagged image is never dangling, so a sweep can never remove the last-good image. A failed rebuild does not retag.

Secrets

Builds inject no build-args and no project credentials; the credential master key is never in scope. Public-only builds by design.

Files

New

• src/router/worker-dockerfile-compose.ts — pure composeDockerfile (throws on FROM), computeContentHash, computeFullBuildHash.
• src/router/worker-image-build.ts — WorkerImageBuildDeps + handleWorkerImageBuild (mirrors WorkerImageValidationDeps), plus default Docker deps (in-memory ustar tar builder, docker.buildImage progress-error handling, base-digest resolution, built-image inspect).

Touched

• src/db/repositories/projectsRepository.ts — recordWorkerImageBuildResult (build-hash-guarded, keepActive no-strand) + readWorkerImageBuildInputs.
• src/queue/client.ts — WorkerImageBuildJob in the DashboardJob union + enqueueWorkerImageBuildJob + workerImageBuildJobId (deterministic per-project id; re-enqueue supersedes).
• src/router/worker-manager.ts — processDashboardJob worker-image-build branch (dispatched directly, no worker slot), beside validation.
• src/router/config.ts — workerBuildTimeoutMs from WORKER_BUILD_TIMEOUT_MS (default 10 min).
• src/router/worker-image-validation.ts — extracted/exported runWorkerImageSmokeTest so the build engine reuses the exact same smoke-test.

Tests

• tests/unit/router/worker-dockerfile-compose.test.ts — compose shape, reject-FROM (case/whitespace/comment tolerant), CRLF, both hashes.
• tests/unit/router/worker-image-build.test.ts — superseded drop, happy path (verified + local pin + :latest tag), build failure (build failed:), smoke-test non-zero (runtime requirement missing:), wall-clock timeout, keep-active vs first-build failure, content-hash reuse skips build, unexpected throw caught + Sentry-captured + never rejects, stale-result drop, tar/tag helpers.
• tests/unit/queue/worker-image-build-job.test.ts — enqueue + deterministic jobId + distinct from validation.
• tests/unit/router/worker-manager.test.ts (extend) — build branch invokes handleWorkerImageBuild, not guardedSpawn.
• tests/unit/router/dangling-image-cleanup.test.ts (extend) — superseded built digest reclaimed via the existing filter; active tagged image structurally exempt.
• tests/integration/db/projectsRepository.test.ts (extend) — build-hash guard, keep-active on failed rebuild with prior verified, first-build failure sets status=failed, readWorkerImageBuildInputs.

Verification

npm run build, npm test (586 files / 10595 tests), the projectsRepository integration suite (60 tests), npm run lint (biome check --error-on-warnings on changed files), and npm run typecheck all green.

Out of scope (later tickets)

Set-mutation enqueue + reject-FROM at set time + mutual exclusivity + audit + manual-rebuild trigger (4/5); dashboard rendering + rebuild button + operator docs (5/5); build-time secrets / BuildKit / registry push / base-bump fan-out.

🤖 Generated with Claude Code

🕵️ claude-code · claude-opus-4-8 · run details

[codecov]

Codecov Report

❌ Patch coverage is 66.07143% with 133 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/router/worker-image-build.ts	71.42%	80 Missing ⚠️
src/db/repositories/projectsRepository.ts	0.00%	49 Missing ⚠️
src/router/worker-image-validation.ts	25.00%	3 Missing ⚠️
src/router/worker-dockerfile-compose.ts	97.29%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[nhopeatall]

Summary

APPROVE — a clean, fail-closed, well-tested build engine that faithfully extends the spec-022 validation pattern. I independently verified the two riskiest novel pieces (the hand-rolled tar and the dockerode labels path) and the fail-closed / no-strand / GC invariants all hold. All 98 tests across the touched files pass locally; CI is 7/7 green.

What I verified

• Hand-rolled ustar tar (createSingleFileTar) — header field offsets (name/mode/uid/gid/size/mtime/chksum/typeflag/magic/version), the space-filled checksum computed over all 512 bytes, 512-block body padding, and the 1024-byte zero trailer are all correct. dockerode's prepareBuildContext passes a raw Readable straight through (no re-tarring), and Readable.from(buffer) emits the whole Buffer as one chunk (Node special-cases Buffer/string — confirmed empirically), so the build context is not corrupted byte-by-byte.
• cascade.build_hash label via the buildImage({ labels }) option — the reuse fast-path depends on this label actually landing on the image. dockerode doesn't special-case labels, but docker-modem.buildQuerystring JSON-stringifies any non-array object option, so labels reaches POST /build as the JSON map the Engine API expects. cascade.managed=true is separately a composed LABEL line, so GC matching does not depend on the option path.
• GC claims — the superseded built image is untagged-but-labeled (cascade.managed=true from the composed LABEL), matching the existing dangling=true AND label=cascade.managed=true reaper filter; the active :latest tag is structurally never dangling. No new loop needed. Confirmed against dangling-image-cleanup.ts.
• pull: false correctness — the base is pulled by tag in defaultResolveBaseDigest first, so the composed FROM repo@sha256:… digest ref resolves locally without a build-time pull.
• No-strand / fail-closed — keepActive is derived from the pre-build snapshot; the verified/first-fail/keep-active branches in recordWorkerImageBuildResult match the four integration cases exactly; the handler never rejects and Sentry-captures unexpected throws. The early-throw default of keepActive=false is the correct fail-closed choice (guarantees a terminal failed rather than a stuck building).
• Distinct reasons (build failed: vs runtime requirement missing …), smoke-test against the immutable pin (not the retag-able tag), schema columns present from ticket 1/5, and the defaultRunImageCheck → runWorkerImageSmokeTest rename has no dangling references.

Optional (non-blocking, low severity)

One defense-in-depth nit on the pin step — see inline comment. Not a merge blocker.

🕵️ claude-code · claude-opus-4-8 · run details

[nhopeatall]

Optional hardening (low severity, non-blocking): the pin step takes built.id without re-asserting built.fullHash === opts.fullHash, whereas the reuse check on line 326 does compare it. In the narrow window where a timeout-abandoned zombie build (dockerode has no cancel hook, as the comment on withBuildTimeout notes) or a concurrent supersede retags cascade-built-<proj>:latest between buildImage() resolving and this inspect, the pin could point at a different image than the one just built, and it would still be recorded verified. Re-checking the label here — if (!built?.id || built.fullHash !== opts.fullHash) return { ok: false, reason: 'build failed: …' } — converts that silent wrong-pin into a clean fail-closed record, mirroring the reuse path. Very unlikely in practice (requires a >workerBuildTimeoutMs build racing a re-enqueue), so purely a defense-in-depth suggestion.