Skip to content

Runtime validation made whole — checks see the service, forms author it, the ship ends e2e - #42

Merged
gifflet merged 3 commits into
mainfrom
feat/runtime-validation-whole
Sep 12, 2026
Merged

Runtime validation made whole — checks see the service, forms author it, the ship ends e2e#42
gifflet merged 3 commits into
mainfrom
feat/runtime-validation-whole

Conversation

@gifflet

@gifflet gifflet commented Sep 12, 2026

Copy link
Copy Markdown
Owner

What this is

Dogfood round 15, born from a scouting question — does e2e happen post-implementation in alc? — whose investigation found the runtime-validation subsystem complete on paper and broken at its heart. Three items, each closed by its own regression-guard pass and committed separately, as requested.

1. Checks see the live service (finding 50) — 53c8217

A needs_service run threaded $ALC_BASE_URL to the engine (tested) and to capture: (documented) — but Verifier.run spawned check subprocesses with the bare inherited environ. The builder pack's flagship qa Blueprint could not pass its own inline e2e-smoke check: reproduced live with a real service — health poll green, curl -sf "$ALC_BASE_URL" failing 4/4 attempts with an empty URL, three repair turns burned on an unfixable check, run FAILED.

The Verifier now takes the run's env at construction and merges it over os.environ — the same contract as the engine turn. The identical reproduction passes on the first attempt. None/{} keeps every existing caller byte-identical; flow.py's verify_only stages pass nothing and stay unchanged. Four unit tests (env reaches command checks, merge preserves PATH, no-env inherits untouched, metric path same contract) plus two integration tests through execute_mandate (service run's check sees the URL; plain run's env stays clean).

The battery caught a second, unrelated defect: test_no_stack_detected_still_audits_the_security_set embedded the assumption that gitleaks is NOT installed — on a machine with the scanner, the scaffold declares it live and the audit correctly has nothing to propose, so the test only ever passed on machines without the tool. It is hermetic now: it drops the security set from the manifest first, pinning the scenario it always meant.

2. The forms author runtime validation (finding 51) — 0d90e88

The e2e knobs were YAML-only. BlueprintForm gains a Runtime validation section: a needs_service checkbox — on writes the boolean, off deletes the key, because absence is the model default — with capture beside it, now carrying a hint and an honest placeholder. ManifestForm gains a Service section: Start authors service.start; Health path and Ready timeout reveal only once a start command exists (a service block without start would not validate); clearing Start deletes the whole block, clearing health falls back to the model default instead of writing an empty string. Five new vitest cases.

Validated live on desktop CDP against the real qa blueprint (the checkbox reads its needs_service: true, capture shows its shipped command) and at a 411px emulated viewport with no horizontal overflow. The physical Android was disconnected this round — on-device validation is owed when it returns, and the friction log says so.

3. The ship ends end-to-end — de3ea0d

The builder pack gains ship-e2e: plan → build → harden → static gate (verify_only) → a final qa stage. The last stage is a real engine stage, deliberately not verify_only — the service lifecycle lives in the runner, and a verify-only stage would bypass it. Additive: ship-hardened is untouched, so projects without a service: keep their flow, and ship-e2e's header comment states the service requirement out loud.

Proved live in a scratch project (mock engine, python3 -m http.server as the service): all five stages SUCCESS, the gate with zero engine turns, e2e-smoke ✓ against the app ALC started and tore down — post-implementation e2e as the flow's last word, exactly the round's brief.

Regression guard, per item

Each item ran the full battery before its commit:

  • Item 1: ruff clean · pytest 2665 · vitest 671 · build clean (one environment-hardened test, reasoned above)
  • Item 2: ruff clean · pytest 2665 · vitest 676 · build clean (no findings)
  • Item 3: ruff clean · pytest 2667 · vitest 676 · build clean (one intended-change pin: builder hire count 2 → 3, updated with its reasoning)

🤖 Generated with Claude Code

…fier

A needs_service run threaded ALC_BASE_URL/PORT to the engine and to
capture:, but Verifier.run spawned check subprocesses with the bare
inherited environ — so the builder pack's own e2e-smoke check
(curl -sf "$ALC_BASE_URL") could never pass: reproduced live with a
real service (health green, check failed 4/4 attempts, three repair
turns burned on an unfixable check, run FAILED). The Verifier now takes
the run's env at construction and merges it over os.environ — the same
contract as the engine turn — so checks verify against the same live
service the engine talked to. The same reproduction now passes on the
first attempt. None/{} keeps checks byte-identical to before; flow.py's
verify_only stages pass nothing and stay unchanged (finding 50).

Also hardens test_no_stack_detected_still_audits_the_security_set: it
embedded the assumption that gitleaks is NOT installed — on a machine
with the scanner the scaffold declares it live and the audit correctly
has nothing to propose, so the test only ever passed on machines
without the tool. It now drops the security set from the manifest first,
pinning the scenario it always meant: a project without the set.
The e2e knobs were YAML-only (finding 51): needs_service had no field
anywhere, capture sat unexplained at the bottom of the Blueprint form,
and the Manifest form had no service section at all — the only UI path
to runtime validation was the raw source editor.

BlueprintForm gains a Runtime validation section: a needs_service
checkbox (on writes the boolean, off deletes the key — absence IS the
model default) with capture beside it, now carrying a hint and a real
placeholder. ManifestForm gains a Service section: Start authors
service.start, and Health path / Ready timeout reveal only once a start
command exists — a service block without start would not validate.
Clearing Start deletes the whole block; clearing health falls back to
the model default instead of writing an empty string.

Validated on desktop CDP against the real qa blueprint (checkbox reads
its needs_service: true, capture shows its live command) and at a 411px
emulated viewport with no horizontal overflow. The physical Android is
currently disconnected — on-device validation pending its return.
The builder pack gains ship-e2e: plan, build, harden, static gate, then
a final qa stage — a real engine stage, not verify_only, because the
service lifecycle lives in the runner and a verify-only stage would
bypass it. ALC starts the Manifest's service, hands the engine
$ALC_BASE_URL, and the qa Blueprint's e2e-smoke check verifies against
the running app before the flow may succeed. Additive: ship-hardened is
untouched, so projects without a service: keep their flow; ship-e2e's
header comment says the service requirement out loud.

Proved live in a scratch project (mock engine, python http.server as
the service): all five stages SUCCESS, the gate with zero engine turns,
and e2e-smoke green against the app ALC started and tore down.

The test pinning builder's hire count moves 2 -> 3 with its reasoning:
the pack deliberately grew by one flow.
@vercel

vercel Bot commented Sep 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
alc Ready Ready Preview Sep 12, 2026 4:50pm UTC

@gifflet
gifflet merged commit 4ab4ae7 into main Sep 12, 2026
5 checks passed
@gifflet
gifflet deleted the feat/runtime-validation-whole branch September 12, 2026 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant