Runtime validation made whole — checks see the service, forms author it, the ship ends e2e - #42
Merged
Merged
Conversation
…fier
A needs_service run threaded ALC_BASE_URL/PORT to the engine and to
capture:, but Verifier.run spawned check subprocesses with the bare
inherited environ — so the builder pack's own e2e-smoke check
(curl -sf "$ALC_BASE_URL") could never pass: reproduced live with a
real service (health green, check failed 4/4 attempts, three repair
turns burned on an unfixable check, run FAILED). The Verifier now takes
the run's env at construction and merges it over os.environ — the same
contract as the engine turn — so checks verify against the same live
service the engine talked to. The same reproduction now passes on the
first attempt. None/{} keeps checks byte-identical to before; flow.py's
verify_only stages pass nothing and stay unchanged (finding 50).
Also hardens test_no_stack_detected_still_audits_the_security_set: it
embedded the assumption that gitleaks is NOT installed — on a machine
with the scanner the scaffold declares it live and the audit correctly
has nothing to propose, so the test only ever passed on machines
without the tool. It now drops the security set from the manifest first,
pinning the scenario it always meant: a project without the set.
The e2e knobs were YAML-only (finding 51): needs_service had no field anywhere, capture sat unexplained at the bottom of the Blueprint form, and the Manifest form had no service section at all — the only UI path to runtime validation was the raw source editor. BlueprintForm gains a Runtime validation section: a needs_service checkbox (on writes the boolean, off deletes the key — absence IS the model default) with capture beside it, now carrying a hint and a real placeholder. ManifestForm gains a Service section: Start authors service.start, and Health path / Ready timeout reveal only once a start command exists — a service block without start would not validate. Clearing Start deletes the whole block; clearing health falls back to the model default instead of writing an empty string. Validated on desktop CDP against the real qa blueprint (checkbox reads its needs_service: true, capture shows its live command) and at a 411px emulated viewport with no horizontal overflow. The physical Android is currently disconnected — on-device validation pending its return.
The builder pack gains ship-e2e: plan, build, harden, static gate, then a final qa stage — a real engine stage, not verify_only, because the service lifecycle lives in the runner and a verify-only stage would bypass it. ALC starts the Manifest's service, hands the engine $ALC_BASE_URL, and the qa Blueprint's e2e-smoke check verifies against the running app before the flow may succeed. Additive: ship-hardened is untouched, so projects without a service: keep their flow; ship-e2e's header comment says the service requirement out loud. Proved live in a scratch project (mock engine, python http.server as the service): all five stages SUCCESS, the gate with zero engine turns, and e2e-smoke green against the app ALC started and tore down. The test pinning builder's hire count moves 2 -> 3 with its reasoning: the pack deliberately grew by one flow.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Dogfood round 15, born from a scouting question — does e2e happen post-implementation in alc? — whose investigation found the runtime-validation subsystem complete on paper and broken at its heart. Three items, each closed by its own regression-guard pass and committed separately, as requested.
1. Checks see the live service (finding 50) —
53c8217A
needs_servicerun threaded$ALC_BASE_URLto the engine (tested) and tocapture:(documented) — butVerifier.runspawned check subprocesses with the bare inherited environ. The builder pack's flagshipqaBlueprint could not pass its own inlinee2e-smokecheck: reproduced live with a real service — health poll green,curl -sf "$ALC_BASE_URL"failing 4/4 attempts with an empty URL, three repair turns burned on an unfixable check, run FAILED.The Verifier now takes the run's env at construction and merges it over
os.environ— the same contract as the engine turn. The identical reproduction passes on the first attempt.None/{}keeps every existing caller byte-identical;flow.py's verify_only stages pass nothing and stay unchanged. Four unit tests (env reaches command checks, merge preserves PATH, no-env inherits untouched, metric path same contract) plus two integration tests throughexecute_mandate(service run's check sees the URL; plain run's env stays clean).The battery caught a second, unrelated defect:
test_no_stack_detected_still_audits_the_security_setembedded the assumption that gitleaks is NOT installed — on a machine with the scanner, the scaffold declares it live and the audit correctly has nothing to propose, so the test only ever passed on machines without the tool. It is hermetic now: it drops the security set from the manifest first, pinning the scenario it always meant.2. The forms author runtime validation (finding 51) —
0d90e88The e2e knobs were YAML-only. BlueprintForm gains a Runtime validation section: a
needs_servicecheckbox — on writes the boolean, off deletes the key, because absence is the model default — withcapturebeside it, now carrying a hint and an honest placeholder. ManifestForm gains a Service section: Start authorsservice.start; Health path and Ready timeout reveal only once a start command exists (a service block withoutstartwould not validate); clearing Start deletes the whole block, clearing health falls back to the model default instead of writing an empty string. Five new vitest cases.Validated live on desktop CDP against the real
qablueprint (the checkbox reads itsneeds_service: true, capture shows its shipped command) and at a 411px emulated viewport with no horizontal overflow. The physical Android was disconnected this round — on-device validation is owed when it returns, and the friction log says so.3. The ship ends end-to-end —
de3ea0dThe builder pack gains
ship-e2e: plan → build → harden → static gate (verify_only) → a final qa stage. The last stage is a real engine stage, deliberately notverify_only— the service lifecycle lives in the runner, and a verify-only stage would bypass it. Additive:ship-hardenedis untouched, so projects without aservice:keep their flow, andship-e2e's header comment states the service requirement out loud.Proved live in a scratch project (mock engine,
python3 -m http.serveras the service): all five stages SUCCESS, the gate with zero engine turns,e2e-smoke ✓against the app ALC started and tore down — post-implementation e2e as the flow's last word, exactly the round's brief.Regression guard, per item
Each item ran the full battery before its commit:
ruffclean · pytest 2665 · vitest 671 · build clean (one environment-hardened test, reasoned above)ruffclean · pytest 2665 · vitest 676 · build clean (no findings)ruffclean · pytest 2667 · vitest 676 · build clean (one intended-change pin: builder hire count 2 → 3, updated with its reasoning)🤖 Generated with Claude Code