DeltaWire: measured performance envelope + first real-model admission run - #138
Merged
Conversation
Executes the Iteration plan (docs/13-performance-and-admission-plan.md): turn DeltaWire's estimate-only selection into a measured envelope, then run the first admissible real-model B0/D1 comparison. Iteration 1 — performance & conformance closure - Add hidden `deltawire bench` subcommand (internal/cli/bench*.go): runs the real pipeline per family x scale, emitting compiled/output bytes + phase timings; determinism/runtime-boundary preserved (zero model calls). - Add optimizer.PerfProfile + AnalyzeWithProfile (internal/optimizer/profile.go), threaded through candidates/select with safe fallback (profile_missing / _stale / _missing_family reason codes); Analyze == AnalyzeWithProfile(nil), so no caller churn. Collected profile in eval/results/perf-profile-v1/. - Close GAPS.md item 10 (Pareto report now carries measured bytes/timing). Iteration 2 — real-model admission pack + first call - Author entitlement-matrix task under eval/admission/ (NOT eval/tasks/, which is a closed generic-oracle system) with a frozen independent oracle; deterministic admission gates verified by internal/optimizer/entitlement_test.go. - Add eval/scripts/admission_runner.py: fail-closed, content-addressed two-arm runner scoped to the admission task, reusing the Harbor invocation + fidelity receipts. Run manifest: eval/manifests/entitlement-matrix-run-manifest.json. - First B0/D1 pair (gemini-3.1-pro-preview, Harbor 0.20.0), initial construction: both arms reach the exact oracle-verified artifact (byte-identical), but the treatment agent did NOT adopt DeltaWire when offered optionally (classification not_used) — it wrote plain Python like the control, for ~29k more input tokens. Full write-up + decision-rule mapping (interface/adoption branch) in eval/results/admission/entitlement-matrix/README.md. Raw Harbor evidence retained. The 2.3 modification arm is not yet run. Non-goals held: no universal performance claim, no supervisor-contract change, no model call until every deterministic admission gate was green. .gitignore: exclude eval/admission/*/environment/.generated/ and eval/.secrets/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Executes the approved Iteration plan (
labs/20-deltawire/docs/13-performance-and-admission-plan.md): move DeltaWire from estimate-only selection to a measured performance envelope (Iteration 1), then run the first admissible real-model B0/D1 comparison (Iteration 2). All prior conformance guarantees (determinism, runtime-boundary, zero-model core) are preserved.Iteration 1 — performance & conformance closure
deltawire bench(hidden subcommand,internal/cli/bench*.go): runs the real pipeline per family × scale, emitting compiled/output bytes + phase timings. Determinism and runtime-boundary preserved. Profile collected ineval/results/perf-profile-v1/.optimizer.PerfProfile+AnalyzeWithProfile(internal/optimizer/profile.go): optional measured coefficients threaded through candidates/select, with safe fallback (profile_missing/_stale/_missing_familyreason codes).Analyze == AnalyzeWithProfile(…, nil), so no caller churn.GAPS.mditem 10 (Pareto report carries measured bytes/timing).Iteration 2 — real-model admission pack + first call
entitlement-matrix task authored under
eval/admission/(deliberately noteval/tasks/, which is a closed generic-oracle system) with a frozen independent oracle. Deterministic admission gates verified byinternal/optimizer/entitlement_test.go.eval/scripts/admission_runner.py: fail-closed, content-addressed two-arm runner scoped to the admission task, reusing the Harbor invocation + fidelity receipts.--dry-runvalidates every hash with no model call.First B0/D1 pair (
gemini-3.1-pro-preview, Harbor 0.20.0), initial construction:--skill)* Not a task failure — D1's artifact is correct and byte-identical to B0's. The treatment agent never adopted the framework (
classification=not_used,plan_path=null); it wrote plain Python like the control, for ~29k more input tokens. This lands on the interface/adoption branch of the plan's decision rule. Full write-up + follow-ups ineval/results/admission/entitlement-matrix/README.md. Raw Harbor evidence retained.Key finding
At this scale (54 records, one-shot-correct in plain Python) DeltaWire is neither chosen nor needed when offered optionally. Honest next steps: (1) raise scenario stakes to where the framework's determinism/trust-surface actually bite; (2) make the treatment arm require the framework to measure interface usability at all. This motivates a deterministic trigger mechanism (cf. the Pitot integration) rather than relying on model adoption. The 2.3 modification arm is not yet run.
Non-goals held
No universal performance claim; no supervisor-contract change; no model call until every deterministic admission gate was green.
Verification
go build ./...+go test ./...green.scripts/validate.shEXIT=0 (determinism, runtime-boundary, production-cleanliness;model_calls: 0on the deterministic path).eval/.secrets/, scanned before commit); staged linux binary gitignored undereval/admission/*/environment/.generated/.