Skip to content

DeltaWire: measured performance envelope + first real-model admission run - #138

Merged
bigboateng merged 1 commit into
mainfrom
worktree-deltawire-iteration-plan
Jul 23, 2026
Merged

DeltaWire: measured performance envelope + first real-model admission run#138
bigboateng merged 1 commit into
mainfrom
worktree-deltawire-iteration-plan

Conversation

@bigboateng

Copy link
Copy Markdown
Contributor

Executes the approved Iteration plan (labs/20-deltawire/docs/13-performance-and-admission-plan.md): move DeltaWire from estimate-only selection to a measured performance envelope (Iteration 1), then run the first admissible real-model B0/D1 comparison (Iteration 2). All prior conformance guarantees (determinism, runtime-boundary, zero-model core) are preserved.

Iteration 1 — performance & conformance closure

  • deltawire bench (hidden subcommand, internal/cli/bench*.go): runs the real pipeline per family × scale, emitting compiled/output bytes + phase timings. Determinism and runtime-boundary preserved. Profile collected in eval/results/perf-profile-v1/.
  • optimizer.PerfProfile + AnalyzeWithProfile (internal/optimizer/profile.go): optional measured coefficients threaded through candidates/select, with safe fallback (profile_missing / _stale / _missing_family reason codes). Analyze == AnalyzeWithProfile(…, nil), so no caller churn.
  • Closes GAPS.md item 10 (Pareto report carries measured bytes/timing).

Iteration 2 — real-model admission pack + first call

  • entitlement-matrix task authored under eval/admission/ (deliberately not eval/tasks/, which is a closed generic-oracle system) with a frozen independent oracle. Deterministic admission gates verified by internal/optimizer/entitlement_test.go.

  • eval/scripts/admission_runner.py: fail-closed, content-addressed two-arm runner scoped to the admission task, reusing the Harbor invocation + fidelity receipts. --dry-run validates every hash with no model call.

  • First B0/D1 pair (gemini-3.1-pro-preview, Harbor 0.20.0), initial construction:

    arm DeltaWire offered used oracle exact end-to-end input tok wall
    B0 (control) no n/a 1 pass 39,949 55s
    D1 (treatment) yes (--skill) no 1 fail* 68,585 81s

    * Not a task failure — D1's artifact is correct and byte-identical to B0's. The treatment agent never adopted the framework (classification=not_used, plan_path=null); it wrote plain Python like the control, for ~29k more input tokens. This lands on the interface/adoption branch of the plan's decision rule. Full write-up + follow-ups in eval/results/admission/entitlement-matrix/README.md. Raw Harbor evidence retained.

Key finding

At this scale (54 records, one-shot-correct in plain Python) DeltaWire is neither chosen nor needed when offered optionally. Honest next steps: (1) raise scenario stakes to where the framework's determinism/trust-surface actually bite; (2) make the treatment arm require the framework to measure interface usability at all. This motivates a deterministic trigger mechanism (cf. the Pitot integration) rather than relying on model adoption. The 2.3 modification arm is not yet run.

Non-goals held

No universal performance claim; no supervisor-contract change; no model call until every deterministic admission gate was green.

Verification

  • go build ./... + go test ./... green.
  • scripts/validate.sh EXIT=0 (determinism, runtime-boundary, production-cleanliness; model_calls: 0 on the deterministic path).
  • Live API key kept out of the repo (gitignored eval/.secrets/, scanned before commit); staged linux binary gitignored under eval/admission/*/environment/.generated/.

Executes the Iteration plan (docs/13-performance-and-admission-plan.md):
turn DeltaWire's estimate-only selection into a measured envelope, then
run the first admissible real-model B0/D1 comparison.

Iteration 1 — performance & conformance closure
- Add hidden `deltawire bench` subcommand (internal/cli/bench*.go): runs the
  real pipeline per family x scale, emitting compiled/output bytes + phase
  timings; determinism/runtime-boundary preserved (zero model calls).
- Add optimizer.PerfProfile + AnalyzeWithProfile (internal/optimizer/profile.go),
  threaded through candidates/select with safe fallback (profile_missing /
  _stale / _missing_family reason codes); Analyze == AnalyzeWithProfile(nil),
  so no caller churn. Collected profile in eval/results/perf-profile-v1/.
- Close GAPS.md item 10 (Pareto report now carries measured bytes/timing).

Iteration 2 — real-model admission pack + first call
- Author entitlement-matrix task under eval/admission/ (NOT eval/tasks/, which
  is a closed generic-oracle system) with a frozen independent oracle;
  deterministic admission gates verified by internal/optimizer/entitlement_test.go.
- Add eval/scripts/admission_runner.py: fail-closed, content-addressed two-arm
  runner scoped to the admission task, reusing the Harbor invocation +
  fidelity receipts. Run manifest: eval/manifests/entitlement-matrix-run-manifest.json.
- First B0/D1 pair (gemini-3.1-pro-preview, Harbor 0.20.0), initial construction:
  both arms reach the exact oracle-verified artifact (byte-identical), but the
  treatment agent did NOT adopt DeltaWire when offered optionally (classification
  not_used) — it wrote plain Python like the control, for ~29k more input tokens.
  Full write-up + decision-rule mapping (interface/adoption branch) in
  eval/results/admission/entitlement-matrix/README.md. Raw Harbor evidence
  retained. The 2.3 modification arm is not yet run.

Non-goals held: no universal performance claim, no supervisor-contract change,
no model call until every deterministic admission gate was green.

.gitignore: exclude eval/admission/*/environment/.generated/ and eval/.secrets/.
@bigboateng
bigboateng merged commit a08fba1 into main Jul 23, 2026
6 checks passed
@bigboateng
bigboateng deleted the worktree-deltawire-iteration-plan branch July 23, 2026 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant