DeltaWire evaluation evidence and direct-specification research plan

[bigboateng]

Summary

Builds the DeltaWire V8 evaluation boundary, preserves its terminal evidence, adds an isolated six-trial range-large comparison, and documents the next direct/optimal specification research phases.

ZCA separates the product question into two steps:

1. Prove current DeltaWire accuracy and compare its JSON-plan agent workflow with ordinary agent-authored generation code.
2. Research a direct declarative specification, then optimize representation/backend selection only after the direct interface succeeds.

READY_FOR_72 remains false. This PR makes no benchmark-wide or statistical claim.

Six-trial isolated comparison

Three counterbalanced range-large D1/B0 pairs completed. B0 used the same Gemini runtime but had no DeltaWire binary, skill, .deltawire state, DeltaWire contract, or DeltaWire trajectory activity.

Pair	D1 input + output	B0 input + output	Delta	D1/B0	D1 treatment
1	86,537	59,763	+26,774	1.448x	failed: skeleton unchanged
2	73,041	47,344	+25,697	1.543x	passed
3	69,323	64,590	+4,733	1.073x	passed

Accuracy

• All six final artifacts were exact: 6/6.
• Both genuine DeltaWire finalizations were exact: 2/2.
• Each artifact contained exactly the ordered records {"index":1} through {"index":500} with no extra fields.
• Accuracy is established only for this simple range. Earlier V8 evidence showed that a structurally valid matrix-large plan could still fail semantic verification.

Token result

For the two semantically exact, fully finalized DeltaWire pairs:

• median D1 overhead: 15,215 input-plus-output tokens;
• median D1/B0 ratio: 1.308x, approximately 30.8% overhead;
• current JSON-plan treatment did not reduce model tokens in either valid pair.

The valid DeltaWire plan was 673 bytes and rendered 6,892 bytes, a 10.24x expansion. Deterministic expansion is compact; the measured overhead comes from agent acquisition and interaction: activation, reference discovery, repeated reads, JSON editing, and finalization.

B0 compressed the same task into executable code: two Bash loops and one Node.js loop. Current DeltaWire does not accept Python or other scripts as plan input. Python appears only in evaluation infrastructure and verification.

The B0 verifier fixture omitted Harbor's reward file, so Harbor recorded a post-execution RewardFileNotFoundError. Independent semantic verification passed all three outputs, and token/trajectory evidence is intact.

Evidence is retained under labs/20-deltawire/eval/results/deltawire-v9-six-trial-evidence/; the machine-readable result is comparison.json.

Earlier V8 campaign result

The broader V8 campaign stopped after seven meaningful trials because B0 independently invoked the dormant DeltaWire binary in two task families. That evidence remains preserved under labs/20-deltawire/eval/results/deltawire-v8-terminal-evidence/.

It established that environment delivery and skill assignment worked, but produced zero valid per-protocol pairs. The systemic-contamination stop avoided 17 additional causally unusable calls.

Next research phase

The committed plan is labs/20-deltawire/eval/docs/11-direct-optimal-spec-research-plan.md.

Stage 1: direct specification

The agent describes the desired dataset relationship—domains, relations, derived fields, invariants, ordering, and assertions—without authoring Bash, Python, JavaScript, or backend-specific generator code. A system-owned translator compiles only to the existing DeltaWire plan and reuses the existing validator, renderer, checker, and semantic oracle.

Compare three arms on range, matrix, piecewise-derived, variants, and abstention cases:

• unrestricted agent-authored code;
• current DeltaWire JSON plan;
• direct declarative specification.

Stage 1 succeeds only if exactness is retained and the direct interface reduces tool calls and median tokens versus current JSON-plan authoring.

Stage 2: optimal specification

Only after Stage 1 succeeds, evaluate admissible internal realizations and choose the lowest-cost one subject to exact semantics, deterministic output, bounded expansion, and no agent-authored executable code. A future system-generated Python backend is allowed as compiler output, not agent input.

Validation and security

• All V8 validation and manifest checks pass locally.
• Evidence-head workflows contain no live provider calls.
• Six-trial exact credential scan: 0 hits.
• Generic secret scan: 0 hits.
• Temporary credential files remaining: 0.
• Raw results, structured trajectories, outputs, plans, receipts, and reports are committed.

Readiness

{
  "READY_FOR_72": false,
  "six_trial_comparison_complete": true,
  "all_six_semantic_pass": true,
  "valid_deltawire_pairs": 2,
  "current_json_plan_token_reduction_observed": false
}

[bigboateng]

Six-trial isolated baseline comparison

Completed three counterbalanced range-large D1/B0 pairs. All six outputs independently matched records 1–500 exactly. B0 had no DeltaWire binary, skill, state, contract, or trajectory activity.

Pair	D1 I+O	B0 I+O	Delta	D1/B0	D1 treatment
1	86,537	59,763	+26,774	1.448x	failed: skeleton unchanged
2	73,041	47,344	+25,697	1.543x	passed
3	69,323	64,590	+4,733	1.073x	passed

Among the two semantically correct, fully finalized DeltaWire pairs, median D1 overhead was 15,215 input-plus-output tokens, or 1.308x B0 (+30.8%). Current JSON-plan treatment therefore did not reduce reported tokens in this small range-only evaluation.

The valid DeltaWire plan was 673 bytes and rendered 6,892 output bytes (10.24x byte expansion), so the generator representation itself is compact; the measured overhead is in agent interaction/context rather than expanded output generation.

Raw results and summary: labs/20-deltawire/eval/results/deltawire-v9-six-trial-evidence/ at commit cbf02550.

Caveat: the B0 verifier script omitted Harbor's reward file, so Harbor marked those trials with RewardFileNotFoundError; independent output verification passed all three and the token/trajectory records are intact.