Skip to content

feat(addie): add dated evaluation pricing cohort - #7312

Merged
bokelley merged 4 commits into
mainfrom
conductor/eval-dated-pricing-cohort
Sep 6, 2026
Merged

feat(addie): add dated evaluation pricing cohort#7312
bokelley merged 4 commits into
mainfrom
conductor/eval-dated-pricing-cohort

Conversation

@bokelley

@bokelley bokelley commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a credential-free, immutable, dated prospective pricing cohort for the current Addie evaluation candidates. The resolver returns immutable profiles and a domain-separated SHA-256 digest only after candidate, interval, evidence, and returned-model checks pass. It is accounting input only and does not create providers or authorize dispatch.

This revision makes the router evaluator consume that same immutable profile for admission, reservation, settlement, per-cell actual cost, and evidence hashing. Cache accounting is profile-driven: subset cache reads are never charged twice, while additive cache reads/writes are reserved conservatively and settled explicitly. Missing, unavailable, ambiguous, malformed, overlapping, or drifted cache usage fails closed; pre-dispatch refusal leaves no reservation.

OpenAI Luna cache writes are now recorded as a subset replacement bucket at $0.25/MTok (the official model card's documented 1.25× $0.20 uncached-input rate). Shared reservation accounting selects the costliest valid subset category, so that higher cache-write rate is covered before dispatch. Router reservations also retain the module-issued profile identity, digest, candidate, profile ID, provider, and model; settlement rejects cross-profile or frozen-substitute pricing before it can account spend.

Router reservation handles are opaque frozen identity tokens. Their USD amount, active/dispatched lifecycle, and profile binding reside only in the owning ledger's private WeakMap, preventing a caller from altering a refund or clearing another live reservation. Clone, proxy, lookalike, cross-ledger, inactive, and reused handles fail before accounting.

The fixed-trace diagnostic fixtures now obtain their dated cohort evidence and rates from the canonical profile, rather than carrying stale standalone policy data.

Base and head

  • Base: main / origin/main at ebe3f6c87bc61b2cab5bccd97253f845e056ceba (#7313); verified as this branch's exact merge base.
  • Head: 5ea401d334829fecd3051e2d4d6ee8cda7dc044a

The #7305 dormant fixed-trace CLI, its local-tsx child contract, unknown-exposure evidence, and restored active coverage are retained unchanged. The fixed-trace budget continues to use the shared cache-aware dated pricing reservation helper.

The #7313 fixed-trace smoke-overlay source and tests remain untouched by this branch and pass alongside the #7305 and pricing/router coverage.

Coverage and validation

  • Router-ledger tests cover OpenAI subset cached input and 1.25× cache writes, Anthropic additive cache reads/writes, zero-cache usage, malformed/overlapping or unavailable cache categories, reservation ceilings, settlement/refund, cross-profile settlement rejection, frozen profile substitution, opaque concurrent handles, immutable-profile admission, and exact fixed-trace pricing parity.
  • Focused pricing/router, all fix(addie): make fixed-trace dispatch dormant #7305 fixed-trace coverage, and fix(addie): add evaluator smoke overlays #7313 smoke overlays — 12 files, 215 tests passed.
  • Prior required server-unit shard 2/4 passed on the immediately preceding fix(addie): make fixed-trace dispatch dormant #7305 base rebase: 144 files, 1,979 tests.
  • npm run typecheck passed.
  • git diff --check origin/main...HEAD passed.
  • Credential-free manual router inventory and the fix(addie): make fixed-trace dispatch dormant #7305 dormant fixed-trace CLI passed with provider credentials unset; no provider construction, dispatch, or spend occurred.

The repository's broad automatic pre-commit suite still encountered unrelated existing server database-fixture failures outside this change; the affected shard and focused changed-area suite above are green.

Review feedback disposition

Scope and authorization

No production provider activation, routing, replay execution behavior, rollout decision, quality claim, or spend authority is added. This remains a diagnostic-only pricing/admission change. No provider calls were made, no API key or secret is committed, and this PR is ready for merge with exact-head Sol and Ladon approval.

@bokelley bokelley changed the title feat(addie): add dated pricing cohort feat(addie): add dated evaluation pricing cohort Sep 6, 2026
@bokelley
bokelley force-pushed the conductor/eval-dated-pricing-cohort branch from bccf770 to 53ace87 Compare September 6, 2026 05:47
@bokelley
bokelley force-pushed the conductor/eval-dated-pricing-cohort branch from 53ace87 to 5ea401d Compare September 6, 2026 05:58
@bokelley
bokelley marked this pull request as ready for review September 6, 2026 06:08

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — Addie eval/pricing refactor, no blocking findings.

This PR consolidates Addie eval pricing into a dated-cohort module, rewires the fixed-trace budget, updates the router-eval harness, and adjusts the OpenAI billing-identity/cache-write mappings. All changes are within Addie/eval scope (server/src/addie/** and tests), off the protocol wire surface (static/schemas/source/**, docs/reference/**, dist/**), so no changeset is warranted and none is expected.

Checks performed by the reviewer and confirmed against the gate rules:

  • Accounting math verified (conservative subset/additive reservation, exact integer-micro path), immutability, fail-closed settlement, and error-contract preservation.
  • OpenAI(subset 0.25)/Google(unsupported) cache-write data changes match the actual provider adapter usage mappings.
  • No changeset required — change is off the published protocol surface.

Decision-table walk: no critical/high findings (row 1 n/a); gated_paths is false (row 2 n/a); no (deleted) high-risk reasons (row 3 n/a); no medium findings at all, so rows 4/5/8 n/a; no prior decision (row 6 n/a); no no-auto-approve team match (row 7 n/a). high_risk is true only via (added)/(modified) reasons with zero medium-or-higher findings — per the high_risk guidance this is not escalation-worthy on the flag alone. Falls through to row 9 → approve.

Non-blocking follow-up (informational): the Google fixed-trace pricing policy has a post-2026 wall-clock expiry worth revisiting before then.

@bokelley
bokelley merged commit 7d74363 into main Sep 6, 2026
30 checks passed
@bokelley
bokelley deleted the conductor/eval-dated-pricing-cohort branch September 6, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant