fix(routing): define hard cost-cap behavior when cost evidence is unknown

[abhisheksharma2411]

Fixes #1181.

Summary

limits.maxEstimatedCostUsd is documented as a hard per-request ceiling, but it never fires on the live routing path.

evaluatePolicyProfile only excludes a candidate when the estimate is a finite number (src/routing/evaluator.ts):

const overCostLimit = costLimit !== undefined
  && typeof estimatedCost === "number"   // an unknown estimate passes silently
  && Number.isFinite(estimatedCost)
  && estimatedCost > costLimit;

…while routeModel assembles cost evidence without usage (src/router.ts):

cost: costEvidenceForCandidate({
  provider: candidate.provider,
  model: candidate.model,
  limitUsd: profile.limits.maxEstimatedCostUsd,   // no `usage`
}),

costEvidenceForCandidate correctly returns { limitUsd, incomplete: true } with no estimatedUsd, so on the live path the estimate is always unknown and any candidate passes a cap the operator configured as hard. The existing coverage passes only because it supplies usage directly — exercising a path production does not take.

As #1181 notes, failing closed unconditionally is not safe either: with usage unwired it would reject every live candidate whenever a cap is set, and it would silently change the documented dry-run contract.

This adds an explicit, opt-in policy instead:

{
  "limits": {
    "maxEstimatedCostUsd": 0.25,
    "onUnknownCost": "exclude"   // "allow" (default) | "exclude"
  }
}

• "allow" (default) — preserves current behaviour and the documented contract exactly. No existing deployment changes on upgrade.
• "exclude" — the ceiling becomes genuinely hard: a candidate whose cost cannot be proven under the cap is ineligible.

Unknown-cost exclusions emit a distinct cost-limit-unknown code, so a trace distinguishes known above the cap from cost is unknown — the operator-facing distinction the issue asks for.

Kept deliberately separate from unknownEvidence.cost, which governs how an unknown-cost candidate is scored rather than whether the ceiling applies. A test asserts the two mechanisms stay distinguishable.

File	Change
src/types.ts	OcxRoutingUnknownCostCapMode; onUnknownCost on OcxRoutingProfileLimits
src/routing/profile.ts	Validation ("allow" / "exclude") + normalization
src/routing/evaluator.ts	Cap policy under unknown evidence; cost-limit-unknown exclusion
tests/cost-cap-unknown-evidence.test.ts	New — 5 cases

Verification

The defect was reproduced with a failing test before the fix: with a $0.000001 cap and live-path evidence (no usage), the candidate was still eligible and selected.

Commands run against the dev base:

bun test tests/cost-cap-unknown-evidence.test.ts     → 5 pass, 0 fail
bun test tests/cost-scoring.test.ts \
         tests/routing-profile.test.ts               → 27 pass, 0 fail
bun x tsc --noEmit                                   → clean
bun scripts/test.ts                                  → 10134 pass, 7 skip, 0 fail

New test cases:

1. Repro — live-path evidence carries no estimate, so the hard cap never fires (documents current behaviour)
2. Fail-closed — onUnknownCost: "exclude" excludes the candidate and emits cost-limit-unknown
3. Default unchanged — with no onUnknownCost, behaviour is identical to today
4. Mechanism distinctness — unknownEvidence.cost: "exclude" emits unknown-price, not cost-limit-unknown
5. Inert without a cap — onUnknownCost has no effect when maxEstimatedCostUsd is unset

Coverage on touched files: src/routing/profile.ts 89.8% lines; every added line in src/routing/evaluator.ts covered. This change is server-side routing logic only; no front-end surface is touched, so no screenshot applies.

Checklist

• Scope stays focused and avoids unrelated cleanup.
• Docs or release notes were updated when needed. — The new field is documented via JSDoc on OcxRoutingProfileLimits and its validation message. Happy to add a docs entry if you point me at the right file.
• Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. — No secrets or auth touched. The default is deliberately "allow" so an upgrade cannot silently make every live candidate ineligible; the stricter behaviour is opt-in.

Notes for review

• Default is unchanged, so this is additive and non-breaking.
• Wiring real usage into the live routing path is deliberately not attempted here — that is the larger change Routing: define hard cost-cap behavior when runtime cost evidence is unknown #1181 warns about, and it is separable from defining the policy.
• Happy to rename onUnknownCost, or fold it into unknownEvidence, if you would prefer a single mechanism. I kept them separate because they answer different questions.

---

Supersedes #1344, which was opened against main by mistake and closed. This one targets dev, is rebased onto the current dev head (025c3791), and addresses the CodeRabbit finding from that PR — the test docblock and two test names no longer describe the exclusion case as "fails until fixed", since the evaluator change ships in this PR.

Review readiness checklist

This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:

• All CI tests are green on my local testing.

• I pushed my PR to the latest dev commit.

• I resolved all correct Codex and CodeRabbit findings.

• My PR is ready for review.

Summary by CodeRabbit

• New Features

  • Added configurable handling for candidates with unknown cost estimates when a maximum cost is configured.
  • Unknown costs are allowed by default, with an option to exclude them.
  • Added distinct cost-limit reporting for candidates excluded due to unknown costs.

• Bug Fixes

  • Known costs continue to be excluded when they exceed the configured maximum.
  • Existing behavior remains unchanged when no cost cap is configured.

• Tests

  • Added coverage for default behavior, opt-in exclusion, and related cost-evidence scenarios.

[github-actions]

✅ Deterministic PR hygiene checks passed.

[coderabbitai]

Important

Review skipped

No new commits to review since the last review.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 98f4a248-453b-498f-92c6-7f424abec7a7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The routing profile API now supports an explicit policy for unknown cost estimates. The evaluator preserves finite cost-cap exclusions and optionally excludes unknown costs with cost-limit-unknown. Regression tests cover default and fail-closed behavior.

Changes

Unknown cost cap handling

Layer / File(s)	Summary
Profile policy contract and normalization src/types.ts, src/routing/profile.ts	Adds OcxRoutingUnknownCostCapMode, validates onUnknownCost as "allow" or "exclude", and preserves the setting during normalization.
Candidate cost-cap evaluation src/routing/evaluator.ts	Keeps cost-limit for known costs above the cap and uses cost-limit-unknown for unknown costs when exclusion is configured.
Unknown-cost behavior coverage tests/cost-cap-unknown-evidence.test.ts	Tests default allow behavior, opt-in exclusion, separate exclusion codes, scoring exclusions, cap-independent behavior, and omitted-policy defaults.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• lidge-jun/opencodex#1344: Directly related unknown-cost cap handling across the same routing files and tests.
• lidge-jun/opencodex#1015: Related cost-limit and unknown-cost handling in routing evaluation and profile normalization.
• lidge-jun/opencodex#1012: Related candidate eligibility policy logic in src/routing/evaluator.ts.

Suggested reviewers: wibias, ingwannu, lidge-jun

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes implement the requested allow-by-default and opt-in exclude policies, distinct exclusion code, validation, typing, and regression coverage for issue [#1181].
Out of Scope Changes check	✅ Passed	The changes are limited to cost-cap evaluation, profile configuration, public types, and tests required by issue [#1181].
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: defining behavior for routing cost caps when cost evidence is unknown.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

✅ READY

• all PR quality gates passed; the review readiness checklist is complete.

Review readiness checklist

• ✅ All CI tests are green on my local testing.
• ✅ I pushed my PR to the latest dev commit.
• ✅ I resolved all correct Codex and CodeRabbit findings.
• ✅ My PR is ready for review.

✅ 4/4 boxes ticked.

This pull request has been marked Ready for Review.
The review-ready label marks this PR as ready; review automation runs independently. If no CodeRabbit review appears, comment @coderabbitai review to request one.
Maintainers notified: @lidge-jun @Ingwannu @Wibias

[coderabbitai]

@github-actions[bot] I am replying to the latest comment only. The earlier comment chain is not available in this context.

I will run an independent review of PR #1345.

⚠️ Action not completed

Already reviewed.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[Wibias]

Requesting changes on the current head (a8f004f014a8b3084d9037391a58c2bcde160bd3).

The fail-closed path is implemented cleanly, but #1181's operator-facing contract is still incomplete.

1. High — default allow still leaves the cap outcome ambiguous.

#1181 explicitly requires operators to distinguish a candidate that is known below the cap from one whose cost is unknown and therefore cannot be evaluated against the cap. The PR currently emits cost-limit-unknown only when onUnknownCost: "exclude" blocks the candidate. With the default/omitted "allow", the candidate remains eligible but the trace gains no stable indication that the cap was not actually proven satisfied.

Please preserve fail-open eligibility for the default, but emit a stable non-excluding policy outcome for the unknown-cost case (for example a structured cap status or a distinct cost-limit-unknown-allowed-style reason) and carry that through the operator-visible dry-run/live trace/log surfaces. Do not represent it as an exclusion if the candidate stays eligible.

2. Medium — operator docs and integration coverage are incomplete.

There is already a public Routing Profile Editor/config guide that documents limits.maxEstimatedCostUsd. onUnknownCost, its default, and the allow-vs-fail-closed semantics should be documented there; JSDoc/validation text alone is not sufficient for a new operator-facing policy field.

The new tests currently call costEvidenceForCandidate() and evaluatePolicyProfile() directly, which validates evaluator behavior but only simulates the live-path evidence shape. Please add acceptance coverage that proves the policy outcome is consistent through the actual dry-run/live routing trace/log path, including:

• unknown cost + default/allow => eligible with an explicit unknown-cap outcome;
• unknown cost + exclude => ineligible with cost-limit-unknown;
• the two remain distinguishable from unknownEvidence.cost handling.

3. Merge gate — rebase onto current dev and rerun CI.

This head was based on 025c3791; current dev has advanced substantially since then. Rebase/update the branch after the contract fixes and rerun exact-head CI before merge.

The evaluator's known-over-cap behavior, the opt-in fail-closed logic, and the separation from unknownEvidence.cost otherwise look coherent.

[abhisheksharma2411]

Superseded by #1404, which completes the operator contract from my review feedback (non-excluding capOutcome on the allow path, operator docs, and dry-run/live trace coverage) and is rebased onto current dev.

@Wibias — confirming the open item on your test plan: happy for this to be closed once #1404 lands, no cherry-pick needed. I've left a review there.

Thanks for the detailed review; the fail-open ambiguity under the default was a genuine gap in what I had.

[Wibias]

Superseded by #1404.

Thanks @abhisheksharma2411 — your PR was the right foundation for #1181. It correctly diagnosed that live routing builds cost evidence without usage, added the opt-in onUnknownCost: "exclude" fail-closed path with a distinct cost-limit-unknown code, and kept the default allow behavior so upgrades stay non-breaking. That separation from unknownEvidence.cost was the right design call.

#1404 keeps that core, rebases onto current dev, and finishes the remaining operator contract from review: non-excluding cost.capOutcome on the allow path, docs/editor wiring, and dry-run/live integration coverage.

Closing this draft in favor of #1404 so the work can land on tip.