Skip to content

v0.7.141

Choose a tag to compare

@kfallah kfallah released this 01 Oct 20:05
42e516c

Catalog-backed evaluation spending limits are now opt-in. New evaluations display their estimated cost with the spending limit off. Set a positive finite dollar amount to enable a cap, or choose none to remove it during review or resume. Uncapped execution requires the reviewed Start action or --yes; the configured warning is displayed before that choice, and above-threshold reviews initially select Back.

The SDK accepts None for the aggregate evaluation or request-ledger limit. Per-request token and retry-inclusive cost reservations remain finite, and completed charges, unknown-dispatch holds, and exact response replay remain durable in both modes. Restoring a lower numeric limit blocks unaffordable new requests without rewriting past spend, completed results or the judging lineage. Prior required-limit hints are advisory and clear on resume, so the live ledger evaluates each newly chosen cap. Persisted runs must explicitly contain a numeric limit or null; omitted settings require a new preparation.

EvaluationServices preserves its prior omitted-argument behavior using the execution budget; explicitly passing spending_limit_usd=None removes only the aggregate cap. The lower-level evaluate_models API retains its finite simulation-envelope requirement and gives an early actionable error when it is absent. Use run_prepared_model_evaluation to derive that envelope from the frozen catalog and request limits automatically.

The native dependency is exp-gateway-native>=0.3.121,<0.4; validation used 0.3.121.