Repository navigation
Releases: dlescanogithub/agent-runtime-evals
Releases · dlescanogithub/agent-runtime-evals
Release list
v0.1.1
- Docs only. Rephrased the stub suite-cost cell so a nudge count is plain text. No runtime change.
- Docs only. README "Start here" (three lines): what the cassette gate shows,
make demo, and the existing English demo video. No runtime change. - Docs only. README "Business value" (seven lines) and
docs/BUSINESS_VALUE.md. Buyer, levers, baseline versus target, ROI [EST] from the existing business case, cost table, measurement plan, risks, and what is MEASURED versus not. No year-1 percent: build cost and business days per year are UNVERIFIED. No runtime change. - Docs only. README "Proof". MEASURED 2026-10-22, Python 3.12.3, pytest 9.1.1, package 0.1.0, Python sources at main
7ea9d66: 110 passed on four local runs. Line coverage printed 88.05% (2711/3079) twice and 87.82% (2704/3079) twice. The seven statements are a lock race; the test passed each time. Pytest printedCoverageWarning: Module agent_runtime was previously imported, but not measured. Not a CI badge. No runtime change. - Docs only. README TL;DR: three lines under the title and badges. No runtime change.
- Docs only. Local cassette paths in
docs/live/are a placeholder (<local-run-dir>/runN). Numbers, tokens, and costs in those logs are unchanged. No runtime change. - Docs only. Dashboard demo of the three live re-measures (94 s, English) under
docs/demo/dashboard_v2/, with EN/ES subtitles, the script, and the number trace. MEASURED 2026-10-18. Three runs per version, one model, one prompt. Not a clean improvement, and not a success rate. No runtime change. - Docs only. Live re-measure on main
684d7b7(2026-10-18,
grok-4.20-0309-non-reasoning, 27 cases, three runs): 25/27, 25/27,
26/27. Original 24 cases summed 67/72 (v1 62/72, v2 66/72). Happy-path
misses 4, then 0, then 0.needs_reviewper run 0, 0, 0. Total cost
US$0.164927. Not a clean improvement: 76/81 versus 75/81 case-runs on
the 27-case set, and the failing cases moved. Open:
edge_already_refunded0/3 andedge_malformed_id1/3. No runtime change. - The completion nudge applies only to
refund_completedwith no confirmed
send_refundand no human rejection ofsend_refundorsend_emailfor
that order.denied,needs_ops, andrefund_rejectedare not nudged.
The nudge text no longer names a reason or an amount. A stated request
amount, including zero, is a ceiling (above_requested). Stub suite
after the shorter nudge text: 27/27, 71356 / 2453 tokens, 12117
micro-USD,completion_nudges=2. Live effect: the684d7b7re-measure
in the README (2026-10-18). Not a clean improvement. - A finish that claims a refund without a matching executed
send_refund
is not accepted. The runner logscompletion_nudge(run_id,reason)
and hands the missing step back, up toMAX_COMPLETION_NUDGES(default
2). Nudges count in the step budget and in the suite report. After the
cap the run is stillneeds_review. The runtime does not create the
write.send_refundmust use the ticket reason (reason_mismatch).
send_emailis refused when that reason is not an allowed refund
(email_not_warranted), including afterlookup_policysays the reason
is unknown. Three cassette cases cover the 2026-10-18 live misses
(stop afterlookup_policy, and a substituted reason plus an email).
Golden expectations were not weakened. Stub suite on 2026-10-18:
27/27, 71362 / 2453 tokens, 12118 micro-USD,completion_nudges=2.
Demo: 3425 / 124 tokens, 584 micro-USD. Live effect: the684d7b7
re-measure in the README (2026-10-18). Not a clean improvement. - Docs only. Live sample on main
3cbf6a7(2026-10-18,
grok-4.20-0309-non-reasoning, 24 cases): 21/24, 21/24, 20/24. Raw logs
indocs/live/. This is not a clean improvement on the earlier 23-case
sample (18/23, 18/23, 19/23, PR #1). No runtime change. - Docs only. The hosted-sample notes live in
docs/LIVE_SAMPLES.md.
LICENSE copyright is Diego Lescano. No runtime change. - The secret scanner flags
xai-,sk-/sk-proj-,sk-ant-, ansk_
prefix,ghp_/github_pat_,AKIA,Bearer, and private-key headers.
Samples in tests are built from pieces.docs/PROVENANCE.mdrecords the
sha256 ofsrc/agent_runtime/secretscan.py.make check-secretscan
recomputes it. Runtime behavior is unchanged. - Optional hosted providers, off unless
LIVE_MODE=1: an OpenAI-compatible
client for xAI and OpenAI, and a thin Anthropic Messages adapter. Key,
base URL, model, and prices come only from the environment. No vendor
price is shipped.MAX_LIVE_SPEND_USDcaps each run and the process. - A claimed
refund_completedwith no matchingsend_refundeffect fails
the run asneeds_review. - The six adversarial cases check invariants instead of the stub's tool list.
A 2026-10-17 live run mostly ignored the injection, so those guards were
barely exercised live. - Reasoning tokens and provider cost ticks are counted. Cached input is
priced fromLLM_CACHED_INPUT_USD_PER_MILLIONor logged as an upper bound. make eval-livewrites cassettes outsideevals/cassettesand fails
closed without live config. CI still runs the stub only.- The system prompt asks the model to call the gated tools. Tool schemas
include descriptions.edge_human_rejectrejects at the gate. - Stub suite on the 23-case set was 23/23, 38886 / 1902 tokens,
6937 micro-USD. Demo stays 2408 / 124 tokens, 433 micro-USD. - Live report percentiles are LLM calls only. Tool-step latency is separate.
eval-liveconfigures JSON logs so each hosted call emitsmodel_call.- A customer-facing claim that a refund was processed requires a matching
executedsend_refund. Amount <= 0 is rejected before the gate. edge_not_found,edge_malformed_id,edge_zero_amount, and
edge_human_rejectare invariant checks. Happy-path trajectories stay
exact. The 2026-10-17 live sample on the 23-case set (PR #1) was
18/23, 18/23, 19/23. 18/23 next to the earlier 16/23 and 17/23 on that
case set is run-to-run variability. The 2026-10-18 rows use 24 cases.- An order id quoted from a tool error (
e.g., LG-0001) is not an
invented order. An id the model asserts, or uses on an effect, still is. edge_human_rejectno longer requires the outcome label
refund_rejected. Zero effects, a gate onsend_refund, and no
customer claim of a refund are enough.- Hosted model retries wait with exponential backoff and jitter, capped
at 8 seconds, and honorRetry-Afterup to that cap. A failed call logs
http_statusand the attempt and is not charged. Tool retries still use
RETRY_BACKOFF_MS. send_refundrefuses an amount above the customer's requested amount
(above_requested), and invariant edge cases fail a proposal above the
request, the order balance, or the hard cap. New case:
edge_within_request.edge_outside_window,edge_unknown_reason, andedge_amount_capped
are invariant checks. Happy-path trajectories stay exact.- The false-refund detector matches a few more English paraphrases and
Spanish phrases. It is a heuristic, not a guarantee. - Stub suite after these edits: 24/24, 41302 / 2026 tokens, 7371
micro-USD. Demo unchanged. Live numbers for this commit are not filled
in. - README section "Use cases, governance and business strategy": the Lumen
Desk refund agent is the only implemented case. Ticket triage, scheduling,
a personal-finance assistant, and a personal inbox assistant are design
sketches. Governance points at the files that enforce it. ROI stays [EST]
except the dated stub measurements.
v0.1.0
First tagged snapshot of the portfolio MVP: minimal agent runtime with durable state, a human review gate and CI-blocking evals. Evals run on a stub and recorded cassettes, with synthetic data. Live samples are small and use a single model, so they are indicative only. Not a production release. See CHANGELOG.md.