0.79.0
Replay a Passing Scenario Instead of Asking the AI Again (#415, #417, #418)
A scenario that already passed can now re-run by replaying the actions it recorded, with no
per-step AI call at all:
settings:
cacheStrategy:
aiDecisionCacheStrategy: { type: "Disk" }
replayWithFallback: trueThe existing AI-decision cache only helps when the UI tree and the context hash match exactly, so
a clock or any rotating content on screen makes it miss and the AI is asked again. For a scenario
that already passed, the useful information is the sequence of actions, not a per-screen lookup.
The two mechanisms form a ladder — replay first, then the layout-hash cache, then the AI — so a
scenario degrades gracefully instead of failing when the app moves.
- This buys cost, not wall-clock time. Replay deliberately waits out the gap between the
recorded steps before reading the next screen. Without the AI's own latency between steps the app
is driven faster than it was when the run passed, and a screen gets read before it settles. - Image assertions are what verify a replayed run. They still run, and they are the only thing
that can tell a replayed run apart from a wrong one, so a scenario with none of them is run
normally rather than replayed. Only the scenario you run needs them; the dependencies it pulls in
do not. - Anything unexpected falls back to a normal AI-driven run. A recorded element that is gone, an
index that no longer exists on screen, or a failed image assertion restarts the whole scenario in
normal mode. The replay attempt does not consumemaxRetry, and it keeps the AI-decision cache so
the fallback run can still skip the AI for the steps that were already correct. - The report tells the two apart. Steps now carry a source —
Ai,CacheorReplay— in
place of the oldcacheHitflag, so a replayed step is no longer reported as a cache hit.
Replay is a project-wide switch: it is decided once for a whole run, so there is no per-scenario
key that would promise a granularity the runner does not have.
Token Usage for Every AI Call, Including Image Assertions (#416)
Arbigent only logged API calls made inside decideAgentActions, so image assertions — delegated to
Roborazzi's assertion model — and scenario generation were billed against the same key and the same
model while appearing nowhere in the logs. Any cost estimate built from jsonls silently missed
them.
Usage is now recorded in the shared network interceptor, so one place covers decision calls, image
assertions and scenario generation. Each billed response becomes one file in a new usages
directory:
{"request_uuid":"...","model":"...","input_tokens":3762,"cached_input_tokens":2560,"output_tokens":106,"total_tokens":3868}- Cached input is now visible.
cached_input_tokensseparates discounted cached input from
full-price input. - One file means one billed call. A response with no usage object — a rate limit, say — is not
billed and is not recorded. - All four providers are covered.
openai,geminiandazureopenaishare the same client,
andanthropicrecords the same way.
What's Changed
- Add replayWithFallback mode to skip per-step AI calls by @takahirom in #415
- Record token usage for image assertion calls too by @takahirom in #416
- Make replayWithFallback a project-wide setting by @takahirom in #417
- Fix report export failing after a replayed run by @takahirom in #418
Full Changelog: 0.78.1...0.79.0