Repository navigation
v0.3.0
Minor Changes
-
#32
ebaa5c0Thanks @anuj-sia! - Add registered LLM-judged agent evaluations.Agents can register evaluation folders through
evaluations/index.js. Each
evaluation defines its judge provider and model alongside predefined metrics
and score thresholds. Enabled evaluations score the cumulative durable session
after every completed run and return normalized scores, reasoning, evidence,
usage, and pass status separately from the agent result.Evaluation lifecycle and results are persisted as durable session events.
Registered evaluations also run during agent tests, where judge errors and
missed metric thresholds fail the test without changing production agent
completion semantics.
Patch Changes
-
#32
ebaa5c0Thanks @anuj-sia! - Run registered agent evaluations asynchronously.Completed agent runs now resolve
execution.resultwithout waiting for LLM
judges. Applications can optionally awaitexecution.evaluations, while agent
tests continue to await all registered evaluations and enforce their metric
thresholds. Evaluation lifecycle events and results remain durably persisted
in the session log.