Skip to content

v0.3.1 — eval isolation fix, auto-approval audit record

Choose a tag to compare

@danielstpaul danielstpaul released this 25 Jul 07:55
24efe28

Two correctness fixes, both surfaced by building the new examples/playground reference app.

Fixed: only the first scenario in a silas:eval run was really tested. Silas.configure never invalidated the memoized resolved engine, so a second configure in the same process kept serving the first one — every eval scenario after the first silently ran the first scenario's script while still reporting PASS/FAIL as though it hadn't. If you gate deploys on bin/rails silas:eval with more than one scenario, you were getting false green on 0.3.0 and earlier. Reconfiguring now re-resolves both the engine and the sandbox. The gem's own specs missed this because the spec helper resets config between examples; a real eval suite does not.

Automatic approvals are now recorded. A graded gate that cleared a call — a lambda under its threshold, or an :once rule already satisfied — recorded nothing, leaving the audit trail unable to distinguish "gate ran and passed automatically" from "no gate at all" on a money-moving tool. It now sets approval_state: "approved" with no approved_by; that absence is what marks it automatic, and the inbox renders it as auto-approved by policy. Ungated calls still read nil.

Also:

  • The generated ruby_llm.rb initializer opts into use_new_acts_as, silencing RubyLLM's legacy-API deprecation warning on every boot — Silas never uses acts_as_*. Existing initializers are still never touched.
  • examples/playground — the first genuinely runnable example: a real Rails 8.1 app (Tinker & Co) whose support agent looks up orders and issues refunds. issue_refund is transactional! behind a graded approval lambda (under £20 immediate, over £20 parks for a human), with four evals pinning the flow including exactly-once execution across a park-and-approve.

No migration. 268 specs green on SQLite and Postgres.