Skip to content

EvalFence v0.3.0

Latest

Choose a tag to compare

@tiramitree tiramitree released this 28 Jul 21:44

EvalFence v0.3.0 adds an independent fail-closed custom-agent boundary contract while preserving the v0.1 interval/metric and v0.2 keyed-manifest contracts.

Included:

  • evalfence.agent-boundary.v1 and the audit-agent-boundary CLI command.
  • Declared checks for input allowlists, task/scoring-oracle classifications, mutable source aliases, read/write capability groups, callable and mediated counts, and baseline-snapshot order.
  • Stable findings for oracle exposure, unapproved inputs, unmediated write callables, pre-baseline constructor code, mutable aliases, invalid names/digests/policies/counts, contradictory order, aggregate count overflow, and inconsistent absent-input declarations.
  • A source- and runtime-bound adapter for microsoft/STATE-Bench@4efcbf2d4fe60df04878859b692d9391f3d5b33a with exact Git blob registration, AST guards, a clean-worktree gate, Python UTF-8 normalization, three offline domain observations, and one deterministic no-API causal control.

Registered STATE-Bench observations at the pinned revision:

  • Three public test tasks expose 14 state-requirement items, nine task-requirement items, and 20 callable write handlers at the declared custom-agent constructor boundary.
  • The observed event order in each domain is constructor, baseline snapshot, first agent turn, final snapshot.
  • One bounded shopping control reaches deterministic state score 1 with two normal harness-executed tool calls when using the supplied state requirements; removing only state_requirements produces score 0, no calls, no errors, and an empty state diff.
  • EvalFence emits three EF301, three EF303, one EF304, and two EF305 findings for the registered declaration.

Verified at commit de3c49e6a6960fbafea59381a69439558a239e0e:

  • main CI run 30401556015 passed quality, Windows, macOS, and ContextBench jobs.
  • main SWE-bench manifest run 30401556000 and STATE-Bench boundary run 30401556102 passed.
  • tag CI run 30401692397, tag SWE-bench manifest run 30401692383, and tag STATE-Bench boundary run 30401692378 passed.
  • Rust formatting, Clippy with warnings denied, all-target tests, release build, all three source-bound controls, exact evidence reproduction, and generated-evidence privacy gates passed in public CI.

Boundaries:

  • Input and oracle classifications are adapter declarations; the Rust core does not infer them or prove that an adapter observed the real runtime.
  • The STATE-Bench control is not a protocol-compliant or official benchmark score, model evaluation, security assessment, defect report, cheating claim, or leaderboard reproduction.
  • Removing one context field does not isolate arbitrary same-process Python. Filesystem access, task-ID-based lookup, and other same-process hooks remain outside this control.
  • The control does not establish that any submitted agent used the declared fields or callables, or that any public result was affected.
  • This release is not a production deployment, external review, adoption, or endorsement signal.
  • STATE-Bench remains copyright its authors under the MIT License; EvalFence does not vendor its source, tasks, environments, prompts, or trajectories.
  • The repository was built with substantial AI assistance under tiramitree.

No precompiled binaries are attached. The release contains GitHub-generated source archives only.