v0.14.0-beta — Gatekeeper: runtime fail-closed enforcement (+ real MAF Agent Harness)

Glass Box tells you what your agent did. Gatekeeper stops it from doing the wrong thing — at runtime, fail‑closed.

AgentEval no longer only measures agents; it can now stop them. This release ships the Gatekeeper: the same checks you red‑team with, moved into the request path so a forbidden tool call, a poisoned argument, an exfiltration attempt, or a prompt injection is blocked before it happens — and every block is recorded as honest evidence, never a false success.

🏰 The Beachhead — the deterministic floor (no LLM, no calibration)

• RunBudgetGate — denial‑of‑wallet / runaway‑loop cap over a per‑run RunLedger (total + per‑tool + monetary sum; atomic check+record).
• DomainAllowListGate — default‑deny egress over the URLs in tool arguments (exfiltration's favourite exit).
• RenderedOutputExfilGate — neutralizes the beacons a client fetches when it renders an answer (markdown image, data: URI, zero‑width chars, fetching tags).

⚖️ The Tribunal — judges that must earn the right to block

• CompositeJudgeGate<TRubric> — a single‑axis LLM judge as a runtime gate (prefilter → model‑under‑timeout → decisive verdict; inconclusive fails closed).
• ParallelJudgeFanOut, JudgeVerdictCache, and GateCalibrationHarness (the Bar) — a judge is scored against a both‑directions gold set and must beat a deterministic baseline before it goes inline. No blanket accuracy claim — the harness is how you measure that, honestly, on your own data.
• IndirectInjectionRubric — the flagship rubric.

🤖 Real agents — including a genuine MAF Agent Harness

Every Gatekeeper sample drives a real agent on a live model. Two of them wrap a genuine MAF Agent Harness agent — IChatClient.AsHarnessAgent(new HarnessAgentOptions { … }) (planning + todo + mode) — and show the Gatekeeper protecting it end‑to‑end: one adds an autonomous re‑invocation loop and has its runaway loop capped by RunBudgetGate; the other sits behind defense‑in‑depth (RunBudgetGate + SequenceGate + DomainAllowListGate, with the read→POST exfiltration blocked mid‑run). You can now build a capable "claw" on the Agent Harness and put a real safety boundary around it — runaway loops capped and off‑host exfiltration blocked at the tool boundary. (The calibrated indirect‑injection judge from the Beachhead+Tribunal sample composes on top — it isn't wired into the two harness samples themselves.)
→ Examples: https://agenteval.dev/gatekeeper/examples.html · the harness itself: https://learn.microsoft.com/en-us/agent-framework/agents/harness

🚪 The Gatekeeper's verdict, surfaced

The gated samples surface why a gate acted — the policy, action, and reason (and, for the Tribunal, the judge's rationale + cited evidence spans) — read straight from the Glass Box gate.* trace. A blocked run shows its reasoning, not a dead "(none)".

🔍 Glass Box — now a first‑class feature

The dual‑boundary trace the Gatekeeper records its evidence into is now documented as a first‑class feature (Observability, in the docs nav): https://agenteval.dev/glass-box.html

🙏 Thanks

• Daniel Costea — our conversations about middleware and what the Glass Box should truly be sparked this whole direction. Thank you, Daniel. 👉 https://www.linkedin.com/in/danielcostea/
• Wesley Steyn and the MAF team for the Agent Harness — the harness makes an agent capable; the Gatekeeper makes that power safe.

📚 Docs

• Gatekeeper — introduction: https://agenteval.dev/gatekeeper/introduction.html
• Gate reference (ranked): https://agenteval.dev/gatekeeper/gate-reference.html
• Examples: https://agenteval.dev/gatekeeper/examples.html
• Glass Box: https://agenteval.dev/glass-box.html

Full changelog: v0.13.2-beta...v0.14.0-beta