0.16.0-beta — Gatekeeper Tribunal & CLI bridge, Copilot Studio MVP, MAF 1.13.0
Pre-releaseGatekeeper reaches production-grade runtime enforcement — Tribunal, CLI bridge, Copilot Studio red-team target, MAF 1.13.0
Gatekeeper reaches production-grade runtime enforcement: a calibrated flagship judge for indirect prompt injection, three more Tribunal judges guarding the model's output (exfiltration intent, system-prompt extraction, and an honesty-preserving over-refusal valve), two deterministic flow-control gates, a defense-in-depth sample, a credential-free attack-the-gate CI recipe, and a language-neutral CLI bridge so any process — not just .NET — gets a policy verdict. Also ships a --sut copilot-studio red-team target (credential-free scaffold; live connector deferred) via a new polymorphic built-in-target seam, and bumps Microsoft Agent Framework to 1.13.0.
Gatekeeper — the flagship calibrated judge (#86)
IndirectInjectionJudge— a calibrated LLM judge for the one attack a keyword list can't catch reliably (paraphrasable natural-language injection). It earns inline enforcement: scored against a canonical 52-case both-directions gold set and a deterministicKeywordOracleGatebaseline, only inline-ready if it beats the oracle with zero missed attacks.ReferentialIntegrityGate+TaintTrackingGate— deterministic flow-control tripwires. Referential integrity blocks a side-effecting call from referencing an id an injection invented; taint tracking blocks a confidential source's value from reaching an external sink.- Sample
07_GatekeeperDefenseInDepth— the calibrated judge alongside all three deterministic gates, driven through a multi-step injection campaign where a different gate catches each step. Verified live against gpt-4o-mini. docs/gatekeeper/attack-the-gate.md— the closed-loop CI recipe: baseline a gated agent, then fail the build the moment a change lets a probe through.
Gatekeeper Stage-2 — "The Bench" (#87)
ExfiltrationIntentJudge,SystemPromptExtractionJudge,OverRefusalJudge— three more calibrated, run-post Tribunal judges. The over-refusal judge is the utility valve: it flags reasonless refusals without ever blocking, protecting usefulness while the rest of the fleet stays fail-closed ("never punish honesty").- The Panel — the output judges composed into a run-post
ParallelJudgeFanOut(fail-closed OR), proven inline against a live model. - Sample
08_GatekeeperOutputPanel— end-to-end on Azure OpenAI: calibrates both judges (gpt-4o-mini: exfil 100%, system-prompt-extract 98%, both zero-missed), shows detection, redaction, and the utility valve.
Gatekeeper CLI interop bridge (#88)
agenteval gatekeeperverb group — runs Gatekeeper gates from any language viagatekeeper list-gates/gatekeeper inspect --gate <id>, emitting a versioned verdict JSON. Covers deterministic gates (keyword, tool/flow-control) credential-free, and judge gates behind an honesty guard — a judge only runs inline if a calibration certificate proves it's inline-ready for that exact model, otherwise it refuses (NotCertified, exit 7).panel:<a,b,…>— a CLI-owned fan-out with per-child sensitive-span redaction.samples/interop/python/gatekeeper_smoke.py— proves the whole contract from a non-.NET process.- New exit codes
GateBlocked(5) /GateInconclusive(6) /NotCertified(7), deliberately off the overloaded exit 2.
Copilot Studio — --sut copilot-studio red-team target MVP (#89)
redteam --sut copilot-studio— red-teams a live Microsoft Copilot Studio agent at text-only/Verbalfidelity, behind a ship-blocking safety gate (--i-understand-live-side-effectsconsent required before any network call).IRedTeamBuiltInTarget— a polymorphic built-in-SUT seam:RedTeamCommandno longer special-cases any target's options or construction, so a future built-in target needs zero edits to the command.- The live MCS connector is deliberately deferred; the credential-free scaffold, safety gates, and CI path are complete and tested via a
sutOverrideseam.
Dependencies — Microsoft Agent Framework 1.13.0 (#90)
- MAF 1.12.0 → 1.13.0 —
Microsoft.Agents.AI/.OpenAI/.Workflows/.Workflows.Generators; sample-only Foundry/Harness previews to the matching1.13.0-preview. No source changes required — the only relevant upstream breaking change (a file-store API rename) isn't used anywhere in AgentEval. - A real transitive floor cascade (
Azure.AI.Projects→Azure.Core 1.60.0→System.Memory.Data/Microsoft.Extensions.Hosting.Abstractions→10.0.9) was caught by an actualdotnet restore, not a static check, and fixed.
A note on versioning
0.15.0-beta already shipped (NuGet + GitHub Release, 2026-07-08) from a branch whose version-bump commit never landed on main. This release starts from 0.16.0-beta rather than reusing that version number.
Full Changelog: https://github.com/AgentEvalHQ/AgentEval/blob/main/CHANGELOG.md#0160-beta---2026-07-13