Skip to content

0.16.0-beta — Gatekeeper Tribunal & CLI bridge, Copilot Studio MVP, MAF 1.13.0

Pre-release
Pre-release

Choose a tag to compare

@joslat joslat released this 13 Jul 16:22
· 158 commits to main since this release
f58af05

Gatekeeper reaches production-grade runtime enforcement — Tribunal, CLI bridge, Copilot Studio red-team target, MAF 1.13.0

Gatekeeper reaches production-grade runtime enforcement: a calibrated flagship judge for indirect prompt injection, three more Tribunal judges guarding the model's output (exfiltration intent, system-prompt extraction, and an honesty-preserving over-refusal valve), two deterministic flow-control gates, a defense-in-depth sample, a credential-free attack-the-gate CI recipe, and a language-neutral CLI bridge so any process — not just .NET — gets a policy verdict. Also ships a --sut copilot-studio red-team target (credential-free scaffold; live connector deferred) via a new polymorphic built-in-target seam, and bumps Microsoft Agent Framework to 1.13.0.

Gatekeeper — the flagship calibrated judge (#86)

  • IndirectInjectionJudge — a calibrated LLM judge for the one attack a keyword list can't catch reliably (paraphrasable natural-language injection). It earns inline enforcement: scored against a canonical 52-case both-directions gold set and a deterministic KeywordOracleGate baseline, only inline-ready if it beats the oracle with zero missed attacks.
  • ReferentialIntegrityGate + TaintTrackingGate — deterministic flow-control tripwires. Referential integrity blocks a side-effecting call from referencing an id an injection invented; taint tracking blocks a confidential source's value from reaching an external sink.
  • Sample 07_GatekeeperDefenseInDepth — the calibrated judge alongside all three deterministic gates, driven through a multi-step injection campaign where a different gate catches each step. Verified live against gpt-4o-mini.
  • docs/gatekeeper/attack-the-gate.md — the closed-loop CI recipe: baseline a gated agent, then fail the build the moment a change lets a probe through.

Gatekeeper Stage-2 — "The Bench" (#87)

  • ExfiltrationIntentJudge, SystemPromptExtractionJudge, OverRefusalJudge — three more calibrated, run-post Tribunal judges. The over-refusal judge is the utility valve: it flags reasonless refusals without ever blocking, protecting usefulness while the rest of the fleet stays fail-closed ("never punish honesty").
  • The Panel — the output judges composed into a run-post ParallelJudgeFanOut (fail-closed OR), proven inline against a live model.
  • Sample 08_GatekeeperOutputPanel — end-to-end on Azure OpenAI: calibrates both judges (gpt-4o-mini: exfil 100%, system-prompt-extract 98%, both zero-missed), shows detection, redaction, and the utility valve.

Gatekeeper CLI interop bridge (#88)

  • agenteval gatekeeper verb group — runs Gatekeeper gates from any language via gatekeeper list-gates / gatekeeper inspect --gate <id>, emitting a versioned verdict JSON. Covers deterministic gates (keyword, tool/flow-control) credential-free, and judge gates behind an honesty guard — a judge only runs inline if a calibration certificate proves it's inline-ready for that exact model, otherwise it refuses (NotCertified, exit 7).
  • panel:<a,b,…> — a CLI-owned fan-out with per-child sensitive-span redaction.
  • samples/interop/python/gatekeeper_smoke.py — proves the whole contract from a non-.NET process.
  • New exit codes GateBlocked (5) / GateInconclusive (6) / NotCertified (7), deliberately off the overloaded exit 2.

Copilot Studio — --sut copilot-studio red-team target MVP (#89)

  • redteam --sut copilot-studio — red-teams a live Microsoft Copilot Studio agent at text-only/Verbal fidelity, behind a ship-blocking safety gate (--i-understand-live-side-effects consent required before any network call).
  • IRedTeamBuiltInTarget — a polymorphic built-in-SUT seam: RedTeamCommand no longer special-cases any target's options or construction, so a future built-in target needs zero edits to the command.
  • The live MCS connector is deliberately deferred; the credential-free scaffold, safety gates, and CI path are complete and tested via a sutOverride seam.

Dependencies — Microsoft Agent Framework 1.13.0 (#90)

  • MAF 1.12.0 → 1.13.0 — Microsoft.Agents.AI / .OpenAI / .Workflows / .Workflows.Generators; sample-only Foundry/Harness previews to the matching 1.13.0-preview. No source changes required — the only relevant upstream breaking change (a file-store API rename) isn't used anywhere in AgentEval.
  • A real transitive floor cascade (Azure.AI.Projects → Azure.Core 1.60.0 → System.Memory.Data/Microsoft.Extensions.Hosting.Abstractions → 10.0.9) was caught by an actual dotnet restore, not a static check, and fixed.

A note on versioning

0.15.0-beta already shipped (NuGet + GitHub Release, 2026-07-08) from a branch whose version-bump commit never landed on main. This release starts from 0.16.0-beta rather than reusing that version number.

Full Changelog: https://github.com/AgentEvalHQ/AgentEval/blob/main/CHANGELOG.md#0160-beta---2026-07-13