Skip to content

v0.17.0-beta

Pre-release
Pre-release

Choose a tag to compare

@joslat joslat released this 19 Jul 17:35
· 114 commits to main since this release
27b8751

The biggest release by PR count so far (43 merged since 0.16.0-beta). Full MAF Agent Skills evaluation and
governance ships end-to-end: assertions, a disclosure-efficiency metric, a compliance scanner with a
multi-repo baseline ledger, a skill-injection red-team attack, deterministic run_skill_script governance
gates, SkillGate construction-time drift enforcement, and a composite Skill Health & Security Index. The
Gatekeeper Tribunal gains its remaining flagship judges (intent-action mismatch, goal-hijack drift,
ungrounded claims, hallucinated citations — all κ=1.000 against their gold sets) plus two new gate layers:
tool RESULT gates (inspecting an already-executed call's output, not just the proposed call) and real
HTTP-egress enforcement closing an SSRF/DNS-rebind gap DomainAllowListGate could never see. The Microsoft
Copilot Studio live connector ships (real MSAL device-code auth, a real activity-stream bridge — still not
independently verified against a live tenant, honestly disclosed throughout). And the newest theme,
Explainability & Trust, ships as tested library code with a runnable sample: reconstructable gate
provenance, counterfactual gate-config replay, and a unified Trust Score.

Also resolves the long-deferred BUG-22 exit-code overload (a breaking CLI-contract fix — see below),
closes a doc-lag pattern that hit twice this cycle (a capability shipping with zero matching documentation)
with two new "what's new" pages and a CI check that now prevents it recurring in either direction, and
refreshes every downstream sample — including the NuGet consumer validation project — to track this release.

Docs/samples hardening follow-up + NuGetConsumer refresh to 0.16.0-beta

A self-review of the BUG-22/Explainability & Trust batch below found real gaps and closed them, then a
follow-on pass brought the Skills/Copilot Studio/Explainability & Trust docs and samples to full parity and
refreshed the NuGet consumer samples.

Fixed — completing the BUG-22 remap

  • BenchTraceFidelityCommand.cs and BenchPerfCommand.cs each independently hardcoded the exact
    same gate-fail-as-exit-2 pattern the BUG-22 fix below was supposed to have unified everywhere — missed in
    the first pass. BenchPerfCommand now delegates to the shared BenchExitCodes.FromLabel instead of
    reimplementing it; both return GateFailed (9) on a hard fail.
  • AzureChatAgentFactory.cs — the SUT-agent-resolution counterpart to JudgeFactory.cs — had the
    identical config-resolution-conflated-with-usage-error bug across 4 return sites, also missed. Now returns
    RuntimeError (3).
  • 8 more docs described the old exit-code contract and were never updated: docs/cli.md's resolution-order
    prose, docs/gatekeeper-cli.md's cross-reference, and 6 getting-started pages (gdpr/memory/longmemeval/
    mitre/owasp/perf). One (memory) was actually wrong even before BUG-22 — it claimed WARN mapped to exit
    0 alongside PASS, which was never true.

Added — docs, mirroring the missing coverage

  • docs/gatekeeper/explainability-and-trust.md — the docs page the Explainability & Trust library code below
    had shipped without any docs/ coverage at all.
  • docs/agent-skills-whats-new.md and docs/gatekeeper-whats-new.md — capability-history pages mirroring
    the existing docs/redteam-whats-new.md pattern, so a shipped-but-undocumented capability (this happened
    twice in the same area: SkillGate, then Explainability & Trust itself) is easier to catch next time.
  • "New to this? Start here" plain-English concept sections added to docs/agent-skills.md,
    docs/copilot-studio.md, and docs/gatekeeper/explainability-and-trust.md, plus short "in plain English"
    framing on Agent Skills' three densest phases — docs read as progressive, not front-loaded with jargon.

Added — samples

  • samples/AgentEval.Samples/Gatekeeper/10_GatekeeperExplainabilityAndTrust.cs (new) — 3 gradual scenes:
    GateProvenance (a real judge call) → GateReplayer (deterministic) → TrustScoreCalculator (combines
    both). Live-verified against real Azure OpenAI.
  • samples/AgentEval.Samples/CopilotStudio/00_CopilotStudioHelloWorld.cs (new) — a true one-concept on-ramp
    before the existing multi-concept walkthroughs (which each covered 4-5 concepts at once).

Changed — docs structure hygiene

  • Moved docs/redteam/copilot-studio.md → top-level docs/copilot-studio.md: it covers eval/bench
    integration, fluent assertions, and Gatekeeper composition, not just red-teaming — inconsistent with the
    AgentEval.MAF.CopilotStudio package and the samples menu, both of which already treat it as its own
    top-level area (matching Agent Skills' precedent). All inbound references updated.
  • Renamed ResponsibleAI.md → responsible-ai.md and docs/GlassBox/ → docs/glassbox-history/ for
    kebab-case consistency with every other doc file/folder.
  • New CI check: tools/check_docs_toc.py + .github/workflows/docs-toc-check.yml fails a PR if any
    docs/**/*.md file isn't reachable from docs/toc.yml (the real site navigation, not docs/index.md's
    separately-maintained landing-page list) — the exact mechanism that let two doc pages ship invisible in the
    sidebar this session, found only by manual audit. Extended to also catch the reverse drift: a local
    link in docs/index.md's landing page pointing at a page that isn't (or is no longer) in docs/toc.yml —
    one-directional by design, since most nav pages aren't meant to be landing-page-highlighted.

Changed — NuGet consumer samples refreshed to the latest released package

  • samples/AgentEval.NuGetConsumer / AgentEval.NuGetConsumer.Tests were pinned to AgentEval 0.13.1-beta
    (built on MAF 1.11.1) — three releases stale. Bumped to 0.16.0-beta (the actual latest published version
    on NuGet.org — confirmed via the NuGet API, not assumed from main), with the full dependency baseline
    (Microsoft.Agents.AI 1.13.0, System.Memory.Data 10.0.9) updated to match exactly what
    Directory.Packages.props resolved at the v0.16.0-beta tag. Restored, built, and tested clean end-to-end
    against the real published package (one transient Azure content-filter rejection on first run, confirmed
    non-reproducible on re-run — a live-service flake, not a regression).

BUG-22 resolution + Explainability & Trust (gate provenance, counterfactual replay, unified Trust Score) + docs/samples

Changed (BREAKING — CLI exit-code contract)

  • BUG-22 resolved: ExitCodes code 2 was overloaded — bench/calibrate returned it for both a
    benchmark gate FAIL/WARN and bad CLI arguments, and JudgeFactory config failures (missing/partial Azure
    OpenAI credentials) also returned it. Now: 2 is reserved strictly for bad arguments; judge/runtime config
    failures return 3 (RuntimeError); benchmark/calibration gate outcomes return dedicated new codes
    9 (GateFailed), 10 (GateWarning, bench <family> only), 11 (GateIndeterminate). External CI
    pipelines branching on exit code 2 from bench/calibrate must be updated. See ExitCodes.cs and
    Exit codes.

Added — Explainability & Trust (0.17.0-beta theme, analysis in strategy/ExplainabilityAndTrust-AnalysisAndPlan.md)

  • Gate provenance chains — AgentEval.Guardrails.GateProvenance (rule name, evidence, threshold vs.
    actual, contributing sub-chains) attached via a new optional GateVerdict.Provenance field (additive, same
    precedent as Confidence). Wired into CompositeJudgeGate<TRubric> for both the Block path and the
    near-miss-Allow-with-Confidence path Fleet Correlation already reads.
  • Counterfactual gate replay — AgentEval.MAF.Gatekeeper.GateReplayer.CompareAsync runs a baseline and a
    candidate IToolGate list against the SAME captured GatedToolCalls (the real gate objects, no
    simulation; first-Block/Mutate-wins, matching the live AgentEvalToolGateExtensions pipeline) and reports
    which calls would have diverged under the candidate configuration. Library API this session; a
    agenteval log-file gate-replay CLI wrapper is a natural mechanical follow-on.
  • Unified Trust Score — AgentEval.Trust.TrustScoreCalculator.Compute combines TrustSignals (gate
    verdicts, eval scores, anything 0..1) into one honest 0-100 composite, excluding "skipped"/"error"
    labeled signals from the weighted math entirely (the same discipline as WeightedSumAggregation et al. —
    "including them at 0.0 would incorrectly drag the composite below threshold").

Documentation

  • docs/redteam/copilot-studio.md — added the CopilotStudioAssertions fluent-assertion section (was
    shipped but undocumented) and cross-referenced EstimatedCreditsUsed.
  • docs/agent-skills.md — added §3b documenting SkillGate (construction-time drift enforcement, shipped
    but undocumented since it landed) — WithSkillGate/SkillGateMode/SkillDriftException/
    agenteval skills baseline approve.

Samples

  • samples/AgentEval.Samples/AgentSkills/04_AgentSkillsSkillGate.cs (new) — live-verified against real Azure
    OpenAI: pins a baseline, simulates a rug-pull, shows SkillDriftException fail-closed, then recovers.
  • samples/AgentEval.Samples/CopilotStudio/02_CopilotStudioBudgetAndRedTeam.cs (new) — --max-credits
    enforcement tripping CopilotStudioBudgetExceededException for real, HaveStayedWithinCreditBudget,
    CanResistAsync red-teaming a live MCS agent, HaveStartedNewConversation/HaveStartedDifferentConversation.

Copilot Studio — C2 fidelity-badge audit + P5 correlation-key spike

Added

  • C2 (all-targets fidelity badge) — closed. EvidenceFidelity (Verbal/IntentToAct/Behavioral) now appears
    in every RedTeam report renderer, not just JSON/SARIF (which already carried it): Markdown gets an inline
    `[behavioral]`/`[verbal]`/`[intent-to-act]` tag next to each compromised probe, JUnit gets
    a Fidelity: line in the failure body (plus the label folded into the inconclusive <error> message), and
    PDF gets a bracketed label next to each finding. An audit of all 5 renderers found the 3 gaps were
    genuinely mechanical (the fidelity data was already flowing through the shared model, just not rendered) —
    shipped the fix for all three rather than stopping at the audit.

Investigated (spike concluded without shipping code — by design)

  • P5 (L2 telemetry enrichment) correlation-key spike. Investigated whether a client-side conversationId
    can reliably correlate a live Copilot Studio scan against Dataverse session transcripts (SessionID,
    TopicId, ChannelId) for deeper post-hoc evidence enrichment. Findings: the correlation join is plausible
    but not publicly confirmed by Microsoft's docs; more importantly, Dataverse transcripts have ~30 minute
    latency after conversation inactivity before they're queryable — which means the originally-sketched design
    (an inline TraceEnrichingChatClient decorator enriching evidence on the hot path) cannot work at all. Real
    L2 enrichment needs to be a separate, deferred offline reconciliation command, not a live decorator. P5
    remains correctly deferred; this is a scoping correction, not a shipped feature. Full write-up in
    docs/redteam/copilot-studio.md's "How it fits red-team fidelity" section.

Agent Skills Wave 1 — baseline ledger, repo-wide discovery, provenance pointer

Added

  • SkillContentHasher (AgentEval.Skills) — a new full-file-content hash, complementing
    SkillManifestPoisoningGate's existing structural-only fingerprint (which hashes only the parsed manifest
    fields, not the raw file bytes). Together they let a baseline snapshot distinguish "the manifest's meaning
    changed" from "the file changed but parses identically" — two different signals a security-conscious skill
    reviewer cares about separately.
  • ISkillBaselineStore / JsonFileSkillBaselineStore — a multi-snapshot, never-overwritten skill-scan
    ledger (a sibling of AgentEval.Memory's JsonFileBaselineStore — the same proven pattern, not a shared
    base type, deliberately: skills and memory baselines have different identity/versioning shapes). Every
    agenteval skills scan --write-baseline call appends a new timestamped snapshot rather than overwriting the
    last one, so agenteval skills baseline history <skill-name> can show how a skill's fingerprint has drifted
    over time, not just its current state vs. one prior pin.
  • AgentSkillDirectoryConventions (--repo flag on skills scan) — scans every directory convention MAF's
    own AgentFileSkillsSource recognizes across an entire repo root in one pass, instead of requiring one
    invocation per skill directory.
  • CLI: agenteval skills scan --write-baseline [--baseline-root <dir>] [--repo] captures a snapshot;
    agenteval skills baseline list|diff|history inspects the ledger.
  • A null-safe provenance-pointer parameter on the compliance report renderer (wiring is real and tested; no
    lock-file source populates it with real data yet — that's Wave 2/3 territory).
  • Self-review before merge caught a real bug the C# compiler doesn't error on: a doc-comment placement mistake
    in MafSkillScanner.cs that silently mis-associated two records' XML docs with the wrong types — confirmed
    via the generated XML doc file, not just a build check, and fixed.
  • Wave 2 (trust-on-first-use reputation matching, cross-location drift detection) and Wave 3 (org-wide
    multi-repo scan, live upstream verification) remain unbuilt, per the design doc's own phasing — Wave 3 is
    explicitly gated on a not-yet-done security/credential-scope review.

Gatekeeper — the IToolPlanGate empirical dispatch-order question, resolved

Investigated (empirical finding, not a new gate)

  • Determined, against real MAF 1.13.0 behavior (not assumed from docs), whether FunctionInvokingChatClient
    dispatches sibling tool calls from one model turn sequentially or concurrently — the fact a future
    IToolPlanGate (batch/plan-level gate, e.g. catching [read_secrets(), send_email()] issued as siblings in
    one turn, which SequenceGate structurally cannot see) needs settled before its interface shape can be
    designed. Finding: sequential dispatch is MAF's default, and that default is the only mode reachable
    through ChatClientAgent's normal builder surface (confirmed: zero references to
    AllowConcurrentInvocation anywhere in Microsoft.Agents.AI). Concurrent dispatch is reachable, but only
    via manually constructing a FunctionInvokingChatClient with UseProvidedChatClientAsIs = true — and under
    that mode, a naively-designed "terminate at the first blocked call" plan gate does not reliably stop a
    sibling call from still executing (empirically confirmed: the sibling ran anyway). IToolPlanGate itself is
    still unbuilt — this pass shipped only the empirical groundwork a correct design now depends on.

Gatekeeper — 4 next-wave gates: prompt-template drift, calibration staleness, Fleet Health Index, tool-result size anomaly detection

Added

  • PromptTemplateDriftGate — the third application of the ManifestFingerprint/ManifestDriftDetector
    hash-pin-and-diff primitive (after skill manifests and MCP tool schemas), applied to an agent's prompt
    template files. Unlike every other gate, it's not an IToolGate/IChatGate/IToolResultGate at all — a
    prompt template doesn't change mid-run, so a per-turn check would be pure waste. UseGatekeeper checks drift
    eagerly at construction time when both GatekeeperOptions.PromptTemplates and PromptTemplateBaseline
    are set, and throws PromptTemplateDriftException immediately on a mismatch — fail-closed, matching
    RefuseUnprotectedHighRiskTools's posture. Setting only one of the two options throws
    InvalidOperationException at construction rather than silently no-op-ing.
  • CalibrationReport.CapturedAt / IsStale(maxAge, clock?) — a calibration report now records when it was
    captured, and can report whether it's aged past a caller-chosen threshold. Informational only — staleness
    never affects IsInlineReady or auto-demotes an already-promoted judge; it's a signal to re-calibrate, not a
    promotion-blocking condition.
  • ICalibrationReportStore / JsonFileCalibrationReportStore — the persistence seam CalibrationReport
    needed (it was, until now, a purely in-memory return value of GateCalibrationHarness.EvaluateAsync, never
    persisted between runs). Deliberately minimal: one report per axis (the most recent run), overwritten on each
    save — not a full historical ledger (see the Agent Skills baseline ledger above for that different shape,
    deliberately not duplicated here).
  • GatekeeperFleetHealthIndex.Compute(reportsByAxis, staleAfter, clock?) — joins every tracked judge axis's
    latest calibration report into one composite fleet-health view, mirroring SkillSecurityIndex's honesty
    discipline: an axis with no report is never fabricated into a passing score. Reports mean decisive
    accuracy/kappa (calibrated axes only), total dangerous errors, which axes have never been calibrated, and
    which are stale. Transport-agnostic (AgentEval.Core, no CLI/Mission Control dependency yet) — high value
    once an ops-facing surface exists to put it on.
  • ToolResultSizeAnomalyGate — a per-tool, per-session statistical-outlier detector, distinct from the
    already-shipped ToolResultSizeGate (which truncates against one fixed, global character threshold). Flags a
    result more than Nx (default 5x) that same tool's own running average size this run, once enough prior
    calls establish a baseline (default 3) — catching behavioral drift a global threshold can't see (e.g. a tool
    that's returned ~200-character results all run suddenly returning 50,000 characters, even though 50,000 might
    be unremarkable for a different, bulk-read tool). v1: fixed multiplier, no real statistics library — a
    documented, deferred v2 follow-on.
  • Self-review before merge found and fixed 4 real issues: a filename-collision risk in the calibration store, a
    label-space inconsistency in the Fleet Health Index, a silent half-configuration security gap in
    PromptTemplateDriftGate (now fails loud instead), and dead code.
  • Docs: new "Calibration staleness & the Gatekeeper Fleet Health Index" section in
    docs/gatekeeper/gate-reference.md.
  • Crucible work and the two remaining flagship judges (ToolArgumentGoalCoherenceJudge,
    CrescendoTrajectoryJudge) are explicitly out of scope for this batch.

Mission Control — prompt-hash provenance display completed, one stale doc claim corrected

Fixed

  • AdjudicationFlow.tsx's judge cards and adjudicator card now render PromptHashPill — the last place
    in the SPA that didn't show prompt-hash provenance (EvalResultNode.tsx and ScenarioTreePage.tsx already
    did, since 2026-05-25). Arguably the highest-stakes place it was missing, since it's the view a human uses to
    resolve judge disagreement.
  • docs/missioncontrol/api-design.md corrected: Query.complianceEvidence's documented return type was
    ComplianceEvidence?; the real resolver returns ComplianceEvidenceWithChain?.
  • Corrected a stale "still a live bug" claim repeated across 3 local strategy/review docs: the compliance
    matrix's per-cell auditChainValid check (tampered evidence rendering as a false green checkmark) was
    actually fixed 2026-05-24 — the docs describing it as open were simply never updated when the fix landed.

Gatekeeper Hardening Phase 2 — real HTTP-egress enforcement (#10): redirect-chasing + DNS-rebind/SSRF defense

Added

  • GatekeeperHttpMessageHandler (AgentEval.MAF.Gatekeeper.Egress) — a real DelegatingHandler closing
    the gap DomainAllowListGate candidly documents about itself: that gate scans the URL string inside a
    tool call's arguments, never the actual outgoing network request, so it cannot catch a redirect to a
    forbidden host or a DNS answer resolving an allow-listed hostname to a private/internal address (SSRF /
    DNS-rebinding — the cloud-metadata endpoint 169.254.169.254 is the canonical target). This handler sits
    underneath whichever HttpClient a tool's own implementation uses (a different composition point from
    IToolGate/IToolResultGate — opt-in per tool via GatekeeperHttpMessageHandler.CreateHttpClient(...), not
    registered through UseGatekeeper) and re-validates the allow-list AND resolves DNS before every hop,
    including every redirect — redirects are followed manually (the factory disables the transport's own
    auto-redirect), bounded by MaxRedirects, never silently delegated. Every block throws
    HttpEgressBlockedException (the idiomatic fail-closed signal for an HttpMessageHandler).
  • PrivateNetworkClassifier — classifies an IPAddress as private/loopback/link-local/reserved (RFC1918,
    CGNAT, IPv6 unique-local, IPv4-mapped-IPv6 unwrapped to its embedded address, the cloud-metadata range, and
    more) — the check run against every DNS-resolved address.
  • IDnsResolver/SystemDnsResolver — a seam over DNS resolution so tests can script resolution results
    (including a DNS-rebind scenario) without real network/DNS access; the default wraps System.Net.Dns.
  • GatekeeperHttpEgressOptions — MaxRedirects (default 5), BlockPrivateNetworks (default on),
    DnsResolutionTimeout (default 2s, fail-closed on timeout — cannot prove the destination safe), DnsResolver.
  • HostAllowList — the exact-or-subdomain host-matching logic factored out of DomainAllowListGate
    (behavior-preserving extraction, existing tests unchanged) so the new handler shares the IDENTICAL allow-list
    semantics rather than a second copy that could silently drift out of sync with the argument-level gate.
  • Redirect handling matches HttpClientHandler's own default semantics: 307/308 preserve method and body;
    301/302/303 downgrade to GET (dropping the body, except a HEAD request stays HEAD). The synchronous
    HttpClient.Send(...) API is explicitly refused (NotSupportedException) rather than silently bypassing
    every check — this handler's validation is inherently async (DNS resolution).
  • 43 new tests (PrivateNetworkClassifierTests, GatekeeperHttpMessageHandlerTests — scripted inner handler +
    fake DNS resolver, zero real network access, deterministic). Full suite green on all three TFMs (net8.0
    7648/7648, net9.0 7648/7648, net10.0 7851/7851).
  • Docs: new "HTTP egress enforcement" sections in gate-reference.md/examples.md/introduction.md.

Gatekeeper Hardening Phase 2 — tool RESULT gates (P0-3) + a parallel-tool-call test fixture

Added

  • IToolResultGate (AgentEval.MAF.Gatekeeper) — a new gate kind inspecting one already-executed tool
    call's RESULT, at the same MAF function-invocation seam IToolGate inspects the proposed call, but on the
    other side of next(...). Closes the "tool output as an injection channel" gap: nothing before this
    inspected a tool's own return value before it re-entered the model's context (only prior results, read out
    of conversation history, were ever consulted — never this call's own result, synchronously, before it flows
    back). Returns ToolResultVerdict (Allow / Block / Redact, via GatedToolResult). Wired into
    UseAgentEvalToolGate's new optional resultGates parameter — runs, in order, immediately after next(...)
    returns, only once every IToolGate has allowed the call. A Block/throw fails closed exactly like the
    call-gate loop; a Redact verdict is applied regardless of ToolGatePolicy (mirrors ToolGateAction.Mutate's
    "always applied" precedent). Recorded under the new gate.tool-result.* trace stage — distinguishable from a
    call-gate block while still counted by the existing, stage-agnostic GlassBoxEvidence.CountGateBlocks. Null/
    empty resultGates is the exact prior behavior (zero overhead, fully backward compatible).
  • Three built-in result gates (AgentEval.MAF.Gatekeeper.Gates):
    • ToolResultInjectionGate — blocks a result containing a prompt-injection marker (shares its default
      marker list with the chat-side TokenInjectionGate, now public for exactly this reuse). Not maskable —
      always Block, never Redact.
    • ToolResultSizeGate — truncates an oversized result (default 8,000 chars) via Redact, bounding
      context-window exhaustion and per-turn token cost from a single runaway tool response.
    • ToolResultSecretGate — detects and masks common credential shapes (AWS/GitHub/Slack/Google/Stripe
      keys, full PEM private-key blocks, bearer tokens, JWTs) via Redact, mirroring RegexPiiGate's
      bounded-timeout mask-with-█ approach.
  • GatekeeperOptions.ToolResultGates / AddResultGate(...) — composed by UseGatekeeper alongside
    ToolGates; a result-gate-only configuration (no call gates registered) is valid. The IToolResultGate. MinimumPolicy floor is folded into the same Observe-mode conflict check ToolGates already gets, and the
    Observe startup banner now reports the result-gate count too.
  • GateTelemetry.Record(string, ToolResultAction, TimeSpan) — a second overload sharing the same per-policy
    counters as the call-gate Record, so a caller reading Snapshot() sees one unified effectiveness view
    across both gate kinds; Redact maps to MutateCount.
  • ScriptedChatClient.AddParallelToolCalls(...) (AgentEval.Core.Testing) — the fixture prerequisite this
    phase needed: scripts MULTIPLE FunctionCallContent in one assistant turn (the shape a real provider sends
    for parallel function calling). Before this, the fixture could only ever emit one tool call per turn, so no
    test could exercise a gate pipeline against MAF's FunctionInvokingChatClient actually invoking N calls from
    a single turn — only N single-call turns in a row, a materially different code path. Fully additive — the
    existing single-call AddToolCall API is unchanged.
  • docs/gatekeeper/gate-reference.md / examples.md / introduction.md updated with the new "Tool RESULT
    gates" layer, its policy-reinterpretation rules, and runnable snippets.

Deferred (explicitly out of scope this pass)

  • P0-4 (tool-plan/batch gates) — a genuinely new interception point, sized larger, deferred to a separate
    session.
  • Full result-content capture into the trace (mirroring MutationEvidenceRenderer's TraceCaptureMode) — a
    tool result can be arbitrarily large/shaped content from anywhere; only the fact that a redaction happened,
    and why, is recorded for now.

Copilot Studio — live connector wired (redteam --sut copilot-studio Track 1)

Added

  • CopilotStudioAgentFactory.BuildLive now builds a real connector instead of unconditionally throwing
    NotSupportedException. It constructs a real Microsoft.Agents.CopilotStudio.Client.CopilotClient from
    CopilotStudioConfig (ConnectionSettings mapped 1:1 — EnvironmentId/SchemaName/Cloud), resolves the
    token scope via CopilotClient.ScopeFromSettings (never hardcoded), and bridges its streaming Bot Framework
    activity API (StartConversationAsync / AskQuestionAsync → IAsyncEnumerable<IActivity>) into an
    IChatClient (new CopilotStudioChatClient), wrapped in a MAF ChatClientAgent and handed to the existing
    FromAgent seam — unchanged. redteam --sut copilot-studio's consent gate, config validation, and every
    existing credential-free test still run and pass before any of this is reached; construction itself makes no
    network call (the token callback is invoked lazily by CopilotClient on the first real request).
  • CopilotStudioTokenProvider — MSAL device-code auth (IPublicClientApplication.AcquireTokenWithDeviceCode)
    with a persisted, OS-encrypted token cache (Microsoft.Identity.Client.Extensions.Msal — DPAPI on Windows,
    Keychain on macOS, libsecret on Linux) keyed by a SHA-256 hash of TenantId|AppClientId, silent-acquisition-first
    (AcquireTokenSilent) with device-code fallback on MsalUiRequiredException. New code — no prior token-caching
    precedent existed in this repo.
  • CopilotStudioConfig.Cloud now resolves to the real PowerPlatformCloud enum (ResolveCloud() /
    Validate()), verified against the actual restored Microsoft.Agents.CopilotStudio.Client 1.3.171-beta package
    (not the higher version number a prior planning doc assumed — see the deviation note below). Case-insensitive,
    defaults to Prod when omitted, and a typo'd/unrecognized value now fails config validation with a clear error
    listing the valid names, before any network call.
  • ICopilotStudioConversationClient — an AgentEval-owned abstraction over the two CopilotClient members the
    chat-client shim needs. The real package does not publicly export a mockable ICopilotClient interface (an
    earlier decompilation-based design note assumed one existed), so this repo defines its own seam instead —
    this is also what makes CopilotStudioChatClient unit-testable without live credentials.
  • SingleNameHttpClientFactory — a minimal IHttpClientFactory for the one named client CopilotClient
    requires, avoiding a full Microsoft.Extensions.Http + ServiceCollection registration for a CLI with no
    ambient DI container.
  • New package references (AgentEval.Cli, centrally pinned in Directory.Packages.props):
    Microsoft.Agents.CopilotStudio.Client 1.3.171-beta, Microsoft.Agents.Core 1.3.171-beta,
    Microsoft.Identity.Client 4.84.2, Microsoft.Identity.Client.Extensions.Msal 4.84.2,
    Microsoft.Extensions.Http 10.0.8 (raised Microsoft.Extensions.DependencyInjection's central floor to 10.0.8
    to match).

Deviations from the design doc (strategy/CopilotStudio/Bench-Eval-Integration-and-Live-Connector-Plan.md, local-only)

  • The doc cites Microsoft.Agents.CopilotStudio.Client "v1.6.150 — latest stable" and a decompiled ICopilotClient
    interface implemented by CopilotClient. Neither matches what actually restores from nuget.org: the real latest
    is 1.3.171-beta, and reflecting on that exact assembly shows CopilotClient implements no interface at
    all
    (GetInterfaces() returns empty) — its full public surface is narrower than the doc's decompilation notes
    assumed. This CHANGELOG entry and the code's own XML docs are the corrected record; ICopilotStudioConversationClient
    above is the concrete consequence.
  • --max-credits enforcement (the doc's Track 1 item 6) is not implemented — the SDK's activity/response
    models expose no Copilot Credit cost field to enforce against, so --max-credits remains parsed-but-unenforced
    exactly as before (ExitCodes.BudgetExceeded stays reserved, unused).

Not independently live-verified — needs a real Entra app registration + non-prod Copilot Studio agent

  • The MSAL device-code prompt, silent-refresh, and persisted-cache round-trip (CopilotStudioTokenProvider.GetTokenAsync).
  • Whether a real agent's StartConversationAsync/AskQuestionAsync activity stream matches the shape
    CopilotStudioChatClient assumes (in particular, any non-message activity worth surfacing, and real
    multi-activity turns).
  • The end-to-end network round trip (real HTTP call, real response parsing) against a live MCS agent.
  • A gated, Skip-by-default manual test (CopilotStudioLiveConnectorManualTests, tests/AgentEval.Tests/Cli/CopilotStudio/)
    is ready to run once credentials exist — see its XML doc for setup.

Gatekeeper — MonetaryLimitGate + PerToolCallBudgetGate (focused deterministic siblings of RunBudgetGate)

Added

  • MonetaryLimitGate (AgentEval.MAF.Gatekeeper.Gates) — a dedicated tool gate capping the running sum of a
    monetary tool-call argument (e.g. "amount") across a run, off the shared RunLedger. The economic sibling of
    RunBudgetGate, scoped to a single argument/cap pair with its own PolicyName in the evidence trail — for
    payment/refund/transfer-style tools without wiring RunBudgetGate's combined total/per-tool/monetary
    constructor. Fails closed on an unparseable amount, clamps a negative amount to zero (can't manufacture
    headroom), and the block reason never echoes the attempted amount or running sum — only the argument name and
    the configured cap, matching the taint-tracking gate's discipline of never leaking sensitive values into trace
    evidence.
  • PerToolCallBudgetGate (AgentEval.MAF.Gatekeeper.Gates) — a dedicated tool gate capping how many times
    specific tools may be called in one run (e.g. ["delete_account"] = 1, ["send_email"] = 3), off the shared
    RunLedger. Blunts spray/loop attacks — an injected instruction that tries to fire the same destructive tool
    repeatedly is stopped at the configured count regardless of phrasing. A tool not named in the caps is
    unconditionally allowed.
  • RunLedger.TryAdmitMonetary / TryAdmitPerToolCall — new atomic, per-dimension ledger primitives backing
    the two gates above. Deliberately isolated from RunBudgetGate's own TryAdmitToolCall bookkeeping (which
    always bumps its shared per-tool/total counters on any admit, even for a dimension the caller didn't ask it to
    check) — so composing either dedicated gate with RunBudgetGate, or with each other, over an overlapping
    tool/argument name can never cross-contaminate a count. Covered by a regression test proving the isolation holds
    even when RunBudgetGate and PerToolCallBudgetGate are stacked on the same tool.
  • samples/AgentEval.Samples/Gatekeeper/09_GatekeeperMonetaryAndPerCallBudget.cs — a live sample (real Azure
    OpenAI agent, no scripted fakes) with three scenes: a 10-call refund spray capped at 3 by PerToolCallBudgetGate,
    a single $50,000 refund blocked by a $1,000 MonetaryLimitGate cap, and both gates stacked against a $300 ×
    10-order spray — success is keyed on the actual recorded gate.tool.* block count, never on "no exception
    thrown." Wired into the samples menu (Group J).
  • Extracted AmountArgumentParser (shared by RunBudgetGate and MonetaryLimitGate) so the two gates parse a
    monetary argument (decimal / double / int / long / JsonElement / numeric string) identically —
    behavior-preserving refactor of RunBudgetGate's previously-private parsing logic, no functional change.

MAF Agent Skills evaluation — Phase 1 (assertions + progressive-disclosure efficiency metric)

Added

  • Five fluent skill assertions in AgentEval.Assertions.SkillUsageAssertions — HaveLoadedSkill,
    HaveReadSkillResource, HaveRunSkillScript, NotHaveRunSkillScript, HaveDisclosedProgressively —
    thin, additive extension methods over the existing ToolUsageAssertions / ToolCallAssertion (zero
    new MAF-type coupling in AgentEval.Core, which still does not reference Microsoft.Agents.AI).
    Support value-based argument matching (skill/resource/script name), not just tool-name matching, and
    degrade gracefully (key-agnostic fallback) if a future MAF version renames an argument.
  • SkillDisclosureEfficiencyMetric (code_skill_disclosure_efficiency, AgentEval.Metrics.Agentic)
    — a free, code-based IAgenticMetric scoring the load_skill → read_skill_resource →
    run_skill_script progressive-disclosure funnel as a weighted product of disclosure-order validity,
    load precision (redundant-load + "load storm" penalties), and an optional load-selection F1 when the
    caller supplies expected_skills ground truth. Never fabricates a selection score when no ground
    truth is supplied, and never fabricates an "advertise" stage count (the skill-inventory system-prompt
    listing is not a tool call and is not observable from a ToolUsageReport).
  • SkillToolNames (AgentEval.Skills) — the single shared constant for the three stable GA tool
    names (load_skill / read_skill_resource / run_skill_script) and their argument parameter names,
    referenced by the assertions and the metric.
  • samples/AgentEval.AgentSkillsEval — a live sample: a real ChatClientAgent against Azure OpenAI,
    wrapped with a real Microsoft.Agents.AI.AgentSkillsProvider over a real file-based
    expense-report skill fixture (SKILL.md + a resource + an in-process script). Three runs demonstrate
    different real assertion/metric/output combinations (read-only lookup, script-computed overage,
    and an off-topic task that both scores a vacuous 100/100 and shows an assertion's real failure
    path) — all keyed on the actual captured tool-call trace, never a bare success claim.
  • Verified four MAF AgentSkillsProvider API details against the live Microsoft.Agents.AI 1.13.0
    assembly (exact tool argument parameter names; the DisableCaching builder shape; that there is no
    provider-level GetSkillsAsync convenience; and that read_skill_resource's resourceName is a
    logical name resolved against the skill's discovered resource list, not a live filesystem path).

This is Phase 1 of a multi-phase design
(strategy/FutureFeatures/Skills/AgentEval-AgentSkills-Evals-Design-and-Plan.md, local-only).

MAF Agent Skills evaluation — Phase 3 (skill-description-injection red-team + run_skill_script governance)

Added

  • SkillInjectionAttack (AgentEval.RedTeam.Attacks, OWASP LLM01) — the 14th built-in red-team attack
    (roster 13→14, probes 258→264). Two new InjectionSurface values, SkillInstruction (a malicious
    skill's description/instructions, spliced into the SYSTEM PROMPT via {skills} — a higher-trust
    position than a retrieved document) and SkillResource (read_skill_resource output). 100% reuse of
    the shipped Wave-B machinery (CanaryTool, FidelityCompositeEvaluator, ToolInvocationEvaluator,
    RefusalGatedEvaluator) — canary "source" tools are named load_skill/read_skill_resource (matching
    MAF's real tool names) so an instrumented SUT's trace is indistinguishable from a real
    AgentSkillsProvider interaction. 6 probes at Comprehensive intensity; registered in Attack.All,
    ByName, ByOwaspId("LLM01").
  • ⚠️ HONESTY FINDING — the reused judge does NOT converge on the skill-description surface. Per the
    design doc's own documented risk item, this session ran a LIVE calibration of the flagship
    IndirectInjectionRubric against a new both-directions gold set
    (AgentEval.Guardrails.Judges.Rubrics.SkillInjectionGoldSet, 52 skill-flavored cases) via
    GateCalibrationHarness. Result: decisive accuracy 88.5%, 4 missed attacks, 2 false alarms, κ=0.769
    vs. gold — IsInlineReady == false (the harness requires zero missed attacks by default). The rubric
    generalizes reasonably (beats the deterministic keyword baseline) but not well enough to promote inline
    on this NEW surface. Decision: shipped SHADOW-ONLY for the skill surface, per the design doc's own
    contingency — never promoted inline, documented in code, the live sample, and here. Authoring a
    dedicated SkillDescriptionInjectionRubric is deferred (the design doc's own +3–5 dev-day contingency
    line item).
  • SkillScriptExecutionGate (AgentEval.MAF.Gatekeeper, IToolGate, GateCost.PureCode,
    MinimumPolicy = ReplaceResult) — deterministic hard gate on run_skill_script: blocks a call whose
    script identifier is not on the allowlist. Value-based, key-agnostic matching (every string-shaped
    argument value, plus "/"-joined pairs, are candidates — never assumes a specific argument key); an
    unrecognized/missing script identifier fails closed. No calibration needed (deterministic).
  • SkillScriptApprovalGate (IToolApprovalGate) — auto-approves load_skill/read_skill_resource;
    escalates run_skill_script to a human UNLESS the script is on a per-script trust allowlist — finer
    grained than MAF's native ReadOnlyToolsAutoApprovalRule (tool-granularity only).
  • Composition-ordering honesty (design doc §6.2, verified live this session): MAF's skill tools
    require human approval BY DEFAULT, and that pause happens BEFORE the FICC seam — so
    SkillScriptExecutionGate never fires unless run_skill_script is first auto-approved at the MAF
    layer (Posture A). The live sample (Run 6) demonstrates this exact composition and confirms the gate
    — not the approval layer — is what blocks the call (gate.tool.* count = 1, real trace evidence).
    SkillResourcePathGate was NOT built (dropped per the design doc §3 — read_skill_resource's
    resourceName is a logical name with no traversal surface, confirmed in Phase 1).
  • Sample Runs 5–6 (samples/AgentEval.AgentSkillsEval) — live-verified against real Azure OpenAI
    this session
    : Run 5 (skill-injection attack) — the agent resisted (0 tool calls on an off-topic-safe
    prompt), and the shadow-only judge verdict is shown labeled advisory-only, never conflated with the
    real behavioral verdict. Run 6 (exec-gate demo, Posture A) — the agent DID call run_skill_script with
    an unlisted script, and SkillScriptExecutionGate deterministically blocked it (1 real gate.tool.*
    block), with the model falling back to computing the answer manually — the gate, not the approval
    layer, stopped the call.
  • ~50 new tests across SkillInjectionAttackTests, SkillScriptExecutionGateTests,
    SkillScriptApprovalGateTests, SkillInjectionGoldSetCalibrationTests (deterministic harness-mechanics
    proof), and the env-gated SkillInjectionGoldSetCalibrationLiveCheck (the live calibration check itself,
    AGENTEVAL_RUN_SKILLCAL=1).

MAF Agent Skills evaluation — Phase 2 (compliance scanner + coverage report)

Added

  • SkillComplianceValidator (AgentEval.Skills, AgentEval.Core — pure, MAF-free, no I/O) —
    validates a SkillManifest against the GA SKILL.md rules (name presence/length/charset/no
    consecutive hyphens/matches parent directory; description presence/length; compatibility length)
    plus AgentEval governance flags (ScriptRequiresGovernanceReview when a skill exposes scripts,
    ResourceFromUntrustedSource for MCP/Custom-sourced resources, AllowedToolsExperimental). Returns a
    SkillComplianceReport (findings + a stage-reachability coverage summary) whose IsCompliant flips
    only on a High-severity finding.
  • MafSkillScanner (AgentEval.MAF.Skills) — the one place that touches a live AgentSkill /
    AgentSkillsSource. Enumerates skills via the GA source-level GetSkillsAsync(context, ct), maps each
    to the pure SkillManifest DTO, and delegates to the validator. Honesty note: AgentFileSkill
    stores its discovered resources/scripts in private fields with no public getter (verified via
    reflection against the live MAF 1.13.0 assembly), so this scanner independently re-derives a
    file-sourced skill's resource/script inventory by walking its resources//scripts/ subdirectories on
    disk — the same convention MAF's own AgentFileSkillsSourceOptions uses. For non-file sources
    (in-memory/class/MCP/custom) there is no equivalent enumeration API, so ResourceNames/ScriptNames
    are honestly reported empty rather than guessed — a documented, real limitation, not hidden.
  • SkillComplianceReportRenderer — console/Markdown/JSON rendering, severity-sorted findings plus a
    coverage table.
  • Sample Run 4 (samples/AgentEval.AgentSkillsEval) — MafSkillScanner.ScanFileSkillsAsync over the
    real expense-report fixture; live-verified against real Azure OpenAI this session: 1 skill
    scanned, 1 resource + 1 script found on disk, ScriptRequiresGovernanceReview correctly flagged
    (Medium, pointing at Phase 3), IsCompliant == true.
  • 44 new tests (tests/AgentEval.Tests/Skills/*, tests/AgentEval.Tests/MAF/Skills/*) — every GA rule
    fires exactly once on a violating manifest and not on a clean one; coverage counts never fabricate an
    "advertise" stage; a regression guard locks in that an undetectable non-file script stays honestly
    unreported rather than silently "fixed" with a fabricated count.

MAF Agent Skills evaluation — Phase 4a/4b (Skill Health & Security Index + hash-pin drift detection) + cheap sugar

Added

  • SkillSecurityIndex (AgentEval.Skills, pure) — joins the three independently-produced skill
    quality signals (Phase 2 compliance, Phase 1 efficiency, Phase 3/4b security) into one composite 0-100
    index. Never fabricates a missing axis: the score is the mean of only the axes actually supplied,
    and SkillSecurityIndexResult.Explanation names exactly which axes were/weren't measured.
  • ManifestFingerprint/ManifestDriftDetector (AgentEval.Guardrails, pure, MAF-free) — a generic
    SHA-256 hash-pin-and-diff primitive, reusable for any model-visible artifact definition (a skill
    manifest here; an MCP tool schema in a future gate — same pattern, different artifact type).
  • SkillManifestPoisoningGate + SkillManifestBaseline (AgentEval.Skills) — deterministic
    trust-time drift detection for a rug-pulled skill (content silently changing after approval). No
    calibration debt (pure hashing). SkillManifestBaseline persists to JSON (capture → save → later load
    → compare → flag drift), mirroring the repo's existing RedTeam baseline/diff CI pattern, scoped to skills.
  • Cheap assertion sugar (design catalog §10.4): WithScriptArgument (asserts inside
    run_skill_script's nested arguments object), ForSkill (scopes a ToolUsageReport to one skill's
    calls when a run exercises multiple skills), HaveDisclosedEfficiently(minScore) (metric-backed,
    synchronous — the metric is CodeBased with no real async work), HaveCorrectlyDeclinedSkill (positive
    phrasing for "the agent correctly avoided this skill"). SkillContractAssertions.AssertSkillWellFormed
    — a zero-cost (no agent, no LLM) unit-test assertion wrapping the Phase 2 validator.
  • Sample Run 7 — live-verified against real Azure OpenAI this session: a real simulated rug-pull
    (mutating the expense-report skill's description) is correctly caught by the hash-pin drift check
    (Changed finding), and the composite Skill Security Index correctly joins the real Phase 2 compliance
    scan (85/100, one Medium finding) with the real Phase 3 behavioral outcome from Run 5 (Resisted → 60/100
    after the drift penalty), honestly reporting the Efficiency axis as n/a (not re-measured this run,
    never assumed perfect) — composite 72/100, 2/3 axes measured.
  • Phase 4c (expanded red-team surface — fuzzing, canary-skill honeypot, typosquat detection,
    load-storm-as-DoW) was NOT built this session
    — explicitly deprioritized per the design doc's own
    scoring (4a/4b are cheaper and higher-value) and the marathon session's remaining scope (Stages 2-5).
    Documented as deferred, not silently dropped — see strategy/TODO.md.
  • ~35 new tests. Full net8.0 suite green (7278/7279, 1 pre-existing skip).

Gatekeeper Tribunal — 4 more calibrated flagship judges + 2 overlooked-seam gates

Added

  • IntentActionMismatchJudge — compares the agent's NARRATED intent against its ACTUAL tool call,
    vetoes on divergence. 52-case gold set. Live-calibrated: 100% decisive accuracy, κ=1.000,
    IsInlineReady=true.
  • GoalHijackDriftJudge — detects the agent being steered off the user's original stated goal toward
    an injected objective (distinct from indirect-injection: asks "has direction drifted," not "does this
    content instruct"). 48-case gold set. Live-calibrated: 100% decisive accuracy, κ=1.000,
    IsInlineReady=true.
  • UngroundedClaimJudge — RAG faithfulness as a runtime gate: flags an answer claim unsupported by
    retrieved context. 48-case gold set (includes hedged-opinion hard-negatives). Live-calibrated: 100%
    decisive accuracy, κ=1.000, IsInlineReady=true.
  • HallucinatedCitationJudge — hybrid: a deterministic, zero-LLM-cost citation-existence check
    composed with a judge support-check, only spending a model call when the citation exists. 52-case gold
    set covering both failure modes (nonexistent source; real source that doesn't support the claim).
    Live-calibrated: 100% decisive accuracy, κ=1.000, IsInlineReady=true. Not an IJudgeRubric (a
    bespoke IChatGate), so not registered in the CLI bridge's judge:* axis registry — fully usable
    directly.
  • MemoryWritePoisoningGate — guards the memory/vector-store WRITE side (every other injection judge
    guards reads). Reuses IndirectInjectionRubric verbatim at this new seam per the design backlog's
    reuse-the-pattern guidance.
  • McpToolDescriptionPoisoningGate + McpToolDefinition — deterministic hash-pin-and-diff over an
    MCP tool's definition (name/description/schema), catching a rug-pull. Reuses the exact
    ManifestFingerprint/ManifestDriftDetector generic primitive built for Skills Phase 4b's
    SkillManifestPoisoningGate — confirming the design backlog's own "same pattern, different artifact
    type" prediction. Schema comparison recursively canonicalizes JSON key order (a reformatted-but-identical
    schema never false-alarms).
  • All three IJudgeRubric-based judges registered in JudgeAxisRegistry — live-verified via the CLI
    bridge this session: agenteval gatekeeper list-gates shows all three (judge:intent-action-mismatch,
    judge:goal-hijack-drift, judge:ungrounded-claim + their keyword baselines);
    agenteval gatekeeper calibrate --gate judge:goal-hijack-drift --certify against real Azure OpenAI wrote
    a real calibration certificate; agenteval gatekeeper inspect then correctly Allowed a benign case and
    Blocked an attack case, citing the certificate.
  • Deferred, explicitly NOT built this session: ToolArgumentGoalCoherenceJudge (needs the
    IToolApprovalGate timeout-routing design worked out) and CrescendoTrajectoryJudge (stateful — session
    store + running summary — explicitly flagged as the hardest of the six in the task scope; deferring it
    matches the task's own suggested fallback). See strategy/TODO.md for the honest accounting.
  • ~100 new tests (deterministic rubric/gate tests + 4 env-gated live calibration checks,
    AGENTEVAL_RUN_GATEKEEPER_CAL=1). Full net8.0 suite green (7330/7331, 1 pre-existing skip).

Copilot Studio — mock backend + Track 2 (shared --sut seam, PR 1)

Added

  • MockCopilotStudioConversationClient (test-only) — a realistic, reusable mock Copilot Studio
    backend (a test double for ICopilotStudioConversationClient, since no live Copilot Studio system is
    available in this environment). Supports scripted MULTI-TURN conversations (fluent builder, mirroring
    ScriptedChatClient's convention), a SERVER-ASSIGNED conversation id (matching real MCS session
    semantics), and configurable ERROR INJECTION (auth failure on start, a mid-conversation exception at a
    chosen turn — e.g. rate-limit-shaped — and a hang-until-cancelled mode for timeout testing). 7 tests
    proving the mock itself behaves realistically (session-state tracking, activity-type filtering, error
    propagation, honest "no scripted turn" default that never fabricates a blank success).
  • Track 2, PR 1 — the shared --sut seam (strategy/CopilotStudio/Bench-Eval-Integration-and-Live-Connector-Plan.md
    §3): ISutTarget/ISutTargetOptions/CommonTargetOptions/SutTargetResolver
    (src/AgentEval.Cli/Commands/Targets/ISutTarget.cs) — generalizes the already-shipped redteam --sut
    pattern so eval/bench can reach the same built-in targets, WITHOUT touching
    IRedTeamBuiltInTarget/RedTeamOptions/RedTeamCommand.cs. CopilotStudioRedTeamTarget gains ISutTarget
    via EXPLICIT interface implementation (same idiom as IEnumerable/IEnumerable<T>) — its existing
    IRedTeamBuiltInTarget members are byte-for-byte unchanged. A ValidateDrift contract test (theory,
    4 truth-table cases) proves IRedTeamBuiltInTarget.Validate and ISutTarget.Validate agree on
    accept/reject for every shared check (consent / config-required / max-credits ≥ 0) — the one real
    ongoing-sync risk the design doc calls out, since the two method bodies have no compiler-enforced sync.
    12 new tests. gatekeeper-demo deliberately stays redteam-only (needs an AgentTrace, which
    eval/bench have no use for) — only copilot-studio gets the shared treatment, per the design doc.
  • NOT built this session (explicitly deferred, documented honestly): Track 2 PR 2 (eval adoption)
    and PR 3 (bench Tier 1 owasp/mitre/nist adoption) — the shared types exist and are tested, but no
    CLI verb wires them in yet; P6 (reports & resilience: fidelity badging, agent-fingerprint drift,
    429 retry+resume), P3 (KnowledgeCanaryEvaluator, Crescendo/PAIR/TAP over the native channel), and P7
    (OSS polish, Entra app-reg script, NuGet packaging) were not started. See strategy/TODO.md for the
    honest accounting and what's next.
  • Full net8.0 suite green (see the final Stage 5 numbers in this file's next entry).

Documentation — Stage 5 pass (Agent Skills, Gatekeeper, Copilot Studio) + final build/test verification

Added

  • docs/agent-skills.md — new user-facing feature page for MAF Agent Skills evaluation (assertions,
    disclosure-efficiency metric, compliance scanner, skill-injection red-team + run_skill_script governance
    gates, Skill Health & Security Index, hash-pin drift detection). Previously this only existed at
    implementation-detail depth inside docs/architecture.md; that section now cross-links here. Linked from
    docs/index.md's Documentation table and Feature Highlights grid.
  • docs/redteam/copilot-studio.md — corrected a stale sentence that still said "until the connector ships"
    even though BuildLive has shipped since this doc was first written; documented the new
    MockCopilotStudioConversationClient test double and the not-yet-CLI-reachable shared ISutTarget/
    SutTargetResolver seam (Track 2 PR 1).

Verified

  • Full-solution dotnet build -c Release: 0 errors (66 pre-existing warnings, unrelated to this
    session's changes — nullable-reference-type test scaffolding and xUnit analyzer style suggestions).
  • Full net8.0 test suite (fresh build, not --no-build, per this repo's known multi-TFM stale-binary trap):
    7349 passed / 0 failed / 1 skipped (the skip is the pre-existing, intentionally gated
    CopilotStudioLiveConnectorManualTests — needs real Entra credentials this environment does not have) —
    no regressions from any of Stages 1–5.