Skip to content

**Security, Responsible AI & MAF RC1** 🛡️🤖

Pre-release
Pre-release

Choose a tag to compare

View all tags
@joslat joslat released this 22 Feb 23:13
· 275 commits to main since this release
v0.4.0-beta
cffbebf

Major feature release: Red Team security scanning, Responsible AI metrics, Calibrated multi-model evaluation, MAF RC1 upgrade, and comprehensive tracing improvements. 42 commits, 2,191 tests Ă— 3 TFMs = 6,573 total tests passing.

⚠️ BREAKING CHANGES

  • MAF RC1 Upgrade - Upgraded from Microsoft.Agents.AI 1.0.0-preview.251110.2 to 1.0.0-rc1
    • Microsoft.Extensions.AI upgraded from 10.0.0 to 10.3.0
    • Microsoft.Extensions.AI.OpenAI upgraded from 10.0.0-preview.1.25559.3 to 10.3.0 (preview → stable)
    • Microsoft.Extensions.AI.Evaluation.Quality upgraded from 9.5.0 to 10.3.0
    • System.Numerics.Tensors bumped from 10.0.0 to 10.0.3 (transitive compatibility)
    • Event hierarchy fix: AgentResponseUpdateEvent now inherits WorkflowOutputEvent (critical switch restructuring in MAFWorkflowEventBridge)
    • Type renames: AgentThread → AgentSession, GetNewThread() → CreateSessionAsync() (sync → async)
    • Method renames: StreamAsync → RunStreamingAsync, AddFanInEdge → AddFanInBarrierEdge
    • Naming conflict resolved: using AgentResponse = AgentEval.Core.AgentResponse; alias in adapter files
    • ChatClientAgentOptions.Instructions → ChatOptions.Instructions across all samples (26 occurrences in 14 files)
    • Breaking change (MAF adapters only): Helper methods on MAFAgentAdapter and MAFIdentifiableAgentAdapter were renamed and made async: ResetThread() → ResetSessionAsync(), GetNewThread() → CreateSessionAsync(), and constructor parameter type AgentThread? → AgentSession?. Core evaluation interfaces (IEvaluableAgent, IStreamableAgent) are unchanged; only code that calls these helper methods directly must be updated.

Added

  • Red Team Security Testing Module - Comprehensive AI agent security evaluation
    • 9 attack types: PromptInjection, Jailbreak, PIILeakage (LLM02), SystemPromptExtraction (LLM07), IndirectInjection, ExcessiveAgency (LLM06), InsecureOutput (LLM05), InferenceAPIAbuse (LLM10), EncodingEvasion
    • 192 total probes across all attack categories (expanded InsecureOutput from 18→33)
    • 60% OWASP LLM Top 10 2025 coverage (6/10): LLM01, LLM02, LLM05, LLM06, LLM07, LLM10
    • 6 MITRE ATLAS techniques: AML.T0024, AML.T0037, AML.T0043, AML.T0045, AML.T0051, AML.T0054
    • 6 export formats: JSON, JUnit XML, SARIF (GitHub Security), Markdown, PDF, CSV
    • 4 compliance reports: OWASP, MITRE, SOC2, ISO27001
    • Fluent assertions: result.Should().HaveOverallScoreAbove(85)
    • Attack pipeline API: AttackPipeline.Create().WithAllAttacks().ScanAsync(agent)
    • Baseline comparison for CI/CD regression tracking
    • Real-time progress reporting with ScanProgress callback
    • Rich console output with emoji, colors, and detailed breakdowns
  • Responsible AI Metrics (AgentEval.Metrics.ResponsibleAI namespace)
    • ToxicityMetric - Pattern + LLM hybrid toxicity detection
    • BiasMetric - LLM-based bias detection with counterfactual testing
    • MisinformationMetric - Claim verification and calibration assessment
  • Calibrated Evaluator - Multi-model criteria-based evaluation with CalibratedEvaluator for consensus-driven scoring
  • CSV Export Format - New CsvExporter for Excel and business intelligence tools
  • Sample 23: Responsible AI - Toxicity, bias, misinformation metrics with counterfactual testing
  • Sample 24: Benchmark System - Performance, agentic, standard, and cost benchmarks with comparative analysis
  • SPDX License Identifiers - Added to all source and test files for compliance

Changed

  • Trace Record & Replay Improvements (9 improvements from comprehensive audit)
    • Added IsComplete property to TraceReplayingAgent for cleaner replay loops
    • Implemented RecordStreamingChunks conditional check — streaming chunks now only recorded when option is enabled
    • Wired up SanitizeToolResult in streaming recording — tool results are sanitized consistently
    • Implemented MaxTurns enforcement in ChatTraceRecorder — throws InvalidOperationException when limit reached
    • Fixed documentation API names across docs/tracing.md, docs/conversations.md, docs/workflows.md, and docs/adr/004-trace-recording-replay.md
    • Added cross-reference sections in docs/conversations.md and docs/workflows.md linking to tracing guide
    • Updated ADR-004 phase status to reflect current implementation state
    • Sample 13 Demos 3 & 4 rewritten from mocked to fully operational real AI workflows
    • Added 12 new tracing tests (Contains matching, Warn/Ignore mismatch, sanitization, MaxTurns)
  • Sample 13 Audit Fixes — fixed prompt display mismatch, added DelayMultiplier = 0.1 for fast workflow replay, removed unused System.Text.Json import, corrected Key Takeaways API names
  • docs/tracing.md Performance Baseline example fixed: Entries[0].Duration → Entries.First(e => e.Type == TraceEntryType.Response).DurationMs
  • Added ConfigureAwait(false) to MAF adapter async calls for reliability
  • Replaced Assert.True with Assert.Contains for improved test readability
  • Removed hardcoded version strings from documentation
Assets 2
Loading