Skip to content

Un-monolithing, cross-framework support, CLI tool, architecture improvements, rich output, expoerters, data loaders, benchmarking and more

Pre-release
Pre-release

Choose a tag to compare

View all tags
@joslat joslat released this 28 Feb 10:38
· 252 commits to main since this release
v0.5.1-beta
2b124c5

v0.5.0-beta

Added

  • Monolith Modularization (ADR-016) — Split single src/AgentEval project (~203 files, ~35K lines) into 6 internal sub-projects while shipping a single NuGet package. Resolves dependency coupling: non-MAF users no longer pull Microsoft.Agents.AI, non-RedTeam users no longer pull PdfSharp-MigraDoc. Compiler-enforced dependency direction: Abstractions → Core → DataLoaders/MAF/RedTeam → Umbrella.
    • AgentEval.Abstractions (~48 files) — Public contracts: IMetric, IEvaluableAgent, IStreamableAgent, models
    • AgentEval.Core (~63 files) — Implementations: metrics, assertions, tracing, comparison, DI registration
    • AgentEval.DataLoaders (~23 files) — Dataset loaders (JSON/JSONL/CSV/YAML), exporters, output formatting
    • AgentEval.MAF (7 files) — Microsoft Agent Framework integration (MAFAgentAdapter, MAFEvaluationHarness)
    • AgentEval.RedTeam (61 files) — Security scanning, attack types, compliance reporting, PDF export
    • AgentEval (umbrella) — Single NuGet package containing all 6 DLLs per TFM via TargetsForTfmSpecificBuildOutput
    • All sub-projects use RootNamespace=AgentEval — zero namespace changes, zero API surface changes
    • PrivateAssets="all" on umbrella ProjectReferences with explicit NuGet dependency declarations
    • InternalsVisibleTo on all sub-projects → AgentEval.Tests
    • Phase 0: Fixed 11 cross-cutting coupling anomalies before split
    • See ADR-016 for full rationale and alternatives considered
  • Cross-Framework IChatClient Support — Universal adapter pattern for evaluating any IChatClient-based AI agent regardless of underlying framework (Azure OpenAI, Ollama, Groq, LM Studio, Semantic Kernel, etc.):
    • IChatClient.AsEvaluableAgent() extension method — One-liner wrapping any IChatClient as IStreamableAgent for evaluation. Located in AgentEval.Core.ChatClientExtensions. Parallels .AsIChatClient() from Microsoft.Extensions.AI.
    • TestSummary.ToEvaluationReport() extension method — Bridges evaluation pipeline (TestSummary) to export pipeline (EvaluationReport for IResultExporter). Derives time boundaries from PerformanceMetrics, maps MetricResults to MetricScores, supports agentName/modelName/endpoint provenance, sets Category for JUnit XML grouping.
    • NuGetConsumer Semantic Kernel demo — Real SK with [KernelFunction] plugins (FlightPlugin.cs) evaluated by AgentEval via the AIFunctionFactory.Create() bridge pattern. 8-step demo: Kernel build → plugin registration → SK↔M.E.AI bridge → tool assertions → code metrics → LLM-as-judge → performance summary. Isolated project with Microsoft.SemanticKernel 1.72.0 and Azure.AI.OpenAI 2.7.0-beta.2. Located in samples/AgentEval.NuGetConsumer/.
    • Sample 27: Cross-Framework Evaluation — Universal IChatClient adapter pattern: IChatClient → AsEvaluableAgent() → evaluate → ToEvaluationReport() → export to Markdown.
    • Documentation — docs/cross-framework.md with capability table, SK bridge code example, NuGetConsumer link.
  • AgentEval CLI (agenteval eval) — Evaluate any OpenAI-compatible AI agent from the command line without writing C#. Supports all providers (OpenAI, Ollama, Groq, vLLM, LM Studio, Azure OpenAI, etc.) via the Chat Completions API standard. Features: 15 CLI options, 7 export formats (json, junit, xml, markdown, md, trx, csv), LLM-as-judge via --judge, system prompt from file, stderr progress reporting for Unix piping, and CI/CD exit codes (0=pass, 1=fail, 2=usage error, 3=runtime error). Packaged as a .NET tool (dotnet tool install AgentEval.Cli). Located in src/AgentEval.Cli/.
  • Dependency Injection architecture (ADR-006) — All core services registered via services.AddAgentEval(), services.AddAgentEvalDataLoaders(), services.AddAgentEvalRedTeam(), or services.AddAgentEvalAll(). Interface-first design: IStochasticRunner, IModelComparer, IStatisticsCalculator, IToolUsageExtractor, ISnapshotComparer, ISnapshotStore, and all exporters/loaders registered with appropriate lifetimes. Configurable via AgentEvalServiceOptions (lifetime, harness factory, logger factory). See AgentEvalServiceCollectionExtensions.
  • Rich Evaluation Output subsystem — Structured output formatting moved to AgentEval.DataLoaders/Output/ during modularization, contracts split to AgentEval.Abstractions/Output/:
    • TableFormatter — PrintTable(), PrintComparisonTable(), PrintPerformanceSummary(), PrintToolSummary() with dynamic column selection and ANSI variance color-coding.
    • StochasticResultExtensions — Fluent result.PrintTable("Metrics"), result.PrintSummary(), result.PrintPerformanceSummary(), result.PrintToolSummary(), result.ToTableString().
    • ComparisonResultExtensions — modelResults.PrintComparisonTable(), modelResults.ToComparisonTableString().
    • OutputOptions — 15+ toggle properties (ShowScore, ShowPassRate, ShowDuration, ShowTTFT, ShowTokens, ShowCost, ShowToolCalls, ShowConfidenceInterval, etc.) with Default, Minimal, Full static presets and fluent With() copy method.
    • VerbosityLevel enum (None/Summary/Detailed/Full), VerbositySettings, VerbosityConfiguration with environment variable support (AGENTEVAL_VERBOSITY, AGENTEVAL_SAVE_TRACES, AGENTEVAL_TRACE_DIR).
    • EvaluationOutputWriter — 4-mode writer (Summary/Detailed/Full/None) producing tool timelines, performance sections, metric sections, and full JSON trace to any TextWriter.
    • AgentEvalTestBase — xUnit test base class with automatic tracing, RecordResult(), SaveTrace(), CreateResult() fluent builder pattern (TestResultBuilder).
    • TimeTravelTrace — 22+ model classes for time-travel debugging (ExecutionStep, 13 StepType values, ToolCallStepData, AgentHandoffStepData, etc.).
    • TraceArtifactManager — SaveTestResult(), SaveTrace(), LoadTrace(), ListTraceFiles(), GetMostRecentTrace(), CleanupOldTraces().
  • Exporter registry and DI auto-discovery — Extensible exporter system with runtime registration:
    • IExporterRegistry interface (in Abstractions) — Register(), Get(), GetRequired(), GetAll(), GetRegisteredFormats(), Contains(), Remove(), Clear().
    • ExporterRegistry implementation — Thread-safe ConcurrentDictionary, pre-populated with 5 built-in exporters (JSON, JUnit XML, Markdown, TRX, CSV) via DI.
    • DI auto-discovery: custom IResultExporter services registered in DI are automatically picked up by the registry.
    • FormatName default interface member on IResultExporter for string-based lookup.
    • ResultExporterFactory — Static factory with Create(ExportFormat) and CreateFromExtension(string).
  • DataLoader factory and DI architecture — Extensible dataset loading with runtime registration:
    • IDatasetLoaderFactory interface (in Abstractions) — CreateFromExtension(), Create(), Register().
    • DefaultDatasetLoaderFactory implementation — Dictionary-based registry for .jsonl, .ndjson, .json, .csv, .tsv, .yaml, .yml. Constructor accepts IEnumerable<IDatasetLoader> for DI auto-discovery of custom loaders.
    • DatasetLoaderFactory refactored to static convenience façade delegating to DefaultDatasetLoaderFactory.
    • IsTrulyStreaming property on IDatasetLoader — distinguishes JSONL/CSV true streaming from JSON/YAML buffered loading.
    • .ndjson and .tsv file extension support added.
    • DatasetTestCaseBenchmarkExtensions — ToToolAccuracyTestCase() and ToTaskCompletionTestCase() bridging dataset test cases to benchmark types with required_params metadata mapping.
  • Benchmarking improvements — DI integration and multi-prompt support:
    • AgenticBenchmark now accepts IToolUsageExtractor? via DI (defaults to DefaultToolUsageExtractor.Instance for non-DI usage).
    • PerformanceBenchmark.RunLatencyBenchmarkAsync() gained multi-prompt overload (IEnumerable<string> prompts) to avoid server-side caching and produce more representative latency measurements.
    • AgenticBenchmarkOptions.AddDefaultCompletionCriteria — boolean controlling auto-appended standard criteria.
    • Throughput benchmark Task.Yield() fixes for both success and error paths preventing deadlocks with synchronous agents.
  • Extensibility framework — Plugin system and registry pattern for custom extensions:
    • IMetricRegistry — now DI-registered as singleton with auto-population from IMetric services.
    • IAgentEvalPlugin lifecycle interface — InitializeAsync(), OnBeforeEvaluationAsync(), OnAfterEvaluationAsync(), ShutdownAsync(), with PluginId, Name, Version, Dependencies.
    • IPluginContext — provides Metrics (IMetricRegistry), Logger, Configuration, GetConfig<T>().
    • IResultTransformer — Post-processing with Priority ordering for composable result pipelines.
    • See Sample 26 for custom metrics, exporters, loaders, and attack registration via DI.
  • Sample 22: Responsible AI — Toxicity, bias, misinformation metrics with counterfactual testing.
  • Sample 23: Benchmark System — JSONL-loaded benchmarks: tool accuracy, latency, cost analysis with DatasetTestCaseBenchmarkExtensions.
  • Sample 24: Calibrated Evaluator — Multi-model consensus evaluation with calibrated scoring.
  • Sample 25: Dataset Loaders — Multi-format dataset pipeline: JSONL, JSON, YAML, CSV with IDatasetLoaderFactory.
  • Sample 26: Extensibility — DI registries, custom metrics/exporters/loaders/attacks demonstrating all extension points.

Changed

  • Snapshot Evaluation comprehensive review (28+ fixes) — Major audit and hardening of the snapshot comparison and storage system:
    • Interfaces & DI: Added ISnapshotComparer and ISnapshotStore interfaces with DI registration (ADR-006 compliance). Added InternalsVisibleTo for test project access to internal helpers.
    • Security: Sanitized suffix parameter in GetSnapshotPath to prevent path traversal (CODE-22). Added basePath validation in SnapshotStore constructor (CODE-21). Fixed SanitizeFileName collision resistance with SHA256 hash suffix (CODE-17).
    • Correctness: Fixed JsonValueKind.Null handling in element comparison (CODE-12). Fixed boolean type guard treating True/False as compatible types (CODE-30). Fixed SemanticComparisonResult to store scrubbed values (CODE-33). Fixed ComputeSimpleSimilarity to split on all whitespace (CODE-32). Fixed CompareArrays to continue comparing after length mismatch (CODE-23). Fixed LoadAsync TOCTOU with try/catch pattern (CODE-26/35). Fixed GUID regex word boundaries (CODE-16). Fixed duration regex word boundaries to prevent false positives (CODE-15). Fixed field name passed as parameter through recursion (CODE-20/34).
    • Validation: Added SemanticThreshold [0.0, 1.0] range validation (CODE-31). Added null guards on Compare method (TEST-12).
    • New features: Added AllowExtraProperties option (CODE-6). Added Delete, ListSnapshots, and Count to SnapshotStore (CODE-9/18). Added epsilon-based floating-point comparison (CODE-10). Added CancellationToken support on all async methods (CODE-7).
    • Performance: Added RegexOptions.Compiled on all default patterns (CODE-13). Made JsonSerializerOptions static in SnapshotStore (CODE-14).
    • Testing: Expanded test coverage from 23 to 51+ tests. Moved tests from Benchmarks/ to Snapshots/ directory (TEST-1/7). Added thread safety documentation (CODE-19). Documentation aligned with code defaults and APIs.
  • Sample 27 simplified — Removed redundant MAF flight agent (Part B, ~350 lines) already demonstrated in Samples 2-3, 9-10, and NuGetConsumer. Now focused solely on the unique Universal IChatClient Adapter pattern.
  • Cross-framework documentation fixed — Fixed broken Semantic Kernel code example in docs/cross-framework.md (replaced non-existent AsChatClient() method with working AIFunctionFactory.Create() bridge pattern). Added NuGetConsumer SK demo link. Fixed capability table footnote.
  • README updated — Sample count corrected from 26 to 27 with Sample 27 row added. Test counts now use qualitative descriptions instead of hard-coded numbers. Added CLI, DI, and cross-framework to Key Features. Expanded documentation table.
  • Roadmap updated — Marked Red Team and CLI as shipped; added CLI Phase 2, MCP Server, Benchmark runner, and Verify.Xunit to "What's Next". Updated version history table through 0.6.0-beta.
  • System.CommandLine upgraded from 2.0.0-beta4 to 2.0.3 stable — Breaking API change: SetHandler → SetAction, IsRequired → Required, AddOption() → Options.Add(), AddAlias() → constructor aliases, root.InvokeAsync(args) → root.Parse(args) then parseResult.InvokeAsync(). Only affects the new CLI project; no existing code referenced System.CommandLine.
  • Test count increased from 6,573 to 7,345 — 2,435 (net8.0) + 2,455 (net9.0) + 2,455 (net10.0). New tests for DI service registration, snapshot evaluation improvements, CLI commands, cross-framework adapter, and export pipeline bridging.

Fixed

  • Streaming tool extraction for ChatClientAgentAdapter — InvokeStreamingAsync now yields ToolCallStarted and ToolCallCompleted chunks when the underlying IChatClient streams FunctionCallContent/FunctionResultContent. Previously, streaming evaluations via RunEvaluationStreamingAsync produced empty ToolUsageReport for all IChatClient-based agents. Non-streaming path was unaffected.
    ``
Assets 2
Loading