🚀 EvalGate v0.3.0
EvalGate v0.3.0 brings significant new capabilities for evaluating AI systems, with powerful new evaluators, enhanced reporting, and comprehensive tooling for evaluation workflows.
🎯 New Evaluators
Advanced AI Evaluators
• Conversation Flow - Evaluate multi-turn conversation patterns and dialogue structure
• Tool Usage - Validate AI tool calling behavior against expected sequences
• Workflow DAG - Assess complex workflow execution paths and dependencies
Text & Content Analysis
• ROUGE/BLEU - Standard text similarity metrics for content comparison
• Embedding Similarity - Semantic similarity using vector embeddings
• Regex Patterns - Flexible pattern matching with configurable sources
• Required Fields - Ensure critical data fields are present
• Classification Metrics - Multi-label classification evaluation with comprehensive metrics
🛠️ Enhanced Tooling
CLI Improvements
• Fixture Generator - Auto-generate test fixtures with nested conversation support
• Baseline Updates - Streamlined baseline refresh workflow
• Enhanced Configuration - Flexible evaluator config with pattern field/path support
Development & Testing
• LLM Judge Caching - Cache responses to reduce API costs and improve performance
• Comprehensive Unit Tests - Full test coverage for all evaluators
• CI Integration - Automated testing in continuous integration
📊 Reporting & Integration
GitHub Integration
• Check Run Reporting - Native GitHub status checks for PR evaluation
• Action Result Exposure - Easy access to evaluation results in workflows
Enhanced Reports
• Interactive Markdown - Rich markdown reports with tables and visualizations
• Per-Evaluator Thresholds - Configurable minimum score requirements
• Cost & Performance Tracking - Built-in latency and budget monitoring
🏗️ Architecture Improvements
Core Enhancements
• Evaluator Registry - Clean plugin architecture for custom evaluators
• Type Safety - Enum-based evaluator types for better reliability
• Package Version Exposure - Programmatic access to version information
• Transcript Handling - Enhanced support for conversation transcripts
Full Changelog: v0.2.0...v0.3.0