v0.3.0

🚀 EvalGate v0.3.0

EvalGate v0.3.0 brings significant new capabilities for evaluating AI systems, with powerful new evaluators, enhanced reporting, and comprehensive tooling for evaluation workflows.

🎯 New Evaluators

Advanced AI Evaluators
• Conversation Flow - Evaluate multi-turn conversation patterns and dialogue structure
• Tool Usage - Validate AI tool calling behavior against expected sequences
• Workflow DAG - Assess complex workflow execution paths and dependencies

Text & Content Analysis
• ROUGE/BLEU - Standard text similarity metrics for content comparison
• Embedding Similarity - Semantic similarity using vector embeddings
• Regex Patterns - Flexible pattern matching with configurable sources
• Required Fields - Ensure critical data fields are present
• Classification Metrics - Multi-label classification evaluation with comprehensive metrics

🛠️ Enhanced Tooling

CLI Improvements
• Fixture Generator - Auto-generate test fixtures with nested conversation support
• Baseline Updates - Streamlined baseline refresh workflow
• Enhanced Configuration - Flexible evaluator config with pattern field/path support

Development & Testing
• LLM Judge Caching - Cache responses to reduce API costs and improve performance
• Comprehensive Unit Tests - Full test coverage for all evaluators
• CI Integration - Automated testing in continuous integration

📊 Reporting & Integration

GitHub Integration
• Check Run Reporting - Native GitHub status checks for PR evaluation
• Action Result Exposure - Easy access to evaluation results in workflows

Enhanced Reports
• Interactive Markdown - Rich markdown reports with tables and visualizations
• Per-Evaluator Thresholds - Configurable minimum score requirements
• Cost & Performance Tracking - Built-in latency and budget monitoring

🏗️ Architecture Improvements

Core Enhancements
• Evaluator Registry - Clean plugin architecture for custom evaluators
• Type Safety - Enum-based evaluator types for better reliability
• Package Version Exposure - Programmatic access to version information
• Transcript Handling - Enhanced support for conversation transcripts

Full Changelog: v0.2.0...v0.3.0

v0.2.0: LLM Judge Evaluator

What's Changed

• feat: Add LLM as judge evaluator support by @danieltoshea in #1

New Contributors

• @danieltoshea made their first contribution in #1

Full Changelog: v0.1.0...v0.2.0

EvalGate v0.1.0 - Deterministic LLM/RAG Evaluations for PR Checks

🚀 EvalGate v0.1.0 - Initial Release

EvalGate brings deterministic LLM/RAG evaluations directly to your pull requests! Run systematic checks on your AI system outputs with zero infrastructure setup.

✨ Key Features

• 🔍 Deterministic Evaluations: JSON schema validation, category matching, latency/cost budgets
• 📊 Regression Detection: Compare against main branch baseline to catch performance regressions
• 🤖 GitHub Integration: Automatic PR comments with detailed evaluation summaries
• 🔒 Privacy-First: Local-only evaluation by default (no telemetry)
• ⚡ Zero Infrastructure: Runs entirely in GitHub Actions using uvx

🛠️ What's Included

Evaluation Types

• Schema Validation: Ensure outputs match expected JSON structure
• Category Accuracy: Verify classification/labeling correctness
• Performance Budgets: Enforce latency and cost constraints
• Custom Evaluators: Extensible framework for domain-specific checks

GitHub Actions Integration

• Composite Action: Drop-in solution for any repository
• Sticky PR Comments: Single updating comment with evaluation results
• Configurable Gates: Set score thresholds and regression policies
• Workflow Flexibility: Use as action or integrate directly

📋 Quick Start

# .github/workflows/evalgate.yml
name: EvalGate
on: [pull_request]

jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      
      - name: Generate outputs
        run: python scripts/predict.py --in eval/fixtures --out .evalgate/outputs
      
      - uses: aotp-ventures/evalgate@v0.1.0
        with:
          config: .github/evalgate.yml

🎯 Perfect For

• LLM Applications: Validate structured outputs, classifications, summaries
• RAG Systems: Check retrieval accuracy, response quality, formatting
• AI APIs: Monitor latency, cost, and output consistency
• ML Pipelines: Ensure model outputs meet production requirements

📚 Documentation

• README - Complete setup guide
• CONTRIBUTING - Development guidelines
• PyPI Package - CLI installation

🔧 Technical Details

• Python 3.10+ required
• Dependencies: Minimal (typer, pydantic, jsonschema, rich)
• Installation: Available via PyPI (pip install evalgate) or uvx
• License: MIT

---

Ready to bring systematic evaluation to your AI development workflow?
Add EvalGate to your repository and never ship broken model outputs again! 🛡️