EvalGate v0.1.0 - Deterministic LLM/RAG Evaluations for PR Checks

🚀 EvalGate v0.1.0 - Initial Release

EvalGate brings deterministic LLM/RAG evaluations directly to your pull requests! Run systematic checks on your AI system outputs with zero infrastructure setup.

✨ Key Features

• 🔍 Deterministic Evaluations: JSON schema validation, category matching, latency/cost budgets
• 📊 Regression Detection: Compare against main branch baseline to catch performance regressions
• 🤖 GitHub Integration: Automatic PR comments with detailed evaluation summaries
• 🔒 Privacy-First: Local-only evaluation by default (no telemetry)
• ⚡ Zero Infrastructure: Runs entirely in GitHub Actions using uvx

🛠️ What's Included

Evaluation Types

• Schema Validation: Ensure outputs match expected JSON structure
• Category Accuracy: Verify classification/labeling correctness
• Performance Budgets: Enforce latency and cost constraints
• Custom Evaluators: Extensible framework for domain-specific checks

GitHub Actions Integration

• Composite Action: Drop-in solution for any repository
• Sticky PR Comments: Single updating comment with evaluation results
• Configurable Gates: Set score thresholds and regression policies
• Workflow Flexibility: Use as action or integrate directly

📋 Quick Start

# .github/workflows/evalgate.yml
name: EvalGate
on: [pull_request]

jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      
      - name: Generate outputs
        run: python scripts/predict.py --in eval/fixtures --out .evalgate/outputs
      
      - uses: aotp-ventures/evalgate@v0.1.0
        with:
          config: .github/evalgate.yml

🎯 Perfect For

• LLM Applications: Validate structured outputs, classifications, summaries
• RAG Systems: Check retrieval accuracy, response quality, formatting
• AI APIs: Monitor latency, cost, and output consistency
• ML Pipelines: Ensure model outputs meet production requirements

📚 Documentation

• README - Complete setup guide
• CONTRIBUTING - Development guidelines
• PyPI Package - CLI installation

🔧 Technical Details

• Python 3.10+ required
• Dependencies: Minimal (typer, pydantic, jsonschema, rich)
• Installation: Available via PyPI (pip install evalgate) or uvx
• License: MIT

---

Ready to bring systematic evaluation to your AI development workflow?
Add EvalGate to your repository and never ship broken model outputs again! 🛡️