EvalForge 🔥

Black-Box evaluation of LLMs/RAGs via HTTP

EvalForge is a powerful Python CLI tool for evaluating Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems through HTTP APIs. It provides flexible evaluation metrics, async execution with concurrency control, and beautiful console output.

✨ Features

• 🎯 Multiple Evaluators

  • ExactMatch: Binary exact string matching
  • SemanticSimilarity: Cosine similarity using sentence-transformers (all-MiniLM-L6-v2 local)
  • Faithfulness: LLM-as-judge evaluation using Groq/Llama3-70B or Ollama/Qwen2.5

• 🔌 Universal HTTP Adapter

  • Compatible with n8n, Flowise, Dify, and any REST API
  • Jinja2 templates for request body generation
  • JSONPath extraction for response parsing

• ⚡ High Performance

  • Async execution with Semaphore for concurrency control
  • Automatic retries with Tenacity
  • Configurable timeouts and retry strategies

• 📊 Rich Output

  • Beautiful console output with Rich
  • JSON export for integration with other tools
  • Detailed failure analysis

🚀 Installation

# Clone the repository
git clone https://github.com/Sebastiangmz/Eval.git
cd Eval

# Install with pip
pip install -e .

# Or install with dependencies
pip install -e ".[dev]"

📖 Quick Start

1. Create a Configuration File

Create a YAML configuration file (e.g., config.yaml):

name: "My Evaluation"

target:
  url: "https://api.example.com/chat"
  method: "POST"
  headers:
    Content-Type: "application/json"
  timeout: 30.0

body_template: |
  {
    "question": "{{ question }}"
  }

response_map:
  answer: "$.response.answer"

evaluators:
  enabled:
    - exact_match
    - semantic_similarity

thresholds:
  exact_match: 1.0
  semantic_similarity: 0.8

test_cases:
  - id: "test_001"
    question: "What is the capital of France?"
    expected_answer: "Paris"

concurrency: 5
retry_attempts: 3

2. Run Evaluation

# Run evaluation
evalforge run config.yaml

# Save results to JSON
evalforge run config.yaml -o results.json

# Verbose output
evalforge run config.yaml -v

3. Validate Configuration

# Validate config without running tests
evalforge validate config.yaml

📝 Configuration Guide

Target Configuration

target:
  url: "https://api.example.com/endpoint"  # API endpoint
  method: "POST"                           # HTTP method (GET, POST, etc.)
  headers:                                 # Optional headers
    Content-Type: "application/json"
    Authorization: "Bearer TOKEN"
  timeout: 30.0                           # Request timeout in seconds

Body Template (Jinja2)

Use Jinja2 templating to generate request bodies:

body_template: |
  {
    "question": "{{ question }}",
    "session_id": "{{ id }}",
    "custom_field": "{{ metadata.custom }}"
  }

Available variables:

• question: Test case question
• id: Test case ID
• metadata: Test case metadata dictionary

Response Mapping (JSONPath)

Extract data from responses using JSONPath:

response_map:
  answer: "$.data.response"           # Required: path to answer
  context: "$.data.context"           # Optional: path to context

Common patterns:

• $.field - Direct field access
• $.nested.field - Nested field access
• $.array[0] - Array element access
• $.array[*] - All array elements

Evaluators

Exact Match

evaluators:
  enabled:
    - exact_match

thresholds:
  exact_match: 1.0  # Must be exactly 1.0

Semantic Similarity

evaluators:
  enabled:
    - semantic_similarity
  similarity_model: "all-MiniLM-L6-v2"  # or other sentence-transformer model

thresholds:
  semantic_similarity: 0.8  # 0.0 to 1.0

Faithfulness (LLM Judge)

evaluators:
  enabled:
    - faithfulness
  llm_judge_provider: "groq/llama3-70b-8192"  # or "ollama/qwen2.5"

thresholds:
  faithfulness: 0.7  # 0.0 to 1.0

Note: For faithfulness evaluation, you need:

• For Groq: Set GROQ_API_KEY environment variable
• For Ollama: Have Ollama running locally

Test Cases

test_cases:
  - id: "test_001"                    # Unique identifier
    question: "What is 2+2?"          # Question to ask
    expected_answer: "4"              # Expected answer
    expected_context: "Math context"  # Optional: for faithfulness
    metadata:                         # Optional: custom metadata
      category: "math"
      difficulty: "easy"

Concurrency & Retry

concurrency: 5        # Max concurrent requests
retry_attempts: 3     # Number of retries
retry_delay: 1.0      # Initial delay between retries (seconds)

🔗 Integration Examples

n8n Workflow

target:
  url: "http://localhost:5678/webhook/chatbot"
  method: "POST"

body_template: |
  {
    "chatInput": "{{ question }}",
    "sessionId": "{{ id }}"
  }

response_map:
  answer: "$.output.text"

Flowise

target:
  url: "http://localhost:3000/api/v1/prediction/{CHATFLOW_ID}"
  method: "POST"

body_template: |
  {
    "question": "{{ question }}"
  }

response_map:
  answer: "$.text"
  context: "$.sourceDocuments[*].pageContent"

Dify

target:
  url: "https://api.dify.ai/v1/chat-messages"
  method: "POST"
  headers:
    Authorization: "Bearer {API_KEY}"

body_template: |
  {
    "inputs": {},
    "query": "{{ question }}",
    "response_mode": "blocking",
    "user": "{{ id }}"
  }

response_map:
  answer: "$.answer"

🛠️ Technology Stack

• CLI: Typer - Modern CLI framework
• HTTP: HTTPX - Async HTTP client
• Models: Pydantic - Data validation
• LLM: LiteLLM - Unified LLM API
• Embeddings: Sentence-Transformers - Semantic similarity
• Output: Rich - Terminal formatting
• Templates: Jinja2 - Template engine
• JSONPath: jsonpath-ng - JSON querying
• Retry: Tenacity - Retry logic

📚 Examples

See the examples/ directory for complete configuration examples:

• simple_qa.yaml - Basic Q&A evaluation
• rag_evaluation.yaml - RAG system with faithfulness checking
• n8n_workflow.yaml - n8n webhook evaluation

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

MIT License - see LICENSE file for details.

🙏 Acknowledgments

Built with love using modern Python tools and libraries.