Chatrix 🧪 ✨

Chatrix is an agentic evaluation platform for conversational AI applications. This project enables automated testing and quality assessment of conversational agents through multi-scenario simulations and intelligent evaluation.

✨ Key Features

• 🤖 Multi-Agent Testing: Automated conversation simulation with testing agents.
• 📊 Intelligent Evaluation: AI-powered assessment of conversation quality and outcomes.
• 🎯 Scenario-Based Testing: Support for multiple test scenarios with custom prompts.
• 📈 Comprehensive Reporting: Detailed test results with success rates and analytics.

---

⚙️ Prerequisites

• OS: Linux, macOS, or Windows
• Python: 3.11+
• Services:
  • OpenAI-compatible LLM inference endpoints (OpenAI, OpenRouter, vLLM, Ollama, etc.)

---

🛠️ Installation

1. Clone the repository

   git clone https://github.com/taresh18/chatrix.git
   cd chatrix

2. Create a virtual environment (recommended)

   python -m venv venv
   source venv/bin/activate    # Linux/macOS
   # venv\Scripts\activate   # Windows

3. Install dependencies

   pip install -e .

4. Configure environment variables

   cp .env.example .env
   nano .env  # Add your API keys and endpoints

5. Set up test scenarios

   • Create test scenarios in test_suite/scenarios/
   • Each scenario needs: test_agent.txt, evaluation_agent.txt
   • Configure main agent prompt in test_suite/main_agent.txt

---

🏃 Running the Application

1. Configure your environment

   Ensure all API keys and endpoints are set in your .env file.

2. Run the test suite

   python -m chatrix.main

3. View results

   • Test results are saved to timestamped directories in outputs/
   • Each run generates a comprehensive JSON summary
   • Individual conversation transcripts are saved with evaluation details

---

🏗️ Project Structure

chatrix/
├── test_suite/
│   ├── main_agent.txt          # Main agent system prompt
│   └── scenarios/
│       ├── balance_inquiry/
│       │   ├── test_agent.txt      # Testing agent prompt
│       │   └── evaluation_agent.txt # Evaluation criteria
│       └── payment_confirmation/
│           ├── test_agent.txt
│           └── evaluation_agent.txt
├── chatrix/
│   ├── core/                   # Orchestration and agent factory
│   ├── agents/                 # Agent service implementations
│   ├── models/                 # Data models and schemas
│   └── utils/                  # Logger, settings, and utilities
├── outputs/                    # Test results and transcripts
├── .env.example                # Template for environment variables
├── pyproject.toml              # Project dependencies
├── .gitignore
└── README.md

---

🎯 Creating Test Scenarios

1. Create scenario directory

   mkdir test_suite/scenarios/your_scenario

2. Add testing agent prompt (test_agent.txt)

   • Define the user persona and objectives
   • Specify the conversation flow and expected outcomes

3. Add evaluation criteria (evaluation_agent.txt)

   • Define what constitutes a successful conversation
   • Specify evaluation criteria and outcome checks

4. Configure main agent (main_agent.txt)

   • Set the system prompt for the agent under test
   • This prompt is shared across all scenarios

---

📊 Understanding Results

Chatrix generates comprehensive test reports:

• Overall Result: PASS/FAIL based on scenario success rates
• Success Rate: Percentage of scenarios that passed evaluation
• Scenario Results: Individual pass/fail status for each test
• Conversation Transcripts: Full dialogue with evaluation details
• Outcome Checks: Detailed assessment of conversation quality

Example output:

Starting Chatrix Test Suite...
==================================================

TEST SUITE RESULTS
==================================================
Overall Result: PASS
Success Rate: 100.0%
Output Directory: outputs/test_run_20250907_193206

Summary:
   Total Scenarios: 2
   Passed: 2
   Failed: 0
   Errors: 0

Scenario Results:
   [PASS] balance_inquiry
      └─ Test Result: PASS
   [PASS] payment_confirmation
      └─ Test Result: PASS

---

📚 References

• Pydantic AI: https://github.com/pydantic/pydantic-ai

---

📜 License

This project is released under the Apache License 2.0. See the LICENSE file for details.