🧪 AI Agent Testing Framework

Production-grade testing framework for AI/LLM-powered applications. Built to catch the failures that traditional QA frameworks miss — hallucinations, prompt drift, response inconsistency, and silent agent breakdowns.

---

🎯 The Problem

AI agents are shipping faster than anyone can test them. Traditional testing frameworks were built for deterministic software — they completely fail for systems where:

• The same input can produce different outputs
• Responses can be structurally correct but factually wrong
• Agents hallucinate confidently
• Workflows break silently with no error codes
• Prompt changes cause cascading behavioral shifts

This framework solves that.

---

✨ Features

Core Testing Modules

Module	What It Tests	Why It Matters
Hallucination Detector	Factual accuracy against ground truth	Catches confident lies before users see them
Prompt Consistency Analyzer	Response stability across repeated runs	Detects prompt drift and non-determinism
Response Validator	Schema compliance, safety, format rules	Ensures outputs meet defined contracts
Workflow Tester	Multi-step agent conversation flows	Catches silent mid-workflow failures

Playwright E2E Testing

Capability	Description
Chat UI Testing	Automated testing of AI chat interfaces
Streaming Response Validation	Verifies streaming LLM outputs render correctly
Multi-turn Conversation Flows	Tests complex conversational state management
Visual Regression	Catches UI drift in AI-powered dashboards
Performance Benchmarking	Measures response times and TTFB for LLM endpoints

---

🚀 Quick Start

Installation

git clone https://github.com/nlaky1/ai-agent-testing-framework.git
cd ai-agent-testing-framework
npm install
npx playwright install chromium

Run Tests

# Unit + Integration tests
npm test

# Playwright E2E tests
npm run test:playwright

# All tests
npm run test:all

# Playwright with browser visible
npm run test:playwright:headed

---

📖 Usage

1. Hallucination Detection

Detect when an AI agent generates factually incorrect information:

import { HallucinationDetector } from 'ai-agent-testing-framework';

const detector = new HallucinationDetector({
  threshold: 0.7,          // similarity threshold (0-1)
  method: 'token_overlap', // detection method
  caseSensitive: false
});

const result = detector.detect(
  "The Eiffel Tower is 324 meters tall and located in Berlin.",
  ["The Eiffel Tower is 324 meters tall.", "The Eiffel Tower is in Paris, France."]
);

console.log(result.is_hallucination);  // true — "Berlin" contradicts ground truth
console.log(result.confidence);         // 0.35 — low confidence = likely hallucination
console.log(result.flagged_segments);   // ["located in Berlin"] — exact problem area

2. Prompt Consistency Analysis

Test if your AI agent gives consistent responses to the same prompt:

import { PromptConsistencyAnalyzer } from 'ai-agent-testing-framework';

const analyzer = new PromptConsistencyAnalyzer({
  min_similarity: 0.75,
  runs: 5
});

const responses = [
  "Python is a high-level programming language.",
  "Python is a versatile, high-level language for programming.",
  "JavaScript is the most popular language.", // inconsistent!
  "Python is a high-level, general-purpose programming language.",
  "Python is a popular high-level programming language."
];

const result = analyzer.analyze(responses);

console.log(result.consistency_score);  // ~0.72 — below threshold
console.log(result.is_consistent);       // false
console.log(result.outliers);            // [2] — index of the inconsistent response

3. Response Validation

Validate AI outputs against defined contracts:

import { ResponseValidator } from 'ai-agent-testing-framework';

const validator = new ResponseValidator({
  schema: {
    type: 'object',
    required: ['answer', 'confidence', 'sources'],
    properties: {
      answer: { type: 'string', minLength: 10 },
      confidence: { type: 'number', min: 0, max: 1 },
      sources: { type: 'array', items: { type: 'string' } }
    }
  },
  safety_patterns: [
    /\b(kill|hack|exploit)\b/i  // block unsafe content
  ],
  max_length: 2000
});

const result = validator.validate({
  answer: "The capital of France is Paris.",
  confidence: 0.95,
  sources: ["wikipedia.org/wiki/Paris"]
});

console.log(result.is_valid);        // true
console.log(result.checks_passed);    // ['schema', 'safety', 'length']

4. Workflow Testing

Test multi-step AI agent conversations:

import { WorkflowTester } from 'ai-agent-testing-framework';

const tester = new WorkflowTester();

const workflow = {
  name: 'Customer Support Bot',
  steps: [
    {
      name: 'greeting',
      input: 'Hi, I need help with my order',
      expected_intent: 'support_request',
      validate: (response) => response.includes('order') || response.includes('help')
    },
    {
      name: 'order_lookup',
      input: 'Order #12345',
      expected_intent: 'order_query',
      validate: (response) => response.includes('12345')
    },
    {
      name: 'resolution',
      input: 'I want a refund',
      expected_intent: 'refund_request',
      validate: (response) => response.includes('refund') || response.includes('process')
    }
  ]
};

const mockAgent = async (input, context) => {
  // Your actual AI agent call goes here
  return await callYourAgent(input, { history: context });
};

const result = await tester.run(workflow, mockAgent);
console.log(result.passed);           // true/false
console.log(result.failed_steps);      // which steps broke
console.log(result.total_duration_ms); // performance tracking

5. Playwright E2E Tests

Test AI-powered web interfaces:

// e2e/chat-interface.spec.js
import { test, expect } from '@playwright/test';

test('AI chat responds within acceptable time', async ({ page }) => {
  await page.goto('/chat');
  
  await page.fill('[data-testid="chat-input"]', 'What is machine learning?');
  await page.click('[data-testid="send-button"]');
  
  // Wait for streaming response to complete
  const response = page.locator('[data-testid="ai-response"]').last();
  await expect(response).toBeVisible({ timeout: 10000 });
  
  // Validate response quality
  const text = await response.textContent();
  expect(text.length).toBeGreaterThan(50);
  expect(text.toLowerCase()).toContain('learning');
});

test('handles conversation context correctly', async ({ page }) => {
  await page.goto('/chat');
  
  // First message
  await page.fill('[data-testid="chat-input"]', 'My name is Nikhil');
  await page.click('[data-testid="send-button"]');
  await page.waitForSelector('[data-testid="ai-response"]');
  
  // Follow-up — agent should remember context
  await page.fill('[data-testid="chat-input"]', 'What is my name?');
  await page.click('[data-testid="send-button"]');
  
  const responses = page.locator('[data-testid="ai-response"]');
  const lastResponse = responses.last();
  await expect(lastResponse).toContainText('Nikhil');
});

---

🏗️ Architecture

ai-agent-testing-framework/
├── lib/                          # Core framework modules
│   ├── index.js                  # Main exports
│   ├── hallucination-detector.js # Factual accuracy testing
│   ├── prompt-consistency.js     # Response stability analysis
│   ├── response-validator.js     # Schema & safety validation
│   └── workflow-tester.js        # Multi-step conversation testing
├── tests/
│   ├── unit/                     # Unit tests (Jest)
│   │   ├── hallucination.test.js
│   │   ├── consistency.test.js
│   │   ├── validator.test.js
│   │   └── workflow.test.js
│   └── integration/              # Integration tests
│       └── full-pipeline.test.js
├── e2e/                          # Playwright E2E tests
│   ├── chat-interface.spec.js
│   ├── streaming-response.spec.js
│   └── ai-dashboard.spec.js
├── playwright.config.js          # Playwright configuration
├── jest.config.js                # Jest configuration
└── examples/                     # Usage examples
    ├── test-openai-agent.js
    └── test-custom-agent.js

---

🔧 Configuration

Jest (Unit/Integration)

// jest.config.js — already configured
export default {
  testEnvironment: 'node',
  transform: {},
  testMatch: ['**/tests/**/*.test.js']
};

Playwright (E2E)

// playwright.config.js — already configured  
export default {
  testDir: './e2e',
  timeout: 30000,
  use: {
    baseURL: process.env.BASE_URL || 'http://localhost:3000',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure'
  }
};

---

🤝 Use Cases

• AI Startups — Validate agent behavior before shipping to production
• Enterprise QA Teams — Add AI-specific test coverage to existing pipelines
• LLM Application Developers — Catch regressions when updating prompts or models
• MLOps Teams — Continuous quality monitoring for deployed AI systems
• Conversational AI Platforms — Test chatbot reliability at scale

---

📊 Roadmap

• LangChain integration for chain testing
• OpenAI function calling validation
• Automated prompt regression testing
• CI/CD pipeline templates (GitHub Actions, GitLab CI)
• Dashboard for test result visualization
• Multi-model comparison testing (GPT-4 vs Claude vs Gemini)
• Token usage and cost tracking per test run

---

📄 License

MIT License — see LICENSE for details.

---

👤 Author

Nikhil Laky
Senior SDET | AI Agent Reliability & Test Automation
GitHub • LinkedIn

---

Built because AI agents deserve the same testing rigor as any production software.