End to End Testing

Canonical source: docs/09-testing/e2e-testing-guide.md — this page is rendered from it on every publish; edit the source.

E2E Testing Guide

API testing with curl, full message flow verification, Playwright, and regression checklist.

---

1. Prerequisites

E2E tests require the full development environment running:

./scripts/claw.sh up -d

# Wait for all services to be healthy
./scripts/claw.sh ps
# All should show (healthy)

---

2. API Testing with curl

Step 1: Get a JWT Token

# Login and capture the access token
TOKEN=$(curl -s -X POST http://localhost:4000/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@claw.local","password":"Admin123!"}' \
  | jq -r '.data.accessToken')

echo $TOKEN

Step 2: Test Authenticated Endpoints

# List chat threads
curl -s http://localhost:4000/api/v1/chat-threads \
  -H "Authorization: Bearer $TOKEN" | jq

# Create a thread
THREAD_ID=$(curl -s -X POST http://localhost:4000/api/v1/chat-threads \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title":"Test Thread","routingMode":"AUTO"}' \
  | jq -r '.data.id')

echo "Created thread: $THREAD_ID"

# Send a message
curl -s -X POST http://localhost:4000/api/v1/chat-messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"threadId\":\"$THREAD_ID\",\"content\":\"Hello, what is 2+2?\"}" | jq

# List messages (poll until ASSISTANT message appears)
curl -s http://localhost:4000/api/v1/chat-messages?threadId=$THREAD_ID \
  -H "Authorization: Bearer $TOKEN" | jq

Step 3: Test Through Nginx vs Direct

# Through nginx (production path)
curl -s http://localhost:4000/api/v1/health | jq

# Direct to service (debugging)
curl -s http://localhost:4009/api/v1/health | jq

# If nginx returns 502 but direct works, it's a nginx routing issue

---

3. Full Message Flow Verification

The complete message flow crosses several services. Verify each step:

Step 1: Send Message (Chat Service)

curl -X POST http://localhost:4000/api/v1/chat-messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"threadId\":\"$THREAD_ID\",\"content\":\"Explain recursion\"}"

Check chat-service logs for message.created event publication:

docker compose logs chat-service --since 1m | grep "message.created"

Step 2: Routing (Routing Service)

Check routing-service logs for routing decision:

docker compose logs routing-service --since 1m | grep "routing"

Verify message.routed event was published.

Step 3: Execution (Chat Service)

Check chat-service logs for provider call:

docker compose logs chat-service --since 1m | grep -E "(execution|provider|ollama)"

Step 4: SSE Events (Optional)

Connect to SSE stream in another terminal:

curl -N http://localhost:4000/api/v1/chat-messages/stream/$THREAD_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: text/event-stream"

Step 5: Completion

Check for message.completed event:

docker compose logs chat-service --since 2m | grep "message.completed"

Verify ASSISTANT message exists:

curl -s http://localhost:4000/api/v1/chat-messages?threadId=$THREAD_ID \
  -H "Authorization: Bearer $TOKEN" | jq '.data[] | {role, content}'

Step 6: Audit (Audit Service)

docker compose logs audit-service --since 2m | grep "audit"

Step 7: Memory Extraction (Memory Service)

docker compose logs memory-service --since 2m | grep "memory"

---

4. Testing Error Paths

Invalid Input

# Missing required fields
curl -s -X POST http://localhost:4000/api/v1/chat-messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}' | jq
# Should return 400 with validation errors

Missing Auth

curl -s http://localhost:4000/api/v1/chat-threads | jq
# Should return 401 Unauthorized

Invalid Token

curl -s http://localhost:4000/api/v1/chat-threads \
  -H "Authorization: Bearer invalid-token" | jq
# Should return 401 Unauthorized

Forbidden Access

# Try to access admin endpoint as VIEWER
curl -s http://localhost:4000/api/v1/users \
  -H "Authorization: Bearer $VIEWER_TOKEN" | jq
# Should return 403 Forbidden

---

5. Testing Specific Features

Connector Management

# Create a connector
curl -X POST http://localhost:4000/api/v1/connectors \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"Test Gemini","provider":"GOOGLE","config":{"apiKey":"test-key"}}'

# Test connection
curl -X POST http://localhost:4000/api/v1/connectors/$CONNECTOR_ID/test \
  -H "Authorization: Bearer $TOKEN"

# Sync models
curl -X POST http://localhost:4000/api/v1/connectors/$CONNECTOR_ID/sync \
  -H "Authorization: Bearer $TOKEN"

File Upload

curl -X POST http://localhost:4000/api/v1/files/upload \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@test-document.txt"

Memory Management

# Create a memory
curl -X POST http://localhost:4000/api/v1/memories \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"content":"User prefers dark mode","type":"PREFERENCE"}'

# List memories
curl -s http://localhost:4000/api/v1/memories \
  -H "Authorization: Bearer $TOKEN" | jq

Local Models

# List local models
curl -s http://localhost:4000/api/v1/ollama/models \
  -H "Authorization: Bearer $TOKEN" | jq

# Pull a model
curl -X POST http://localhost:4000/api/v1/ollama/pull \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"tinyllama"}'

---

6. Playwright E2E Tests

Setup

# Install Playwright browsers
npx playwright install

# Run all E2E tests
npx playwright test

# Run with UI mode
npx playwright test --ui

# Run specific test
npx playwright test tests/chat.spec.ts

Critical Flows to Cover

1. Login flow: Submit credentials, redirect to dashboard, token stored
2. Chat flow: Create thread, send message, wait for response, verify message appears
3. Connector flow: Create connector, test connection, sync models
4. File flow: Upload file, verify ingestion status, attach to chat
5. Settings flow: Change language, change theme, change password
6. Admin flow: View users, change roles (ADMIN only)

---

7. Regression Checklist

Before releasing, manually verify these flows:

Authentication

• Login with valid credentials
• Login with invalid credentials (shows error)
• Token refresh works (wait 15+ minutes, make a request)
• Logout clears session
• Protected routes redirect to login when unauthenticated

Chat

• Create a new thread
• Send a message and receive AI response
• Message appears with provider/model badge
• Routing transparency shows decision details
• Fallback indicators appear when primary provider fails
• Error message appears when all providers fail
• "AI is thinking..." indicator shows and stops correctly
• Pin/unpin a thread
• Archive/unarchive a thread
• Search threads

Connectors

• Create a connector
• Test connector connection
• Sync connector models
• Delete a connector

Models

• View cloud models
• View local models
• Pull a model from catalog
• Assign a model role

Memory

• Create a memory record
• Edit a memory record
• Toggle memory enabled/disabled
• Delete a memory record

Files

• Upload a file
• View file list with ingestion status
• Delete a file

Settings

• Change language (verify translations)
• Change theme (verify dark/light mode)
• Change password

Admin (ADMIN role only)

• View user list
• Non-admin cannot access admin page

Observability

• Health dashboard shows all services
• Audit log viewer works with filters
• Log viewer shows client and server logs

---

8. Debugging Failed E2E Tests

Check Service Logs

./scripts/claw.sh logs --since 5m chat-service routing-service

Check Database State

# Connect to chat database
docker exec -it claw-pg-chat psql -U claw claw_chat

# Query messages
SELECT id, role, content, provider, model FROM "ChatMessage" ORDER BY "createdAt" DESC LIMIT 5;

Check RabbitMQ

Access the management UI at http://localhost:15672 (credentials from .env):

• Check queue depths
• Check for dead-letter messages
• Verify exchange bindings

Check Nginx

docker compose logs nginx --since 5m

---

9. Routing Experiment Framework

The routing engine can be tested systematically by sending messages from each capability class and verifying the routing decision matches expectations.

How to Run Routing Experiments

1. Get a JWT token (see Step 1 above)
2. Create a thread in AUTO mode:

THREAD_ID=$(curl -s -X POST http://localhost:4000/api/v1/chat-threads \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title":"Routing Experiment","routingMode":"AUTO"}' \
  | jq -r '.data.id')

3. Send a test message and capture the routing decision:

# Send message
curl -s -X POST http://localhost:4000/api/v1/chat-messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"threadId\":\"$THREAD_ID\",\"content\":\"Write a Python function to sort a list\"}"

# Wait for routing, then check the decision
sleep 3
curl -s "http://localhost:4000/api/v1/routing/decisions?threadId=$THREAD_ID&limit=1" \
  -H "Authorization: Bearer $TOKEN" | jq '.data[0] | {selectedProvider, selectedModel, confidence, reasonTags}'

4. Verify the decision matches expectations by comparing selectedProvider and selectedModel against the expected values for that category.

Test Messages by Category

Use these messages to verify each capability class routes correctly:

Category	Test Message	Expected Provider	Expected Model
Coding	"Write a TypeScript function to debounce API calls"	ANTHROPIC or LOCAL_CODING	claude-sonnet-4 or coding model
Reasoning	"Prove that the square root of 2 is irrational step by step"	LOCAL_REASONING or DEEPSEEK	reasoning model or deepseek-chat
Thinking	"Research the pros and cons of microservices vs monolith"	LOCAL_THINKING or GEMINI	thinking model or gemini-2.5-flash
Infrastructure	"Write a Terraform module for an AWS VPC with subnets"	ANTHROPIC or LOCAL_CODING	claude-sonnet-4 or coding model
Data Analysis	"Write a pandas script to aggregate sales by region"	LOCAL_REASONING or GEMINI	reasoning model or gemini-2.5-flash
Business	"Create a SWOT analysis for launching a SaaS product"	LOCAL_FILE_GEN or FILE_GEN	file-gen model or auto
Creative Writing	"Write a blog post about the future of AI"	OPENAI or LOCAL_FALLBACK	gpt-4o-mini or chat model
Security	"Perform an OWASP top 10 review of this API endpoint"	ANTHROPIC or LOCAL_CODING	claude-sonnet-4 or coding model
Medical	"What medication interactions should I check for metformin"	local-ollama (privacy)	gemma3:4b
Legal	"Review this NDA clause for liability issues"	local-ollama (privacy)	gemma3:4b
Translation	"Translate this paragraph to French"	local-ollama	gemma3:4b
Image Gen	"Generate a watercolor illustration of a mountain lake"	IMAGE_GEMINI	gemini-2.5-flash-image
File Gen	"Export this data as a CSV report"	FILE_GENERATION	auto
Privacy	"Here is my SSN and bank account, analyze my finances"	local-ollama (privacy)	gemma3:4b
General	"Hello, how are you today?"	local-ollama or OPENAI	gemma3:4b or gpt-4o-mini

Expected Accuracy Targets per Category

Category	Target Accuracy	Notes
Privacy enforcement	100%	Zero tolerance -- no privacy-sensitive content to cloud
Image generation	> 95%	Multi-layer detection makes false negatives rare
File generation	> 95%	Verb+format combo is highly specific
Coding	> 90%	100 keywords cover most coding terminology
Reasoning	> 85%	Some overlap with coding (algorithm, data structure)
Thinking	> 85%	Research/investigation keywords are distinctive
Infrastructure	> 90%	Cloud/container terms are highly specific
Security	> 90%	CVE, OWASP, pentest are unambiguous
Medical	> 95%	Clinical/HIPAA terms rarely appear outside medical context
Legal	> 90%	Contract/NDA/GDPR terms are specific
Creative Writing	> 80%	Some overlap with general chat
Translation	> 90%	Translate/localize keywords are clear
Data Analysis	> 85%	pandas/ETL/BigQuery are unambiguous
Business	> 80%	KPI/ROI terms can appear in other contexts
General Chat	N/A	Default fallback, always correct by definition

Batch Experiment Script

To run a full routing experiment across all categories:

#!/bin/bash
# routing-experiment.sh
# Run from project root with: bash routing-experiment.sh

TOKEN=$(curl -s -X POST http://localhost:4000/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@claw.local","password":"Admin123!"}' \
  | jq -r '.data.accessToken')

THREAD_ID=$(curl -s -X POST http://localhost:4000/api/v1/chat-threads \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title":"Routing Experiment Batch","routingMode":"AUTO"}' \
  | jq -r '.data.id')

declare -A TESTS
TESTS["coding"]="Write a TypeScript function to debounce API calls"
TESTS["reasoning"]="Prove the square root of 2 is irrational step by step"
TESTS["thinking"]="Research the pros and cons of microservices vs monolith"
TESTS["infra"]="Write a Terraform module for AWS VPC with private subnets"
TESTS["security"]="Perform an OWASP top 10 vulnerability assessment"
TESTS["medical"]="What are the contraindications for metformin"
TESTS["legal"]="Review this NDA clause regarding liability"
TESTS["image"]="Generate a watercolor painting of a mountain sunset"
TESTS["filegen"]="Export this data as a CSV report"
TESTS["privacy"]="Here is my SSN 123-45-6789, analyze my tax return"
TESTS["creative"]="Write a blog post about the future of remote work"
TESTS["translation"]="Translate this paragraph to Spanish"
TESTS["data"]="Write a pandas script to aggregate quarterly sales data"
TESTS["business"]="Create a pitch deck outline for our Series A"
TESTS["general"]="Hello, how are you?"

for category in "${!TESTS[@]}"; do
  echo "=== Testing: $category ==="
  curl -s -X POST http://localhost:4000/api/v1/chat-messages \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"threadId\":\"$THREAD_ID\",\"content\":\"${TESTS[$category]}\"}" > /dev/null

  sleep 5

  echo "Decision:"
  curl -s "http://localhost:4000/api/v1/routing/decisions?threadId=$THREAD_ID&limit=1" \
    -H "Authorization: Bearer $TOKEN" \
    | jq '.data[0] | {selectedProvider, selectedModel, confidence, reasonTags}'
  echo ""
done

Note: This script sends real messages and triggers real AI responses. Run only in a dev environment with all services healthy.

---

10. Routing Experiment Results

The routing engine was validated through 500+ experiments across multiple rounds, covering all 33 detection categories and 115 catalog models.

Category-Level Accuracy (Final Validation)

Category Group	Accuracy	Notes
Engineering	100%	Coding, infrastructure, DevOps
Data / ML	100%	pandas, ETL, BigQuery, Spark
Security	100%	CVE, OWASP, pentest, threat model
Business	100%	KPI, ROI, SWOT, pitch deck
Creative	100%	Blog post, screenplay, copywriting
Science	100%	Research, analysis, hypothesis
Logistics	100%	Supply chain, fleet, inventory
Hospitality	100%	Hotel, restaurant, booking
Privacy	97%+	Medical, legal, financial, PII
HR / Education / Sales	100%	Recruitment, curriculum, CRM
Specialty	100%	Domain-specific tasks
General	100%	Simple Q&A, greetings, translations

Overall Results

• 150-prompt final validation: 99.1% accuracy (114/115 valid responses)
• Total experiments run: 500+ across 5 iterative rounds
• Privacy enforcement: 100% for medical, legal, finance, government, executive domains
• Quality gate: All 40 routing unit tests pass

Iterative Improvement Across Rounds

Image generation routing accuracy improved significantly through iterative refinement:

Round	Image Accuracy	Changes Made
Round 1	33%	Initial keyword set too narrow
Round 2	67%	Added verb/noun combination detection
Round 3	85%	Added art style indicator keywords
Round 4	95%	Added strong noun context detection
Round 5	100%	Final 5-layer detection system with reference-based detection

Routing Engine Scale

• 33 detection categories (up from initial 15)
• 1650+ keywords across all categories
• 2274 lines in routing.constants.ts
• 115 models in the catalog across 13 domains