Testing Plan

Gatewayz API Testing Plan

Part of: Testing Guide | See also: CM Unit Testing Plan, Test Coverage Audit, Features Acceptance Criteria

---

TL;DR — 250+ manual API test cases covering every feature area. Each test is a curl/Postman call with expected result. Use this for QA sessions and pre-release verification. For automated test status, see Test Coverage Audit.

---

Conformance testing against the Conceptual Model and FEATURES.md. Each section maps to a feature area. Tests are manual API calls (curl/Postman/httpie). Auth: Most endpoints need Authorization: Bearer <api_key>. Admin endpoints need an admin API key.

---

Prerequisites

• A running Gatewayz instance (local or staging)
• A regular user API key (referred to as $USER_KEY)
• An admin API key (referred to as $ADMIN_KEY)
• Base URL (referred to as $BASE, e.g. http://localhost:8000)
• Stripe test-mode keys configured (for payment tests)
• Redis running (for rate limit / circuit breaker tests)
• Supabase running (for DB-backed tests)

---

1. System Utilities & Ping

#	Test	Method	Endpoint	Expected
1.1	Root endpoint returns API info	GET	/	200, body contains version
1.2	Ping returns pong with uptime	GET	/ping	200, body contains pong and uptime
1.3	Ping stats	GET	/ping/stats	200, stats object
1.4	Velocity mode status	GET	/velocity-mode-status	200, returns error_rate, threshold, limits

---

2. Authentication

#	Test	Method	Endpoint	Auth	Expected
2.1	Login with valid Privy token	POST	/auth	Privy token in body	200, returns api_key + credits + subscription status
2.2	Login creates new user if none exists	POST	/auth	New Privy token	200, new user with $5 credits, 3-day trial
2.3	Login rate limit (11th attempt in 15 min)	POST	/auth	Privy token	429 after 10 attempts from same IP
2.4	Register rate limit (4th attempt in 1 hour)	POST	/auth/register	body	429 after 3 attempts from same IP
2.5	Auth health check	GET	/auth/health	None	200, contains DB/Redis/cache/timeout status
2.6	Invalid/expired Privy token rejected	POST	/auth	Bad token	401 or 403

---

3. Chat & Inference (Core)

3.1 OpenAI-Compatible Inference

#	Test	Method	Endpoint	Auth	Expected
3.1.1	Non-streaming chat completion	POST	/v1/chat/completions	$USER_KEY	200, OpenAI-format response with choices, usage
3.1.2	Streaming chat completion (SSE)	POST	/v1/chat/completions	$USER_KEY	200, SSE stream with data: {...} chunks ending in [DONE]
3.1.3	JSON mode (response_format: {"type":"json_object"})	POST	/v1/chat/completions	$USER_KEY	200, response content is valid JSON
3.1.4	Tool/function calling	POST	/v1/chat/completions	$USER_KEY	200, response contains tool_calls array
3.1.5	Logprobs enabled	POST	/v1/chat/completions	$USER_KEY	200, response contains logprobs field
3.1.6	Credits deducted after completion	POST	/v1/chat/completions	$USER_KEY	Balance decreases by token cost
3.1.7	Insufficient credits returns 402	POST	/v1/chat/completions	Key with 0 credits	402
3.1.8	Expired trial returns 402	POST	/v1/chat/completions	Expired trial key	402
3.1.9	Invalid model returns error	POST	/v1/chat/completions	$USER_KEY, model=nonexistent/model	400 or 404
3.1.10	Anonymous request (no auth) works with whitelisted model	POST	/v1/chat/completions	None	200, response from allowed model
3.1.11	Anonymous request with non-whitelisted model rejected	POST	/v1/chat/completions	None	403 or 401
3.1.12	Rate limit enforced (exceed RPM)	POST	/v1/chat/completions	$USER_KEY	429 after exceeding limit

3.2 Anthropic-Compatible Inference

#	Test	Method	Endpoint	Auth	Expected
3.2.1	Non-streaming Anthropic messages	POST	/v1/messages	$USER_KEY	200, Anthropic-format response with content, usage
3.2.2	Streaming Anthropic messages (SSE)	POST	/v1/messages	$USER_KEY	200, SSE stream in Anthropic event format

3.3 Model Aliasing & Routing

#	Test	Method	Endpoint	Auth	Expected
3.3.1	Short alias resolves (e.g. gpt-4o)	POST	/v1/chat/completions	$USER_KEY	200, model resolves to openai/gpt-4o
3.3.2	Canonical ID works (e.g. openai/gpt-4o)	POST	/v1/chat/completions	$USER_KEY	200
3.3.3	Provider failover on provider error	POST	/v1/chat/completions	$USER_KEY	200, served by fallback provider (verify via response metadata or activity log)

3.4 Intelligent Routing

#	Test	Method	Endpoint	Auth	Expected
3.4.1	General router balanced	POST	/v1/chat/completions	$USER_KEY, model=router:general:balanced	200, model selected automatically
3.4.2	General router quality	POST	/v1/chat/completions	$USER_KEY, model=router:general:quality	200
3.4.3	General router cost	POST	/v1/chat/completions	$USER_KEY, model=router:general:cost	200
3.4.4	General router latency	POST	/v1/chat/completions	$USER_KEY, model=router:general:latency	200
3.4.5	Code router auto	POST	/v1/chat/completions	$USER_KEY, model=router:code:auto	200, code-appropriate model selected
3.4.6	Code router quality	POST	/v1/chat/completions	$USER_KEY, model=router:code:quality	200
3.4.7	Code router price	POST	/v1/chat/completions	$USER_KEY, model=router:code:price	200
3.4.8	Code router agentic	POST	/v1/chat/completions	$USER_KEY, model=router:code:agentic	200

3.5 Chat Sessions & History

#	Test	Method	Endpoint	Auth	Expected
3.5.1	List chat sessions	GET	/v1/chat/sessions	$USER_KEY	200, array of sessions
3.5.2	Get session with messages	GET	/v1/chat/sessions/{session_id}	$USER_KEY	200, session + messages array
3.5.3	Save message to session	POST	/v1/chat/sessions/{session_id}/messages	$USER_KEY	200/201
3.5.4	Batch save messages	POST	/v1/chat/sessions/{session_id}/messages/batch	$USER_KEY	200/201
3.5.5	Full-text search across sessions	POST	/v1/chat/search	$USER_KEY	200, matching sessions
3.5.6	Update session metadata	PUT	/v1/chat/sessions/{session_id}	$USER_KEY	200
3.5.7	Delete session	DELETE	/v1/chat/sessions/{session_id}	$USER_KEY	200
3.5.8	Chat stats	GET	/v1/chat/stats	$USER_KEY	200, usage stats

3.6 Chat Feedback

#	Test	Method	Endpoint	Auth	Expected
3.6.1	Submit feedback	POST	/v1/chat/feedback	$USER_KEY	200/201
3.6.2	Get user feedback	GET	/v1/chat/feedback	$USER_KEY	200, array
3.6.3	Feedback stats	GET	/v1/chat/feedback/stats	$USER_KEY	200
3.6.4	Session-specific feedback	GET	/v1/chat/sessions/{session_id}/feedback	$USER_KEY	200
3.6.5	Update feedback	PUT	/v1/chat/feedback/{feedback_id}	$USER_KEY	200
3.6.6	Delete feedback	DELETE	/v1/chat/feedback/{feedback_id}	$USER_KEY	200

3.7 Chat Sharing

#	Test	Method	Endpoint	Auth	Expected
3.7.1	Create share link	POST	/v1/chat/share	$USER_KEY	200/201, returns token
3.7.2	List share links	GET	/v1/chat/share	$USER_KEY	200, array
3.7.3	Access shared chat (public)	GET	/v1/chat/share/{token}	None	200, chat content
3.7.4	Delete share link	DELETE	/v1/chat/share/{token}	$USER_KEY	200

3.8 Chat Metrics

#	Test	Method	Endpoint	Auth	Expected
3.8.1	Tokens per second per model	GET	/v1/chat/completions/metrics/tokens-per-second	None	200
3.8.2	Aggregate tokens per second	GET	/v1/chat/completions/metrics/tokens-per-second/all	None	200

---

4. Models & Catalog

4.1 Model Discovery

#	Test	Method	Endpoint	Auth	Expected
4.1.1	List all models	GET	/v1/models	None	200, array of models with id, name, provider_slug, context_length, pricing
4.1.2	Filter models by provider	GET	/v1/models?provider=fireworks	None	200, only fireworks models
4.1.3	Unique (deduplicated) models	GET	/v1/models/unique	None	200, no duplicate model IDs
4.1.4	Search models	GET	/v1/models/search?q=llama	None	200, matching models
4.1.5	Trending models	GET	/v1/models/trending	None	200, ranked models
4.1.6	Low-latency models	GET	/v1/models/low-latency	None	200
4.1.7	Model detail by provider/model	GET	/v1/models/{provider}/{model}	None	200, full model details
4.1.8	Compare model across providers	GET	/v1/models/{provider}/{model}/compare	None	200, comparison data
4.1.9	Models by developer	GET	/v1/models/{developer}	None	200
4.1.10	Batch compare	POST	/v1/models/batch-compare	None	200
4.1.11	Detailed model info	GET	/api/models/detail?model_id=...	None	200, enriched with HuggingFace data
4.1.12	Every model has pricing data	GET	/v1/models	None	Verify no model has null/zero pricing (conceptual model requirement)

4.2 Modelz (Canonical Registry)

#	Test	Method	Endpoint	Auth	Expected
4.2.1	Full registry	GET	/v1/modelz/models	None	200, all canonical models
4.2.2	All model IDs	GET	/v1/modelz/ids	None	200, array of IDs
4.2.3	Check model exists	GET	/v1/modelz/check/{model_id}	None	200 with data or 404

4.3 Providers

#	Test	Method	Endpoint	Auth	Expected
4.3.1	List all providers	GET	/v1/provider	None	200, providers with stats
4.3.2	Provider stats	GET	/v1/provider/{provider_name}/stats	None	200
4.3.3	Provider top models	GET	/v1/provider/{provider_name}/top-models	None	200
4.3.4	Available routers	GET	/v1/routers	None	200, includes code + general routers

4.4 Gateways

#	Test	Method	Endpoint	Auth	Expected
4.4.1	List gateways	GET	/v1/gateways	None	200, array with name, color, priority, site_url per gateway
4.4.2	Gateway statuses	GET	/v1/gateways/status	None	200
4.4.3	Gateway summary	GET	/v1/gateways/summary	None	200
4.4.4	Specific gateway stats	GET	/v1/gateway/{gateway}/stats	None	200

4.5 HuggingFace Integration

#	Test	Method	Endpoint	Auth	Expected
4.5.1	HuggingFace discovery	GET	/v1/huggingface/discovery	None	200
4.5.2	HuggingFace search	GET	/v1/huggingface/search?q=llama	None	200, results
4.5.3	Author models	GET	/v1/huggingface/author/{author}/models	None	200
4.5.4	Model details (downloads, likes)	GET	/v1/huggingface/models/{model_id}/details	None	200, contains downloads, likes, parameters
4.5.5	Model card	GET	/v1/huggingface/models/{model_id}/card	None	200
4.5.6	Model files	GET	/v1/huggingface/models/{model_id}/files	None	200

4.6 Model Health

#	Test	Method	Endpoint	Auth	Expected
4.6.1	All model health	GET	/v1/model-health	None	200
4.6.2	Model health stats	GET	/v1/model-health/stats	None	200
4.6.3	Provider-level health	GET	/v1/model-health/providers	None	200
4.6.4	Unhealthy models	GET	/v1/model-health/unhealthy	None	200
4.6.5	Specific model health	GET	/v1/model-health/{provider}/{model}	None	200
4.6.6	Provider health summary	GET	/v1/model-health/provider/{provider}/summary	None	200

4.7 Rankings

#	Test	Method	Endpoint	Auth	Expected
4.7.1	Model leaderboard	GET	/ranking/models	None	200, ranked models with trend data

4.8 Model Availability

#	Test	Method	Endpoint	Auth	Expected
4.8.1	All model availability	GET	/availability/models	None	200
4.8.2	Specific model availability	GET	/availability/model/{model_id}	None	200
4.8.3	Availability summary	GET	/availability/summary	None	200
4.8.4	Overall status	GET	/availability/status	None	200, operational or degraded
4.8.5	Quick availability check	GET	/availability/check/{model_id}	None	200
4.8.6	Fallback providers	GET	/availability/fallback/{model_id}	None	200
4.8.7	Best provider	GET	/availability/best/{model_id}	None	200

---

5. Circuit Breakers

#	Test	Method	Endpoint	Auth	Expected
5.1	List all circuit breaker states	GET	/circuit-breakers	None	200, per-provider states (CLOSED/OPEN/HALF_OPEN)
5.2	Specific provider breaker	GET	/circuit-breakers/{provider}	None	200, auto-creates if new provider
5.3	Reset specific breaker	POST	/circuit-breakers/{provider}/reset	None	200, state becomes CLOSED
5.4	Reset all breakers	POST	/circuit-breakers/reset-all	None	200

---

6. Code Router

#	Test	Method	Endpoint	Auth	Expected
6.1	Settings options	GET	/code-router/settings/options	None	200, modes + parameters
6.2	Tiers with benchmarks	GET	/code-router/tiers	None	200, tiered models with SWE-bench/HumanEval scores
6.3	Usage stats	GET	/code-router/stats	None	200
6.4	Test routing with prompt	POST	/code-router/test	None	200, selected model + rationale
6.5	Validate settings	POST	/code-router/settings/validate	None	200

---

7. General Router

#	Test	Method	Endpoint	Auth	Expected
7.1	Settings options	GET	/general-router/settings/options	None	200, strategies + model pools
7.2	Available models	GET	/general-router/models	None	200
7.3	Fallback models per mode	GET	/general-router/fallback-models	None	200, fallback chain
7.4	Usage stats	GET	/general-router/stats	None	200
7.5	Test routing with prompt	POST	/general-router/test	None	200, selected model + reasoning

---

8. Users (Self-Service)

8.1 Profile & Balance

#	Test	Method	Endpoint	Auth	Expected
8.1.1	Get profile	GET	/user/profile	$USER_KEY	200, email, username, credits, trial, plan
8.1.2	Update profile	PUT	/user/profile	$USER_KEY	200
8.1.3	Duplicate email rejected	PUT	/user/profile	$USER_KEY, existing email	400 or 409
8.1.4	Get balance	GET	/user/balance	$USER_KEY	200, purchased_credits + subscription_allowance
8.1.5	Get plan	GET	/user/plan	$USER_KEY	200
8.1.6	Plan entitlements	GET	/user/plan/entitlements	$USER_KEY	200
8.1.7	Plan usage vs limits	GET	/user/plan/usage	$USER_KEY	200
8.1.8	Daily limit	GET	/user/limit	$USER_KEY	200
8.1.9	Credit transactions	GET	/user/credit-transactions	$USER_KEY	200, array of transactions
8.1.10	Environment usage	GET	/user/environment-usage	$USER_KEY	200
8.1.11	User monitor	GET	/user/monitor	$USER_KEY	200
8.1.12	Cache settings	GET	/user/cache-settings	$USER_KEY	200

8.2 Activity

#	Test	Method	Endpoint	Auth	Expected
8.2.1	Activity stats (default 30 days)	GET	/user/activity/stats	$USER_KEY	200, total requests/tokens/spend, daily breakdown
8.2.2	Activity log paginated	GET	/user/activity/log?limit=10	$USER_KEY	200, max 10 entries

8.3 API Key Management

#	Test	Method	Endpoint	Auth	Expected
8.3.1	Create API key	POST	/user/api-keys	$USER_KEY	201, key in gw_{env}_* format
8.3.2	Key creation rate limit (11th in 1 hour)	POST	/user/api-keys	$USER_KEY	429 after 10
8.3.3	List API keys	GET	/user/api-keys	$USER_KEY	200, array with key strings
8.3.4	API key usage stats	GET	/user/api-keys/usage	$USER_KEY	200
8.3.5	API key audit logs	GET	/user/api-keys/audit-logs	$USER_KEY	200
8.3.6	Update API key	PUT	/user/api-keys/{key_id}	$USER_KEY	200
8.3.7	Delete API key	DELETE	/user/api-keys/{key_id}	$USER_KEY	200

8.4 Rate Limits (User)

#	Test	Method	Endpoint	Auth	Expected
8.4.1	Get rate limit config	GET	/user/rate-limits	$USER_KEY	200
8.4.2	Key-specific rate limit usage	GET	/user/rate-limits/usage/{key_id}	$USER_KEY	200
8.4.3	Update rate limit for key	PUT	/user/rate-limits/{key_id}	$USER_KEY	200
8.4.4	Bulk update rate limits	POST	/user/rate-limits/bulk-update	$USER_KEY	200

8.5 Plans & Trials

#	Test	Method	Endpoint	Auth	Expected
8.5.1	List plans	GET	/plans	None	200, plan tiers
8.5.2	Plan details	GET	/plans/{plan_id}	None	200
8.5.3	Subscription plans	GET	/subscription/plans	None	200
8.5.4	Trial status	GET	/trial/status	$USER_KEY	200, active/expired + days remaining

8.6 Referrals

#	Test	Method	Endpoint	Auth	Expected
8.6.1	Get referral code	GET	/referral/code	$USER_KEY	200
8.6.2	Generate referral code	POST	/referral/generate	$USER_KEY	200, new code
8.6.3	Referral stats	GET	/referral/stats	$USER_KEY	200, total referred, conversion, rewards
8.6.4	Validate referral code	POST	/referral/validate	$USER_KEY	200

---

9. Credits & Billing

9.1 Credit Balance & Transactions

#	Test	Method	Endpoint	Auth	Expected
9.1.1	Get credit balance	GET	/credits/balance	$ADMIN_KEY	200, purchased_credits + subscription_allowance
9.1.2	Credit summary	GET	/credits/summary	$ADMIN_KEY	200
9.1.3	Transaction history	GET	/credits/transactions	$ADMIN_KEY	200, paginated
9.1.4	Add credits	POST	/credits/add	$ADMIN_KEY	200, balance increases
9.1.5	Adjust credits	POST	/credits/adjust	$ADMIN_KEY	200
9.1.6	Bulk add credits	POST	/credits/bulk-add	$ADMIN_KEY	200
9.1.7	Refund credits	POST	/credits/refund	$ADMIN_KEY	200

9.2 Conceptual Model Conformance: Credit System

#	Test	Description	Expected
9.2.1	Pre-flight credit check	Send chat request with user who has 0 credits	402 returned WITHOUT provider call being made
9.2.2	Idempotent deduction	Send same request ID twice	Credits deducted only once
9.2.3	Subscription allowance used first	User with both subscription and purchased credits	Subscription allowance decreases first
9.2.4	Provider error auto-refund	Trigger a 5xx provider error	Credits refunded automatically

---

10. Coupons

#	Test	Method	Endpoint	Auth	Expected
10.1	Available coupons	GET	/coupons/available	$USER_KEY	200, global + user-specific coupons
10.2	Redemption history	GET	/coupons/history	$USER_KEY	200, array
10.3	Redeem valid coupon	POST	/coupons/redeem	$USER_KEY	200, credits added
10.4	Redeem expired coupon rejected	POST	/coupons/redeem	$USER_KEY, expired code	400
10.5	Redeem already-used coupon rejected	POST	/coupons/redeem	$USER_KEY, already redeemed	400
10.6	Redeem user-specific coupon by wrong user	POST	/coupons/redeem	Different user's key	400 or 403

---

11. Payments (Stripe)

#	Test	Method	Endpoint	Auth	Expected
11.1	Credit packages (public)	GET	/api/stripe/credit-packages	None	200, available packages with pricing
11.2	Create checkout session	POST	/api/stripe/checkout-session	$USER_KEY	200, Stripe checkout URL
11.3	Get checkout session status	GET	/api/stripe/checkout-session/{session_id}	$USER_KEY	200
11.4	Create payment intent	POST	/api/stripe/payment-intent	$USER_KEY	200
11.5	Payment history	GET	/api/stripe/payments	$USER_KEY	200, paginated
11.6	Get subscription	GET	/api/stripe/subscription	$USER_KEY	200
11.7	Subscription checkout	POST	/api/stripe/subscription-checkout	$USER_KEY	200
11.8	Webhook always returns 200	POST	/api/stripe/webhook	Stripe signature	200, even on processing errors

---

12. Health & Monitoring

12.1 System Health

#	Test	Method	Endpoint	Auth	Expected
12.1.1	Primary health check	GET	/health	None	200, version + status + timestamp
12.1.2	Quick health (minimal)	GET	/health/quick	None	200, static response
12.1.3	Railway health	GET	/health/railway	None	200, DB + Redis + providers
12.1.4	System health	GET	/health/system	None	200, memory + CPU + connections
12.1.5	Database health	GET	/health/database	None	200
12.1.6	All health checks	GET	/health/all	None	200, combined
12.1.7	Health summary	GET	/health/summary	None	200, scores
12.1.8	Uptime	GET	/health/uptime	None	200
12.1.9	Health insights	GET	/health/insights	None	200
12.1.10	Health dashboard	GET	/health/dashboard	None	200

12.2 Provider Health

#	Test	Method	Endpoint	Auth	Expected
12.2.1	All provider health scores	GET	/health/providers	None	200, scores 0-100 per provider
12.2.2	Single provider health	GET	/health/provider/{provider}	None	200
12.2.3	Provider stats	GET	/health/providers/stats	None	200
12.2.4	Provider uptime history	GET	/health/providers/uptime	None	200

12.3 Model Health

#	Test	Method	Endpoint	Auth	Expected
12.3.1	All model health	GET	/health/models	None	200
12.3.2	Single model health	GET	/health/model/{model_id}	None	200

12.4 Gateway Health

#	Test	Method	Endpoint	Auth	Expected
12.4.1	All gateway health	GET	/health/gateways	None	200
12.4.2	Gateway dashboard (HTML)	GET	/health/gateways/dashboard	None	200, HTML content
12.4.3	Gateway dashboard (JSON)	GET	/health/gateways/dashboard/data	None	200, JSON
12.4.4	Single gateway health	GET	/health/{gateway}	None	200

---

13. Metrics & Observability

#	Test	Method	Endpoint	Auth	Expected
13.1	Prometheus metrics	GET	/metrics	None	200, Prometheus text format
13.2	Parsed metrics	GET	/api/metrics/parsed	None	200, JSON with p50/p95/p99
13.3	Metrics status	GET	/api/metrics/status	None	200, live or synthetic mode
13.4	Metrics summary	GET	/api/metrics/summary	None	200
13.5	Metrics health	GET	/api/metrics/health	None	200
13.6	Monitoring health	GET	/api/monitoring/health	None	200
13.7	Real-time stats	GET	/api/monitoring/stats/realtime	None	200
13.8	Error rates	GET	/api/monitoring/error-rates	None	200
13.9	Cost analysis	GET	/api/monitoring/cost-analysis	None	200
13.10	Anomaly detection	GET	/api/monitoring/anomalies	None	200
13.11	Instrumentation health	GET	/api/instrumentation/health	None	200
13.12	Grafana datasource test	GET	/prometheus/datasource	None	200

---

14. Diagnostics

#	Test	Method	Endpoint	Auth	Expected
14.1	Concurrency stats	GET	/api/diagnostics/concurrency	None	200, active requests, queue depth, status
14.2	Provider timing	GET	/api/diagnostics/provider-timing	None	200, slow/very slow counts

---

15. Status

#	Test	Method	Endpoint	Auth	Expected
15.1	Overall status	GET	/v1/status/	None	200, operational or degraded
15.2	Detailed status	GET	/v1/status/detailed	None	200, concurrency + circuit breakers + infrastructure
15.3	Provider statuses	GET	/v1/status/providers	None	200
15.4	Model statuses	GET	/v1/status/models	None	200
15.5	Specific model status	GET	/v1/status/models/{provider}/{model_id}	None	200
15.6	Recent incidents	GET	/v1/status/incidents	None	200
15.7	Model uptime history	GET	/v1/status/uptime/{provider}/{model_id}	None	200
15.8	Search on status page	GET	/v1/status/search?q=llama	None	200
15.9	Status stats	GET	/v1/status/stats	None	200

---

16. Analytics

#	Test	Method	Endpoint	Auth	Expected
16.1	Send single event	POST	/v1/analytics/events	Optional	200 (always, even if services down)
16.2	Batch events	POST	/v1/analytics/batch	Optional	200
16.3	Start session	POST	/v1/analytics/session/start	Optional	200
16.4	Analytics cache	GET	/v1/analytics/cache	None	200
16.5	Analytics cache summary	GET	/v1/analytics/cache/summary	None	200

---

17. Error Monitoring

#	Test	Method	Endpoint	Auth	Expected
17.1	Autonomous monitor status	GET	/error-monitor/autonomous/status	None	200
17.2	Error monitor health	GET	/error-monitor/health	None	200
17.3	Dashboard data	GET	/error-monitor/dashboard	None	200
17.4	Recent errors	GET	/error-monitor/errors/recent	None	200
17.5	Critical errors	GET	/error-monitor/errors/critical	None	200
17.6	Fixable errors	GET	/error-monitor/errors/fixable	None	200
17.7	Error patterns	GET	/error-monitor/errors/patterns	None	200

---

18. Image Generation

#	Test	Method	Endpoint	Auth	Expected
18.1	Generate image	POST	/v1/images/generations	$USER_KEY	200, image data/URL
18.2	Credits deducted for image gen	POST	/v1/images/generations	$USER_KEY	Balance decreases
18.3	Insufficient credits rejected	POST	/v1/images/generations	Key with 0 credits	402

---

19. Audio Transcription

#	Test	Method	Endpoint	Auth	Expected
19.1	Transcribe audio file	POST	/v1/audio/transcriptions	$USER_KEY	200, transcription text
19.2	Transcribe base64 audio	POST	/v1/audio/transcriptions/base64	$USER_KEY	200

---

20. Server-Side Tools

#	Test	Method	Endpoint	Auth	Expected
20.1	List tools (public)	GET	/v1/tools	None	200, web_search + text_to_speech
20.2	Tool definitions	GET	/v1/tools/definitions	None	200, OpenAI function format
20.3	Get specific tool	GET	/v1/tools/web_search	None	200
20.4	Get nonexistent tool	GET	/v1/tools/fake_tool	None	404
20.5	Execute web search	POST	/v1/tools/execute	$USER_KEY	200, search results
20.6	Search augment	POST	/v1/tools/search/augment	Optional	200, formatted context

---

21. IP Allowlist

#	Test	Method	Endpoint	Auth	Expected
21.1	Create IP allowlist entry	POST	/api/admin/ip-whitelist	$ADMIN_KEY	201
21.2	List allowlist entries	GET	/api/admin/ip-whitelist	$ADMIN_KEY	200
21.3	Check IP	POST	/api/admin/ip-whitelist/check	$ADMIN_KEY	200, allowed or not
21.4	Update entry	PUT	/api/admin/ip-whitelist/{entry_id}	$ADMIN_KEY	200
21.5	Delete entry	DELETE	/api/admin/ip-whitelist/{entry_id}	$ADMIN_KEY	200

---

22. Partner Trials

#	Test	Method	Endpoint	Auth	Expected
22.1	Partner config (public)	GET	/partner-trials/config/{partner_code}	None	200, trial config
22.2	Check partner code	GET	/partner-trials/check/{code}	None	200 (always)
22.3	Partner trial status	GET	/partner-trials/status	$USER_KEY	200
22.4	Daily limit	GET	/partner-trials/daily-limit	$USER_KEY	200
22.5	Start partner trial	POST	/partner-trials/start	$USER_KEY	200

---

23. Notifications (User)

#	Test	Method	Endpoint	Auth	Expected
23.1	Get notification preferences	GET	/user/notifications/preferences	$USER_KEY	200
23.2	Send usage report	POST	/user/notifications/send-usage-report	$USER_KEY	200
23.3	Test notification	POST	/user/notifications/test	$USER_KEY	200

---

24. Admin (Key Endpoints)

Full admin testing is extensive (80+ endpoints). Below covers the critical paths.

24.1 Admin Auth

#	Test	Method	Endpoint	Auth	Expected
24.1.1	Non-admin key rejected	GET	/admin/users	$USER_KEY	403
24.1.2	Admin key accepted	GET	/admin/users	$ADMIN_KEY	200

24.2 User Management

#	Test	Method	Endpoint	Auth	Expected
24.2.1	List users	GET	/admin/users	$ADMIN_KEY	200, paginated
24.2.2	Get user by ID	GET	/admin/users/{user_id}	$ADMIN_KEY	200, full profile + keys
24.2.3	User by API key	GET	/admin/users/by-api-key?key=...	$ADMIN_KEY	200
24.2.4	User count	GET	/admin/users/count	$ADMIN_KEY	200, count
24.2.5	User growth	GET	/admin/users/growth	$ADMIN_KEY	200
24.2.6	User stats	GET	/admin/users/stats	$ADMIN_KEY	200

24.3 Credit Operations

#	Test	Method	Endpoint	Auth	Expected
24.3.1	Add credits	POST	/admin/add_credits	$ADMIN_KEY	200, credits added
24.3.2	Add credits over daily limit	POST	/admin/add_credits	$ADMIN_KEY	400 or 403
24.3.3	Credit transactions	GET	/admin/credit-transactions	$ADMIN_KEY	200
24.3.4	Set daily limit	POST	/admin/limit	$ADMIN_KEY	200
24.3.5	Assign plan	POST	/admin/assign-plan	$ADMIN_KEY	200

24.4 System Monitoring

#	Test	Method	Endpoint	Auth	Expected
24.4.1	System monitor	GET	/admin/monitor	$ADMIN_KEY	200, user counts + credit totals + API usage
24.4.2	Chat request monitoring	GET	/admin/monitoring/chat-requests	$ADMIN_KEY	200
24.4.3	Chat request summary	GET	/admin/monitoring/chat-requests/summary	$ADMIN_KEY	200

24.5 Cache Management

#	Test	Method	Endpoint	Auth	Expected
24.5.1	Cache status	GET	/admin/cache-status	$ADMIN_KEY	200
24.5.2	Refresh providers	POST	/admin/refresh-providers	$ADMIN_KEY	200
24.5.3	Clear all caches	POST	/admin/cache/clear	$ADMIN_KEY	200
24.5.4	Clear rate limit cache	POST	/admin/clear-rate-limit-cache	$ADMIN_KEY	200

24.6 Model Sync

#	Test	Method	Endpoint	Auth	Expected
24.6.1	Sync status	GET	/admin/model-sync/status	$ADMIN_KEY	200
24.6.2	Sync health	GET	/admin/model-sync/health	$ADMIN_KEY	200
24.6.3	Syncable providers	GET	/admin/model-sync/providers	None	200, 33 providers
24.6.4	Trigger incremental sync	POST	/admin/model-sync/trigger	$ADMIN_KEY	200

24.7 Roles (RBAC)

#	Test	Method	Endpoint	Auth	Expected
24.7.1	Get user role	GET	/admin/roles/{user_id}	$ADMIN_KEY	200
24.7.2	Update role	POST	/admin/roles/update	$ADMIN_KEY	200
24.7.3	List users by role	GET	/admin/roles/list/{role}	$ADMIN_KEY	200
24.7.4	Role permissions	GET	/admin/roles/permissions/{role}	$ADMIN_KEY	200
24.7.5	Role audit log	GET	/admin/roles/audit/log	$ADMIN_KEY	200

24.8 Trial Analytics

#	Test	Method	Endpoint	Auth	Expected
24.8.1	Trial analytics	GET	/admin/trial/analytics	$ADMIN_KEY	200
24.8.2	Trial users	GET	/admin/trial/users	$ADMIN_KEY	200
24.8.3	Domain analysis	GET	/admin/trial/domain-analysis	$ADMIN_KEY	200
24.8.4	Conversion funnel	GET	/admin/trial/conversion-funnel	$ADMIN_KEY	200

24.9 Downtime Tracking

#	Test	Method	Endpoint	Auth	Expected
24.9.1	List incidents	GET	/admin/downtime/incidents	$ADMIN_KEY	200
24.9.2	Ongoing incidents	GET	/admin/downtime/incidents/ongoing	$ADMIN_KEY	200
24.9.3	Downtime statistics	GET	/admin/downtime/statistics	$ADMIN_KEY	200

24.10 Coupons (Admin)

#	Test	Method	Endpoint	Auth	Expected
24.10.1	List coupons	GET	/admin/coupons	$ADMIN_KEY	200
24.10.2	Coupon analytics	GET	/admin/coupons/{coupon_id}/analytics	$ADMIN_KEY	200
24.10.3	Coupon stats overview	GET	/admin/coupons/stats/overview	$ADMIN_KEY	200

---

25. Conceptual Model Conformance Checks

These tests verify the system behaves as described in the conceptual model, beyond individual endpoint testing.

#	Area	Test	How to Verify
25.1	Drop-in Compatibility	OpenAI SDK works by changing base_url only	Use openai Python SDK with base_url=$BASE/v1, send chat completion
25.2	Drop-in Compatibility	Anthropic SDK works by changing base_url only	Use anthropic Python SDK with base_url=$BASE/v1, send message
25.3	Failover	Provider failure triggers automatic retry	Force primary provider to fail (circuit breaker), verify request succeeds via fallback
25.4	Failover	400 errors do NOT trigger failover	Send malformed request, verify single provider attempt
25.5	Circuit Breaker	5 consecutive failures opens breaker	Send 5 failing requests, verify breaker state is OPEN via GET /circuit-breakers/{provider}
25.6	Circuit Breaker	OPEN breaker recovers after 300s	Wait 5 min after OPEN, verify state transitions to HALF_OPEN
25.7	Rate Limiting	IP-level limits enforced	Exceed 300 RPM from same IP without auth
25.8	Rate Limiting	API key-level limits enforced	Exceed plan RPM with authenticated key
25.9	Rate Limiting	Anonymous limits are stricter	Compare anonymous vs authenticated rate limits
25.10	Rate Limiting	Redis down = in-memory fallback works	Stop Redis, verify rate limiting still functions
25.11	Caching	Catalog served from cache (fast)	Time GET /v1/models -- expect sub-100ms on cache hit
25.12	Caching	Auth cache reduces latency	Compare first auth vs subsequent auth latency (should be 1-5ms cached)
25.13	Pricing	High-value models blocked if pricing missing	Verify GPT-4, Claude, Gemini models all have explicit pricing (not default)
25.14	Credit Deduction	Subscription allowance used before purchased credits	User with both balances: make requests, verify subscription decreases first
25.15	New User	Gets $5 credits + 3-day trial	Register new user, verify balance and trial_end date
25.16	Trial Expiration	Expired trial can still access :free models	Expire a trial, send request with :free suffix model
25.17	API Key Security	Keys encrypted at rest (Fernet)	Verify raw DB value is encrypted, not plaintext
25.18	Streaming	SSE format correct	Verify stream has data: prefix, proper JSON chunks, ends with data: [DONE]
25.19	Model Resolution	Aliases resolve correctly	Test r1 -> deepseek/deepseek-r1, gpt-4o -> openai/gpt-4o
25.20	Health Always 200	Health endpoints never return non-200	Call /health when DB is down, verify still 200 with degraded body

---

Testing Priority Order

1. P0 - Core Inference: Sections 3.1, 3.2, 3.3 (chat completions, messages, aliasing)
2. P0 - Billing: Sections 9.1, 9.2 (credits deduction, pre-flight checks)
3. P0 - Auth: Section 2 (login, key validation)
4. P1 - Failover & Circuit Breakers: Sections 5, 25.3-25.6
5. P1 - Rate Limiting: Sections 8.4, 25.7-25.10
6. P1 - Model Catalog: Section 4.1 (model discovery, pricing verification)
7. P2 - User Self-Service: Sections 8.1-8.6
8. P2 - Payments: Section 11
9. P2 - Health & Monitoring: Section 12
10. P3 - Everything else: Routers, analytics, diagnostics, admin, etc.

---

Notes

• Status endpoints always return 200: Degradation is reported in the response body, not via HTTP status codes. Check body content, not just status.
• Admin auth has two mechanisms: Some monitoring endpoints use ADMIN_API_KEY env var (via get_admin_key), not the standard RBAC check. Test both paths.
• In-memory state: Circuit breakers, model availability, and error monitoring state are lost on restart. Factor this into test sequencing.
• No pagination on some endpoints: GET /admin/balance and coupon stats fetch all rows. Be aware of performance on large datasets.
• Activity log total field: Returns count of current page, not total in DB. Don't rely on it for pagination math.