Skip to content

Testing Guide

Jump to bottom
Norm Brandinger edited this page Nov 20, 2025 · 1 revision

Testing Approach - Best Practices

Table of Contents

Overview

This document explains the best practices for running the 555+ tests in the devstack-core repository.

Test Architecture

Total Test Count: 555+ Tests (479+ run, 76 skipped)

  1. Bash Integration Tests (26+ suites, 174+ tests)

    • Infrastructure, databases, cache, messaging, applications
    • Security: AppRole authentication (21 tests), TLS connections (24 tests)
    • Performance: Redis cluster failover (16 tests)
    • Run directly on host using bash scripts

  2. Python Unit Tests (pytest, 254 tests: 178 passed + 76 skipped)

    • FastAPI application unit tests
    • Run inside Docker container
    • 84.39% code coverage (exceeds 80% target)

  3. Python Parity Tests (pytest, 64 tests from 38 unique test functions)

    • API implementation comparison tests
    • Run from host with uv
    • Some tests parametrized to run against both APIs

Best Approach (Implemented in run-all-tests.sh)

1. Bash Integration Tests

Method: Direct execution on host

./tests/test-vault.sh
./tests/test-postgres.sh
# ... etc

Why: These tests interact with Docker containers from outside, testing real service integration.

2. Python Unit Tests (FastAPI)

Method: Execute inside Docker container

docker exec dev-reference-api pytest tests/ -v

Why this is the BEST approach:

  • ✅ Correct Python version (3.11) - avoids Python 3.14 compatibility issues
  • ✅ All dependencies pre-installed - no local environment setup needed
  • ✅ Production-like environment - tests run in same env as production code
  • ✅ No native extension build issues - asyncpg, etc. already compiled
  • ✅ Consistent across developers - everyone uses same container image

Alternatives rejected:

  • ❌ Local Python 3.14: asyncpg build fails with C compilation errors
  • ❌ Local venv: Requires manual dependency management, version conflicts

3. Python Parity Tests

Method: Run from host using uv

cd reference-apps/shared/test-suite
uv venv && uv pip install -r requirements.txt && uv run pytest -v

Why this is the BEST approach:

  • ✅ Must access both APIs via localhost - localhost:8000 and localhost:8001
  • ✅ Client perspective testing - tests as external client would use APIs
  • ✅ Lightweight dependencies - only httpx, pytest (no heavy native extensions)
  • ✅ uv handles environment - automatic venv creation and dependency management

Alternatives rejected:

  • ❌ Inside container: Would need container networking, can't access localhost ports
  • ❌ Local Python 3.14 directly: Works but uv provides better isolation

Running All Tests

Simple (Recommended)

# Auto-starts required containers and runs all 555+ tests
./tests/run-all-tests.sh

The script:

  1. Checks if containers are running
  2. Auto-starts them if needed (docker compose up -d)
  3. Runs bash tests (113 tests)
  4. Runs pytest in container (254 tests: 178 passed + 76 skipped)
  5. Runs parity tests with uv (64 tests)
  6. Shows comprehensive summary

Manual Container Startup (Faster)

# Pre-start containers
docker compose up -d reference-api api-first

# Run tests
./tests/run-all-tests.sh

Prerequisites

Required Tools

  1. Docker + Docker Compose (for all services)

    docker --version
docker compose version

  2. uv (Python package manager)

    # Install
curl -LsSf https://astral.sh/uv/install.sh | sh
# or
brew install uv

# Verify
uv --version

  3. bash (>= 3.2)

Required Containers

  • dev-reference-api - FastAPI code-first implementation
  • dev-api-first - FastAPI API-first implementation
  • All infrastructure containers (vault, postgres, redis, etc.)

Test Execution Details

Unit Tests (Inside Container)

# The script runs this command:
docker exec dev-reference-api pytest tests/ -v --tb=short

# Output:
# ================= 178 passed, 76 skipped, 6 warnings in 1.28s ==================
# Coverage: 84.39% (exceeds 80% requirement)

What runs:

  • Service unit tests (vault, cache, database)
  • Router unit tests (health, vault, cache, database, messaging, redis)
  • Exception handler tests
  • Request validator tests
  • Middleware tests (caching, circuit breaker, rate limiting)
  • CORS tests

Parity Tests (From Host with uv)

# The script runs this:
cd reference-apps/shared/test-suite
uv venv --quiet
uv pip install -r requirements.txt
uv run pytest -v

# Output:
# ============================== 26 passed in 0.35s ===============================

What runs:

  • Root endpoint parity
  • OpenAPI spec matching
  • Vault endpoint parity
  • Cache endpoint parity
  • Metrics format matching
  • Error handling parity
  • Health check parity

Troubleshooting

Container Not Running

# Error: "dev-reference-api container not running"
# Solution: Script auto-starts it, or manually:
docker compose up -d reference-api

uv Not Found

# Error: "uv not found - required for parity tests"
# Solution:
curl -LsSf https://astral.sh/uv/install.sh | sh
# or
brew install uv

Tests Fail

# Check service health first
./devstack.sh health

# Check specific container logs
docker logs dev-reference-api

# Restart infrastructure
./devstack.sh restart

Why Not Other Approaches?

Why not run everything in containers?

  • Parity tests need to access localhost:8000 and localhost:8001
  • Running from inside container would require complex networking setup
  • Client-perspective testing requires external access

Why not run everything locally with uv?

  • Python 3.14 has compatibility issues with asyncpg (C extension build failures)
  • Would require matching exact Python version (3.11) locally
  • Defeats purpose of containerization
  • Inconsistent across developer environments

Why not use virtualenv or pip directly?

  • uv is faster and handles dependency resolution better
  • uv creates isolated environments automatically
  • uv is the modern standard for Python package management
  • No manual venv creation/activation needed

Success Metrics

All tests passing produces this output:

Test Suites Run: 12
Passed: 12

Results by suite:
  ✓ Vault Integration
  ✓ PostgreSQL Vault Integration
  ✓ MySQL Vault Integration
  ✓ MongoDB Vault Integration
  ✓ Redis Vault Integration
  ✓ Redis Cluster
  ✓ RabbitMQ Integration
  ✓ FastAPI Reference App
  ✓ Performance & Load Testing
  ✓ Negative Testing & Error Handling
  ✓ FastAPI Unit Tests (pytest)
  ✓ API Parity Tests (pytest)

✓ ALL TESTS PASSED!

Integration with CI/CD

The same approach works in CI/CD:

# GitHub Actions example
- name: Run all tests
  run: |
    docker compose up -d reference-api api-first
    ./tests/run-all-tests.sh

Summary

Best approach is a hybrid:

  • Bash tests: Direct host execution ✓
  • Unit tests: Inside Docker containers ✓
  • Parity tests: From host with uv ✓

This provides:

  • Maximum compatibility (no Python version issues)
  • Minimum setup (auto-starts containers, uv manages deps)
  • Maximum reliability (production-like environment)
  • Maximum clarity (each test type uses optimal approach)

Clone this wiki locally