Skip to content

PostgreSQL Operations

Jump to bottom
Norm Brandinger edited this page Nov 20, 2025 · 1 revision

Test Suite Coverage

Table of Contents

This document describes the comprehensive test coverage for the DevStack Core infrastructure and reference applications.

Overview

The DevStack Core project has four distinct test suites and implementations:

  1. Infrastructure Tests (Shell scripts) - Test Docker containers, services, and integration
  2. FastAPI Application Tests (Pytest) - Test Python FastAPI reference application code
  3. Shared Test Suite (Pytest) - Validate code-first and API-first implementation parity
  4. Go Reference API (Manual testing + planned automated tests) - Go implementation with Gin framework

Total Coverage: 330+ tests across all components, plus Go manual validation

1. Infrastructure Integration Tests (Shell Scripts)

1.1 Vault Integration Tests (test-vault.sh)

10 tests - Core infrastructure security and certificate management

  • Vault container running
  • Vault auto-unseal functionality
  • Vault keys and token file existence
  • PKI bootstrap (Root CA, Intermediate CA)
  • Certificate roles for all services
  • Service credentials stored in Vault
  • PostgreSQL credentials validation
  • Certificate issuance functionality
  • CA certificate export
  • Management script commands

What it validates:

  • HashiCorp Vault operational
  • PKI infrastructure properly configured
  • Auto-unseal working correctly
  • All service credentials stored
  • Certificate issuance functional

1.2 Database Integration Tests

PostgreSQL (test-postgres.sh)

~5 tests - PostgreSQL container, Vault credential integration, connectivity

  • Container health and running state
  • Vault credentials retrieval
  • Database connectivity with Vault credentials
  • PostgreSQL version query
  • Configuration validation

MySQL (test-mysql.sh)

~5 tests - MySQL container, Vault credential integration, connectivity

  • Container health and running state
  • Vault credentials retrieval
  • Database connectivity with Vault credentials
  • MySQL version query
  • Configuration validation

MongoDB (test-mongodb.sh)

~5 tests - MongoDB container, Vault credential integration, connectivity

  • Container health and running state
  • Vault credentials retrieval
  • Database connectivity with Vault credentials
  • MongoDB version query
  • Configuration validation

1.3 Redis Cluster Tests (test-redis-cluster.sh)

12 tests - Comprehensive cluster configuration and operations

  • All 3 Redis containers running
  • Node reachability (PING test)
  • Cluster mode enabled on all nodes
  • Cluster initialization state (OK)
  • All 16384 hash slots assigned
  • 3 master nodes present
  • Slot distribution across masters
  • Data sharding functionality
  • Automatic redirection with -c flag
  • Vault password integration
  • Comprehensive cluster health check
  • Keyslot calculation

What it validates:

  • Proper cluster initialization
  • Complete slot coverage (16384 slots)
  • Data distribution and retrieval
  • Cross-node operations
  • Vault-managed authentication
  • Master-replica topology

1.4 RabbitMQ Integration Tests (test-rabbitmq.sh)

~5 tests - RabbitMQ container, Vault credential integration, messaging functionality

  • Container health and running state
  • Vault credentials retrieval
  • RabbitMQ connectivity
  • Queue creation and messaging
  • Management interface accessibility

1.5 FastAPI Container Tests (test-fastapi.sh)

14 tests - Container deployment and API endpoint testing

Container & Endpoints

  1. FastAPI container running
  2. HTTP endpoint accessible (port 8000)
  3. HTTPS endpoint accessible when TLS enabled (port 8443)
  4. Health check endpoint (/health/all)

Redis Cluster API Tests

  1. Redis health check with cluster details

    • Validates cluster_enabled: true
    • Validates cluster_state: ok
    • Validates total_nodes: 3

  2. Redis cluster nodes API (/redis/cluster/nodes)

    • Returns all 3 nodes
    • All nodes have slot assignments
    • Node IDs, roles, and slot ranges present

  3. Redis cluster slots API (/redis/cluster/slots)

    • 16384 total slots
    • 100% coverage
    • Slot distribution across masters

  4. Redis cluster info API (/redis/cluster/info)

    • Cluster state: ok
    • All slots assigned
    • Cluster statistics present

  5. Per-node info API (/redis/nodes/{node_name}/info)

    • Detailed node information
    • Redis version present
    • Cluster enabled flag correct

API Documentation Tests

  1. Swagger UI accessible (/docs)
  2. OpenAPI schema valid and accessible (/openapi.json)

Service Integration Tests

  1. Vault integration (health check)
  2. Database connectivity (PostgreSQL, MySQL, MongoDB)
  3. RabbitMQ integration

What it validates:

  • All Redis Cluster inspection APIs work correctly
  • Dual HTTP/HTTPS support
  • Health checks return cluster information
  • All service integrations functional
  • API documentation generated correctly

2. FastAPI Application Tests (Pytest)

Test Suite Overview

Location: reference-apps/fastapi/tests/

Total Tests: 254 tests

  • Executed: 178 unit tests (100% pass rate)
  • Skipped: 76 integration tests (require full infrastructure)

Code Coverage: 84.39% (exceeds 80% requirement)

Test Framework: pytest with async support, mocking, and coverage reporting

2.1 Caching Tests (test_caching.py)

23 tests - Cache middleware and operations

Cache Key Generation (5 tests)

  • Basic cache key generation with function name and path params
  • Cache keys with query parameters (sorted for consistency)
  • Namespace prefixes for cache organization
  • Long key hashing (>200 chars → MD5 hash)
  • Cache key consistency across requests

Cache Manager (4 tests)

  • Redis connection initialization
  • Graceful failure handling when Redis unavailable
  • Connection cleanup on shutdown
  • Clear all cache entries functionality

Cache Invalidation (4 tests)

  • Pattern-based cache invalidation (wildcards)
  • No-match pattern handling
  • Specific key invalidation
  • Non-existent key handling

Endpoint Caching (4 tests - SKIPPED)

  • Cache consistency across requests
  • Backend call reduction verification
  • Different params = different cache keys
  • Health endpoint caching behavior

Cache Configuration & Metrics (6 tests)

  • TTL configuration (5min for Vault, 30s for health)
  • Cache expiration behavior
  • Prometheus metrics (hits/misses/invalidations)
  • Redis client operations (get/set/delete)

2.2 Cache Demo Unit Tests (test_cache_demo_unit.py)

11 tests - Cache router operations

  • Get existing value from cache
  • Get nonexistent value (returns null)
  • Get value with no expiration
  • Set value without TTL
  • Set value with TTL
  • Delete existing key
  • Delete nonexistent key
  • Redis connection error handling
  • Get Redis client initialization

2.3 Circuit Breaker Tests (test_circuit_breaker.py)

10 tests - Circuit breaker pattern implementation

Event Listeners (4 tests)

  • Circuit open event listener
  • Circuit half-open event listener
  • Circuit close event listener
  • Circuit failure event listener

Metrics (2 tests)

  • Prometheus metrics existence
  • Service label in metrics

Behavior (3 tests)

  • Circuit breaker creation
  • Opens after threshold failures (5)
  • Prevents calls when open
  • Allows successful calls

Integration (1 test)

  • Middleware integration with FastAPI

2.4 CORS Tests (test_cors.py)

13 tests - Cross-Origin Resource Sharing

Headers (2 tests)

  • Content-Type header allowed
  • Authorization header allowed

Preflight Requests (4 tests)

  • Basic OPTIONS request handling
  • POST method preflight
  • Custom header preflight
  • Max-Age header presence

Methods & Origins (5 tests)

  • GET method allowed
  • POST method allowed
  • DELETE method allowed
  • localhost origin allowed
  • No origin header works

Integration (2 tests)

  • CORS with rate limiting
  • Consistent CORS across requests

2.5 Database Demo Tests (test_database_demo.py)

9 tests (6 skipped - require infrastructure)

PostgreSQL (3 tests)

  • Query success (SKIPPED - requires DB)
  • Connection failure handling
  • Vault credential failure

MySQL (3 tests)

  • Query success (SKIPPED - requires DB)
  • Connection failure handling
  • Vault credential failure

MongoDB (3 tests)

  • Query success (SKIPPED - requires DB)
  • Connection failure handling
  • Vault credential failure

2.6 Exception Handler Tests (test_exception_handlers.py + test_exception_handlers_unit.py)

35 tests - Exception handling and error responses

Exception Handlers (12 tests)

  • 503 responses for Vault unavailability
  • 404 responses for resource not found
  • 422 responses for validation errors
  • Retry suggestions in service unavailable responses
  • Request ID tracking in error responses
  • Error detail preservation in responses
  • Error logging with context
  • Debug mode behavior (show/hide stack traces)
  • Prometheus error counter metrics
  • Unhandled exception handling
  • HTTP exception conversion

Unit Tests (23 tests)

  • Custom exception handler registration
  • BaseAPIException formatting
  • ServiceUnavailableError handling
  • ValidationError formatting
  • ResourceNotFoundError handling
  • Request validation error conversion
  • Error response structure

2.7 Exception Hierarchy Tests (test_exceptions.py)

20 tests - Exception classes and inheritance

Exception Creation (9 tests)

  • VaultUnavailableError with secret paths
  • DatabaseConnectionError with connection details
  • CacheConnectionError initialization
  • ValidationError with field details
  • ResourceNotFoundError with resource info
  • AuthenticationError handling
  • RateLimitError with retry-after headers
  • CircuitBreakerError with service names
  • TimeoutError with operation details

Exception Hierarchy (5 tests)

  • All exceptions inherit from BaseAPIException
  • Service exceptions inherit from ServiceUnavailableError
    • VaultUnavailableError
    • DatabaseConnectionError
    • CacheConnectionError
    • MessageQueueError
    • CircuitBreakerError
  • TimeoutError uses 504 Gateway Timeout status

Exception Helpers (6 tests)

  • Exception to_dict() serialization
  • Service-specific error details
  • HTTP error code mapping
  • Error message formatting
  • Status code validation
  • Details dictionary handling

2.8 Health Router Tests (test_health_routers.py)

18 tests - Health check endpoints

Individual Service Health (6 tests - SKIPPED)

  • Vault health check
  • PostgreSQL health check
  • MySQL health check
  • MongoDB health check
  • Redis health check
  • RabbitMQ health check

Aggregated Health (12 tests)

  • All services healthy response
  • Individual service failure detection
  • Overall status calculation (healthy/degraded)
  • Response time tracking
  • Health endpoint format
  • Metrics generation
  • Partial outage handling
  • Complete outage handling
  • Degraded service detection
  • Cache health status
  • Service dependency tracking
  • Health check timeouts

2.9 Rate Limiting Tests (test_rate_limiting.py)

8 tests - Request rate limiting

  • Rate limit enforcement
  • 429 status code on limit exceeded
  • Retry-After header presence
  • Rate limit reset after time window
  • Different endpoints separate limits
  • Sliding window algorithm
  • Prometheus rate limit metrics
  • Rate limit headers in response (X-RateLimit-*)

2.10 Redis Cluster Tests (test_redis_cluster.py)

15 tests (10 skipped - require cluster)

Cluster Operations (10 tests - SKIPPED)

  • Get cluster nodes
  • Get cluster info
  • Get cluster slot distribution
  • Get individual node info
  • Cluster failover testing
  • Slot rebalancing
  • Master-replica verification
  • Cluster state validation
  • Node health checks
  • Cluster topology

Unit Tests (5 tests)

  • Cluster node retrieval
  • Connection failure handling
  • Cluster info parsing
  • Invalid node handling
  • Error response formatting

2.11 Request Validation Tests (test_request_validation.py)

15 tests - Input validation and sanitization

Path Parameters (6 tests)

  • Valid cache key format
  • Invalid special characters rejection
  • Valid service names
  • Invalid service names
  • Path traversal prevention
  • Null byte injection prevention

Input Sanitization (4 tests)

  • Service name lowercase conversion
  • Whitespace trimming
  • Special character removal
  • Maximum length enforcement

Content Type (3 tests)

  • JSON content type validation
  • Invalid content type rejection
  • Missing content type handling

Query Parameters (2 tests)

  • Valid parameter ranges
  • Invalid parameter values

2.12 Request Validators Tests (test_request_validators.py)

Tests - Pydantic model validation

  • ServiceNameParam validation
  • CacheKeyParam validation
  • QueueNameParam validation
  • SecretKeyParam validation
  • Value size limits
  • TTL range validation

2.13 Router Unit Tests (test_routers_unit.py)

30+ tests - Individual router unit tests

Cache Demo Routers (11 tests)

  • Get existing value
  • Get nonexistent value
  • Set value without TTL
  • Set value with TTL
  • Delete existing key
  • Delete nonexistent key
  • Redis error handling
  • Get Redis client
  • Value expiration
  • Cache statistics
  • Pattern matching

Vault Demo Routers (2 tests - SKIPPED)

  • Get vault secret
  • Get vault secret with key

Database Routers (3 tests)

  • PostgreSQL queries (1 SKIPPED)
  • MySQL queries (1 SKIPPED)
  • MongoDB queries (unit test)

Messaging Routers (3 tests - 2 SKIPPED)

  • Publish message
  • Publish failure handling
  • Consume messages (SKIPPED)

Redis Cluster Routers (4 tests)

  • Get cluster nodes
  • Connection failure handling
  • Cluster info parsing
  • Invalid node errors

2.14 Vault Service Tests (test_vault_service.py)

17 tests - Vault client implementation

Secret Retrieval (9 tests)

  • Successful secret retrieval
  • Retrieve specific key from secret
  • 404 handling (secret not found)
  • 403 handling (permission denied)
  • Key not found in secret
  • Connection timeout handling
  • Connection error handling
  • HTTP error handling
  • Unexpected error handling

Health Checks (5 tests)

  • Healthy Vault status
  • Sealed Vault detection
  • Uninitialized Vault detection
  • Standby node detection
  • Connection error handling

Integration Flows (3 tests)

  • Client initialization with settings
  • End-to-end secret retrieval flow
  • End-to-end error handling flow

3. Shared Test Suite (API Parity Tests)

Location: reference-apps/shared/test-suite/

Total Test Runs: 26 (16 test functions, parametrized to run against both APIs)

Purpose: Validate that the code-first and API-first implementations maintain 100% behavioral equivalence.

Test Framework: pytest with async support, parametrized fixtures, and deep comparison

3.1 Test Categories

Parity Tests (@pytest.mark.parity)

Tests that run against both implementations independently using parametrized fixtures.

How it works:

  • Single test function decorated with api_url fixture
  • Pytest automatically runs it twice: once for code-first (port 8000), once for API-first (port 8001)
  • Validates each implementation independently

Example:

async def test_health_check(self, api_url, http_client):
    # This test runs twice automatically
    response = await http_client.get(f"{api_url}/health/")
    assert response.status_code == 200

Comparison Tests (@pytest.mark.comparison)

Tests that directly compare responses from both APIs to ensure identical behavior.

How it works:

  • Test receives both API URLs via both_api_urls fixture
  • Makes requests to both implementations
  • Compares responses for exact matches

Example:

async def test_responses_match(self, both_api_urls, http_client):
    code_first = await http_client.get(f"{both_api_urls['code-first']}/health/")
    api_first = await http_client.get(f"{both_api_urls['api-first']}/health/")
    assert code_first.json() == api_first.json()

3.2 Health Check Tests (test_health_checks.py)

7 test runs (4 test functions)

Parity Tests (6 test runs from 3 functions)

  1. Simple health check (test_simple_health_check)

    • Runs against both APIs (2 runs)
    • Validates /health/ endpoint returns 200 with {"status": "ok"}

  2. Health response structure (test_health_response_structure)

    • Runs against both APIs (2 runs)
    • Validates response is dict with string status field

  3. Vault health check (test_vault_health_check)

    • Runs against both APIs (2 runs)
    • Validates /health/vault returns healthy/unhealthy status

Comparison Tests (1 test run)

  1. Health responses match (test_health_responses_match)
    • Direct comparison between both APIs
    • Ensures identical health response structure and content

3.3 API Parity Tests (test_api_parity.py)

19 test runs (12 test functions)

Root Endpoint Tests (3 test runs)

  • Root endpoint returns info - Parametrized (2 runs)
    • Validates presence of name, version, description
  • Root endpoint structure matches - Comparison (1 run)
    • Ensures both APIs return identical top-level keys

OpenAPI Spec Tests (5 test runs)

  • OpenAPI endpoint accessible - Parametrized (2 runs)
    • Validates /openapi.json returns valid OpenAPI spec
  • OpenAPI specs match - Comparison (1 run)
    • Ensures both APIs expose identical paths
  • OpenAPI version format - Parametrized (2 runs)
    • Validates OpenAPI 3.x.x version format

Vault Endpoints Tests (2 test runs)

  • Vault secret endpoint structure - Parametrized (2 runs)
    • Validates /examples/vault/secret/{path} error handling

Cache Endpoints Tests (3 test runs)

  • Cache GET endpoint exists - Parametrized (2 runs)
    • Validates /examples/cache/{key} endpoint
  • Cache endpoints have same behavior - Comparison (1 run)
    • Ensures identical response structure for cache operations

Metrics Endpoint Tests (3 test runs)

  • Metrics endpoint accessible - Parametrized (2 runs)
    • Validates /metrics returns Prometheus format
  • Metrics format matches - Comparison (1 run)
    • Ensures both APIs return same Prometheus format

Error Handling Tests (3 test runs)

  • 404 response format - Parametrized (2 runs)
    • Validates consistent 404 error structure
  • 404 responses match - Comparison (1 run)
    • Ensures identical 404 error responses

3.4 Test Infrastructure

Fixtures (conftest.py)

@pytest.fixture(params=[CODE_FIRST_URL, API_FIRST_URL])
def api_url(request):
    """Parametrized fixture - tests run twice, once per API"""
    return request.param

@pytest.fixture
async def both_api_urls():
    """Fixture providing both URLs for comparison tests"""
    return {
        "code-first": CODE_FIRST_URL,
        "api-first": API_FIRST_URL
    }

@pytest.fixture
async def http_client():
    """Async HTTP client for making API requests"""
    async with httpx.AsyncClient() as client:
        yield client

Configuration (pytest.ini)

[pytest]
markers =
    parity: Tests that run against both implementations
    comparison: Tests that compare both implementations
    health: Health check related tests

3.5 Coverage Areas

Area Parity Tests Comparison Tests Total Runs
Root Endpoint 1 × 2 1 3
OpenAPI Spec 2 × 2 1 5
Health Checks 3 × 2 1 7
Vault Integration 1 × 2 0 2
Cache Operations 1 × 2 1 3
Metrics 1 × 2 1 3
Error Handling 1 × 2 1 3
Total 10 × 2 6 26

3.6 Running Shared Test Suite

Prerequisites

Both APIs must be running:

# Start both implementations
docker compose up -d reference-api api-first

# Verify both are healthy
curl http://localhost:8000/health/
curl http://localhost:8001/health/

Run Tests

cd reference-apps/shared/test-suite
pip install -r requirements.txt
pytest -v

Expected Results

test_health_checks.py::TestHealthEndpoints::test_simple_health_check[code-first] PASSED
test_health_checks.py::TestHealthEndpoints::test_simple_health_check[api-first] PASSED
test_health_checks.py::TestHealthParity::test_health_responses_match PASSED
test_api_parity.py::TestRootEndpoint::test_root_endpoint_returns_info[code-first] PASSED
test_api_parity.py::TestRootEndpoint::test_root_endpoint_returns_info[api-first] PASSED
...

========================== 26 passed in 2.34s ===========================

Run Specific Categories

# Health checks only
pytest -v -m health

# Parity tests only
pytest -v -m parity

# Comparison tests only
pytest -v -m comparison

3.7 What Gets Validated

Endpoint Consistency:

  • ✅ Identical endpoint paths
  • ✅ Identical response structures
  • ✅ Identical status codes
  • ✅ Identical error handling

API Contract:

  • ✅ OpenAPI specifications match
  • ✅ Request/response formats identical
  • ✅ Content-Type headers consistent
  • ✅ Error response formats match

Behavioral Equivalence:

  • ✅ Health checks return same information
  • ✅ Cache operations behave identically
  • ✅ Metrics format is consistent
  • ✅ 404 errors formatted the same

Integration Points:

  • ✅ Vault integration consistent
  • ✅ Database access patterns match
  • ✅ Cache behavior identical
  • ✅ All service integrations work

3.8 Success Criteria

The shared test suite enforces:

  • 100% pass rate - All 26 tests must pass
  • No manual sync - Automated validation prevents drift
  • CI/CD integration - Tests run on every commit
  • Continuous parity - Both implementations stay synchronized

Current Status: ✅ 26/26 tests passing (100% parity achieved)

3.9 Maintenance

When adding new endpoints to either implementation:

  1. Update OpenAPI spec (reference-apps/shared/openapi.yaml)
  2. Implement in code-first (reference-apps/fastapi/)
  3. Implement in API-first (reference-apps/fastapi-api-first/)
  4. Add parity test to validate endpoint on both implementations
  5. Add comparison test if responses must be byte-for-byte identical
  6. Run shared test suite - must achieve 100% pass rate
  7. Update documentation (this file, README files)

Running Tests

Infrastructure Tests (Shell Scripts)

Run All Infrastructure Tests

./tests/run-all-tests.sh

This runs all test suites in sequence and provides a comprehensive summary.

Run Individual Test Suites

# Infrastructure
./tests/test-vault.sh

# Databases
./tests/test-postgres.sh
./tests/test-mysql.sh
./tests/test-mongodb.sh

# Cache & Messaging
./tests/test-redis-cluster.sh
./tests/test-rabbitmq.sh

# Application Container
./tests/test-fastapi.sh

FastAPI Application Tests (Pytest)

Run All Tests

cd reference-apps/fastapi
pytest tests/ -v

Run with Coverage

pytest tests/ --cov=app --cov-report=html --cov-report=term

Run Specific Test File

pytest tests/test_vault_service.py -v

Run Specific Test

pytest tests/test_vault_service.py::TestVaultClientGetSecret::test_get_secret_success -v

Run Only Unit Tests

pytest tests/ -v -m unit

Run Only Integration Tests

pytest tests/ -v -m integration

Test Dependencies

System Tools (Infrastructure Tests)

  • curl - HTTP client (usually pre-installed)
  • jq - JSON processor 
    # macOS
brew install jq

# Ubuntu/Debian
apt-get install jq
  • docker - Container runtime (via Colima)
  • docker-compose - Container orchestration

Python Dependencies (Application Tests)

cd reference-apps/fastapi
pip install -r requirements.txt

Includes:

  • pytest - Test framework
  • pytest-asyncio - Async test support
  • pytest-cov - Coverage reporting
  • pytest-mock - Mocking utilities
  • httpx - HTTP client for testing
  • fastapi[all] - FastAPI with all extras

Test Results Format

Infrastructure Tests Output

Each shell script test suite provides:

  • Real-time test execution output
  • Color-coded pass/fail indicators (green ✓ / red ✗)
  • Summary with total tests, passed, and failed counts
  • List of failed tests (if any)

Example output:

=========================================
  FastAPI Reference App Test Suite
=========================================

[TEST] Test 1: FastAPI container is running
[PASS] FastAPI container is running

[TEST] Test 5: Redis health check with cluster details
[PASS] Redis health shows cluster enabled with 3 nodes in ok state

[TEST] Test 7: Redis cluster slots API endpoint
[PASS] Redis cluster slots API shows 100% coverage (16384 slots)

=========================================
  Test Results
=========================================
Total tests: 14
Passed: 13

✓ All FastAPI tests passed!

Pytest Output

============================= test session starts ==============================
collected 254 items

tests/test_caching.py::TestCacheKeyGeneration::test_generate_cache_key_basic PASSED [ 0%]
tests/test_caching.py::TestCacheKeyGeneration::test_generate_cache_key_with_query_params PASSED [ 1%]
...
tests/test_vault_service.py::TestVaultServiceIntegration::test_vault_error_handling_flow PASSED [100%]

---------- coverage: platform linux, python 3.11.14-final-0 ----------
Name                                   Stmts   Miss  Cover   Missing
--------------------------------------------------------------------
app/__init__.py                            0      0   100%
app/config.py                             22      0   100%
app/exceptions.py                         86      0   100%
...
--------------------------------------------------------------------
TOTAL                                    974    152    84%

Required test coverage of 80% reached. Total coverage: 84.39%
================= 178 passed, 76 skipped, 6 warnings in 1.76s ==================

Coverage Summary

Infrastructure Tests (Shell Scripts)

Component Test Suite Tests Coverage
Vault test-vault.sh 10 PKI, secrets, certificates, auto-unseal
PostgreSQL test-postgres.sh ~5 Container, credentials, connectivity
MySQL test-mysql.sh ~5 Container, credentials, connectivity
MongoDB test-mongodb.sh ~5 Container, credentials, connectivity
Redis Cluster test-redis-cluster.sh 12 Cluster init, slots, sharding, failover
RabbitMQ test-rabbitmq.sh ~5 Container, credentials, messaging
FastAPI Container test-fastapi.sh 14 Container, APIs, health, cluster endpoints

Subtotal: ~56 infrastructure tests

Application Tests (Pytest)

Category Tests Coverage
Caching 34 Cache middleware, operations, keys, invalidation
Circuit Breakers 10 Pattern implementation, metrics, events
CORS 13 Headers, preflight, methods, origins
Databases 9 PostgreSQL, MySQL, MongoDB integration
Exception Handling 55 Handlers, hierarchy, formatting, errors
Health Checks 18 Service health, aggregation, metrics
Rate Limiting 8 IP-based limits, throttling, headers
Redis Cluster 15 Cluster ops, nodes, slots, info
Request Validation 15+ Input validation, sanitization, types
Routers 30+ Cache, vault, database, messaging routers
Vault Service 17 Secret retrieval, health checks, errors

Subtotal: 178 executed unit tests (76 integration tests skipped)

Code Coverage: 84.39%

Shared Test Suite (Pytest)

Category Test Runs Coverage
Health Checks 7 Simple health, structure, Vault health, response matching
Root Endpoint 3 API info, structure consistency
OpenAPI Spec 5 Spec access, matching, version format
Vault Endpoints 2 Error handling, endpoint structure
Cache Endpoints 3 GET operations, behavior consistency
Metrics 3 Prometheus format, endpoint access
Error Handling 3 404 format, response consistency

Subtotal: 26 parity test runs (16 test functions, parametrized across both APIs)

4. Go Reference API

Location: reference-apps/golang/

Implementation Status: ✅ Complete and tested

Purpose: Demonstrate language-agnostic infrastructure integration patterns using Go and the Gin web framework.

4.1 Implementation Features

The Go reference API provides a production-ready implementation showcasing:

Core Capabilities

  • Vault Integration: HashiCorp Vault client for secrets management
  • Database Connectivity: PostgreSQL (pgx/v5), MySQL (go-sql-driver), MongoDB (mongo-driver)
  • Caching: Redis cluster operations with go-redis/v9
  • Messaging: RabbitMQ integration with amqp091-go
  • Health Checks: Comprehensive service health monitoring
  • Redis Cluster Management: Full cluster inspection and node monitoring

Go-Specific Features

  • Concurrency: Goroutines for concurrent health checks
  • Context Propagation: Proper context.Context usage throughout
  • Graceful Shutdown: Signal handling (SIGINT/SIGTERM) with clean termination
  • Structured Logging: Logrus with JSON formatting and request ID correlation
  • Type Safety: Strong typing with explicit error handling
  • Prometheus Metrics: Native Go Prometheus client integration

4.2 API Endpoints

Container: golang-api (dev-golang-api) Ports: 8002 (HTTP), 8445 (HTTPS) Framework: Gin v1.9.1 Go Version: 1.21

Available Endpoints

Endpoint Method Description
/ GET API information and endpoint discovery
/health/ GET Simple health check (no dependencies)
/health/all GET Aggregate health (concurrent checks)
/health/vault GET Vault connectivity and status
/health/postgres GET PostgreSQL connection test
/health/mysql GET MySQL connection test
/health/mongodb GET MongoDB connection test
/health/redis GET Redis cluster health
/health/rabbitmq GET RabbitMQ connectivity
/examples/vault/secret/:service_name GET Retrieve service secrets
/examples/vault/secret/:service_name/:key GET Retrieve specific secret key
/examples/database/postgres/query GET PostgreSQL query example
/examples/database/mysql/query GET MySQL query example
/examples/database/mongodb/query GET MongoDB query example
/examples/cache/:key GET Get cache value
/examples/cache/:key POST Set cache value with TTL
/examples/cache/:key DELETE Delete cache key
/redis/cluster/nodes GET List all cluster nodes
/redis/cluster/slots GET Slot distribution
/redis/cluster/info GET Cluster information
/redis/nodes/:node_name/info GET Detailed node information
/examples/messaging/publish/:queue POST Publish message to queue
/examples/messaging/queue/:queue_name/info GET Queue information
/metrics GET Prometheus metrics

4.3 Manual Testing Results

Test Date: 2025-10-27

✅ Verified Working Endpoints

  1. Root Endpoint (/)

    • Returns comprehensive API information
    • Lists all available endpoint categories
    • Includes security configuration details

  2. Simple Health Check (/health/)

    • Returns {"status": "ok"}
    • No external dependencies
    • Fast response (<5ms)

  3. Vault Health Check (/health/vault)

    • Successfully connects to Vault
    • Returns initialization status, seal status, version
    • Proper error handling

  4. Aggregate Health Check (/health/all)

    • Uses goroutines for concurrent service checks
    • Returns health status for all available services
    • Fast execution with parallel checks

  5. Prometheus Metrics (/metrics)

    • Exposes Go runtime metrics (goroutines, memory, GC)
    • Standard Prometheus format
    • HTTP request counters and histograms

Expected Behavior - Vault Secrets Not Bootstrapped

Database and cache endpoints properly return errors when Vault secrets are missing:

  • /health/postgres - Returns "secret not found" error (correct behavior)
  • /health/redis - Returns "secret not found" error (correct behavior)
  • /examples/cache/* - Returns "secret not found" error (correct behavior)

This demonstrates the implementation correctly integrates with Vault and handles missing credentials appropriately.

4.4 Architecture Highlights

Project Structure

reference-apps/golang/
├── cmd/api/main.go              # Application entry point
├── internal/
│   ├── config/config.go         # Configuration management
│   ├── handlers/                # HTTP handlers
│   │   ├── health.go
│   │   ├── vault.go
│   │   ├── database.go
│   │   ├── cache.go
│   │   ├── redis_cluster.go
│   │   └── messaging.go
│   ├── middleware/logging.go    # Logging + CORS
│   └── services/vault.go        # Vault client
├── Dockerfile                   # Multi-stage build
├── init.sh                      # Vault integration + TLS
└── start.sh                     # Application startup

Key Design Patterns

Dependency Injection:

healthHandler := handlers.NewHealthHandler(cfg, vaultClient)

Context Propagation:

ctx, cancel := context.WithTimeout(c.Request.Context(), 5*time.Second)
defer cancel()

Resource Cleanup:

defer conn.Close(ctx)

Concurrent Operations:

var wg sync.WaitGroup
for _, service := range services {
    wg.Add(1)
    go checkService(service, &wg)
}
wg.Wait()

4.5 Comparison with Python Implementation

Feature Go (Gin) Python (FastAPI)
Concurrency Model Goroutines (native) asyncio (event loop)
Type System Static typing (compile-time) Type hints (runtime optional)
Performance Compiled, very fast Interpreted, fast with async
Memory Usage Low (~20-30MB) Higher (~80-150MB)
Startup Time Instant (<100ms) Slower (~1-2s)
Deployment Single binary Python + dependencies
Error Handling Explicit returns Exceptions
Context Management context.Context async context
Testing go test pytest

4.6 Testing Status

✅ Completed

  • Docker image builds successfully
  • Container starts and initializes
  • All routes registered correctly
  • Vault connectivity verified
  • Health check endpoints functional
  • Prometheus metrics exposed
  • Error handling for missing Vault secrets
  • Graceful shutdown working
  • Logging middleware operational
  • CORS middleware configured

🔄 Requires Infrastructure Bootstrap

The following features require Vault secrets to be bootstrapped:

  • Database connectivity tests (PostgreSQL, MySQL, MongoDB)
  • Redis cache operations
  • RabbitMQ messaging operations
  • Redis cluster management APIs

These endpoints correctly detect missing credentials and return appropriate error messages.

4.9 Automated Test Suite

Test Status: ✅ 35+ tests, 100% PASS RATE

Test Files Created

  • internal/config/config_test.go - Configuration testing
  • internal/services/vault_test.go - Vault client testing
  • internal/middleware/logging_test.go - Middleware testing

Test Coverage by Package

Config Package (12 tests)

  • Default configuration values
  • Custom HTTP/HTTPS ports
  • Vault address and token configuration
  • Debug mode enabled/disabled
  • Environment settings (development/production)
  • Database configuration (PostgreSQL, MySQL, MongoDB)
  • Redis configuration
  • RabbitMQ configuration
  • Environment variable fallback handling
  • Configuration completeness validation
  • Coverage: 91.7%

Vault Service (9 test groups)

  • VaultClient creation with various configurations
  • GetSecret context handling (timeout, cancellation)
  • GetSecretKey method validation
  • HealthCheck functionality
  • Client structure validation
  • Concurrent access safety
  • Error message formatting
  • Method signature validation
  • Coverage: 62.5%

Middleware (14 tests)

  • Request ID generation and propagation
  • Request logging with structured fields
  • HTTP method logging (GET, POST, PUT, DELETE, PATCH)
  • Status code logging (200, 201, 400, 404, 500)
  • Request duration measurement
  • CORS header configuration
  • OPTIONS preflight request handling
  • Credential allowance
  • Max-Age cache control
  • Standard header allowance
  • Middleware integration
  • Coverage: 100.0%

Test Execution

cd reference-apps/golang
go test ./... -v -cover

Results:

✓ Config:     12 tests PASS (91.7% coverage)
✓ Services:   9 tests PASS (62.5% coverage)
✓ Middleware: 14 tests PASS (100.0% coverage)
────────────────────────────────────────────
TOTAL:        35+ tests, 100% PASS RATE

Test Patterns Used

  1. Table-Driven Tests: Structured test cases for multiple scenarios
  2. Subtests: Organized test output with t.Run()
  3. Context Testing: Timeout and cancellation handling
  4. HTTP Testing: Using httptest.Recorder and httptest.NewRequest
  5. Concurrency Testing: Goroutine safety validation
  6. Error Validation: Comprehensive error message checking

📝 Future Testing Enhancements

  • Handler tests using httptest (database, cache, messaging handlers)
  • Integration tests with mock Vault server
  • Benchmark tests for performance validation
  • E2E tests with full infrastructure stack

4.7 Running the Go API

Start Service

# Build and start
docker-compose up -d golang-api

# View logs
docker-compose logs -f golang-api

# Check health
curl http://localhost:8002/health/

Test Endpoints

# Root endpoint
curl http://localhost:8002/ | jq .

# Health checks
curl http://localhost:8002/health/all | jq .
curl http://localhost:8002/health/vault | jq .

# Metrics
curl http://localhost:8002/metrics

# Cache operations (requires Vault bootstrap)
curl -X POST "http://localhost:8002/examples/cache/test?value=hello&ttl=60"
curl http://localhost:8002/examples/cache/test

# Redis cluster info (requires Vault bootstrap)
curl http://localhost:8002/redis/cluster/nodes | jq .

4.8 Dependencies

Main Dependencies:

  • github.com/gin-gonic/gin v1.9.1 - Web framework
  • github.com/hashicorp/vault/api v1.10.0 - Vault client
  • github.com/jackc/pgx/v5 v5.5.0 - PostgreSQL driver
  • github.com/go-sql-driver/mysql v1.7.1 - MySQL driver
  • go.mongodb.org/mongo-driver v1.13.1 - MongoDB driver
  • github.com/redis/go-redis/v9 v9.3.0 - Redis client
  • github.com/rabbitmq/amqp091-go v1.9.0 - RabbitMQ client
  • github.com/prometheus/client_golang v1.17.0 - Prometheus metrics
  • github.com/sirupsen/logrus v1.9.3 - Structured logging

Total Dependencies: 76 (including transitive)

Total Test Coverage

Infrastructure + Application + Parity: 330+ tests

  • 56 infrastructure integration tests (shell scripts)
  • 178 application unit tests (pytest - executed)
  • 76 application integration tests (pytest - skipped in CI)
  • 26 shared parity test runs (pytest - validates API equivalence)

Combined Coverage:

  • All Docker containers and services
  • Vault PKI and secrets management
  • Database connectivity and credentials
  • Redis cluster operations
  • RabbitMQ messaging
  • FastAPI application endpoints
  • Middleware (caching, circuit breakers, CORS, rate limiting)
  • Exception handling
  • Request validation
  • Health checks
  • Prometheus metrics
  • API parity validation (code-first vs API-first equivalence)

Continuous Testing

Run tests after:

  • Initial infrastructure setup (./devstack.sh start)
  • Service configuration changes
  • Certificate regeneration
  • Vault bootstrap
  • Container restarts
  • Application deployments
  • Code changes (pytest tests/)
  • Pull request reviews
  • Pre-production deployment

This ensures all components remain properly configured, integrated, and functional.

Test Quality Metrics

Infrastructure Tests

  • Real environment testing: Tests run against actual Docker containers
  • Integration validation: Tests verify inter-service communication
  • Credential security: Tests verify Vault integration throughout

Application Tests

  • High unit test coverage: 84.39% code coverage
  • 100% pass rate: All 178 executed tests passing
  • Comprehensive mocking: External dependencies mocked for speed
  • Async support: Full async/await testing with pytest-asyncio
  • Fast execution: Unit tests complete in <2 seconds

CI/CD Considerations

Recommended CI Pipeline

# Example GitHub Actions workflow
jobs:
  infrastructure-tests:
    runs-on: ubuntu-latest
    steps:
      - Install Docker/Colima
      - ./devstack.sh start
      - ./tests/run-all-tests.sh

  application-tests:
    runs-on: ubuntu-latest
    steps:
      - cd reference-apps/fastapi
      - pip install -r requirements.txt
      - pytest tests/ --cov=app --cov-fail-under=80

Test Environments

  • Local Development: All tests (infrastructure + application)
  • CI/CD: Unit tests only (fast feedback)
  • Pre-Production: Full integration tests
  • Production: Smoke tests + health checks

Future Test Improvements

Planned enhancements:

  • Performance testing
  • Load testing with Locust
  • Security testing
  • End-to-end user workflows
  • Automated snapshot testing
  • Chaos engineering tests

Documentation

Clone this wiki locally