Skip to content

v0.3.0 - Alpha Release

Choose a tag to compare

@adriannoes adriannoes released this 26 Jan 23:40
· 1249 commits to main since this release

This release addresses critical test stability issues that were blocking reliable CI/CD execution. What started as intermittent test failures led to a comprehensive refactoring of the test infrastructure, resulting in a more reliable and maintainable codebase.

🔍 The problem: Global state interference in test execution

The test suite exhibited a critical issue: 33 tests would fail when run as part of the full suite, but all tests passed when executed individually. Root cause analysis identified slowapi.Limiter as the culprit—it maintains global state that persists across test executions.

Technical Details:

  • Even with unique storage URIs per test, slowapi.Limiter retains internal state beyond the storage layer
  • After approximately 18 tests using testclient, the accumulated request count triggered rate limiting (HTTP 429) in subsequent tests
  • This created non-deterministic test behavior that made CI/CD unreliable

The issue demonstrated a fundamental problem: tests were not properly isolated, leading to cross-test contamination.

💡 Solution: Three-Layer Isolation Strategy

We implemented a comprehensive three-layer approach to guarantee complete test isolation:

  1. Process Isolation: Using pytest-xdist, each test runs in its own process, eliminating any possibility of shared state between tests.

  2. Aggressive Monkeypatching: We replace the limiter at the module level to ensure complete isolation, even for libraries with persistent global state.

  3. Strategic Organization: We completely reorganized the test structure, clearly separating unit, integration, and end-to-end tests into distinct directories.

🏗️ [New] Test architecture

The test structure was redesigned to follow best practices:

tests/
├── transport/
│   ├── unit/              # Isolated, fast unit tests
│   ├── integration/       # Integration tests with proper isolation
│   └── e2e/               # Complete end-to-end tests

Each test category now has its own namespace, fixtures, and isolation guarantees.

📚 Documentation

Beyond fixing the issue, we documented the entire testing strategy. Created a complete testing guide in docs/testing.md that covers:

  • How to organize new tests
  • Isolation strategies
  • Practical examples for each test type
  • Patterns and anti-patterns

Contributors can now quickly understand how to write tests that won't interfere with each other.

🐛 Additional Fixes

During the investigation, we also fixed a critical bug: an UnboundLocalError in server.py related to rate limiter initialization. This type of bug is exactly what flaky tests hide—with stable tests, issues like this are caught immediately.

📊 Results

  • 33 failing tests0 failing tests
  • 610 tests passing consistently
  • 100% reliability in full suite execution
  • Coverage maintained across all modules (92.47%)

🎉 Impact

The test suite is now stable and deterministic. We can execute all 610 tests with confidence that each test is truly isolated and results are reproducible.

This release doesn't add new features, but significantly improves project stability and reliability. This type of foundational work is essential for long-term project health.


Technical Details

Changed

Test Infrastructure Refactoring (PR #18)

  • Test Organization:
    • Reorganized test structure with clear separation between unit, integration, and E2E tests
    • Created tests/transport/unit/ for isolated unit tests
    • Created tests/transport/integration/ for integration tests with proper isolation
    • Created tests/transport/e2e/ for end-to-end tests
  • Test Stability:
    • Fixed 33 failing tests caused by slowapi.Limiter global state interference
    • Implemented process isolation using pytest-xdist to prevent test interference
    • Added aggressive monkeypatch strategy for complete rate limiter isolation
    • Separated rate-limiting tests from core server tests to prevent cross-contamination
  • Documentation:
    • Added comprehensive testing guide in docs/testing.md
    • Documented test organization strategy and isolation techniques
    • Added examples for writing unit, integration, and E2E tests

Fixed

  • Resolved UnboundLocalError in server.py related to rate limiter initialization
  • Fixed test flakiness caused by global state persistence across test runs
  • Improved test reliability with proper fixture isolation

Technical Details

  • Test Count: 610 tests (all passing)
  • Test Execution: Process isolation via pytest-xdist for complete state separation
  • Test Coverage: Maintained comprehensive coverage across all modules (92.47%)

Full Changelog: v0.1.0...v0.3.0