Fix flaky TestStorageProviderUploadDownload integration test

[jrepp]

Summary

Fixes the flaky TestStorageProviderUploadDownload integration test by improving Docker Compose setup and adding an integrated stability testing capability.

Problem

The integration test was failing due to port conflicts when Docker containers from previous test runs hadn't fully cleaned up. Port 5556 (used by the dex service) was being held by gvproxy even after docker compose down, causing subsequent test runs to fail with "address already in use" errors.

Test Results Before Fix:

• Intermittent failures due to port conflicts
• No systematic way to verify test stability

Solution

1. Improved Docker Compose Setup (testing/integration_test.go)

• Added retry logic (3 attempts) with exponential backoff
• Implemented 2-second wait after cleanup for port release
• Enhanced logging at each setup step

2. Refactored Storage Test with Stability Testing (testing/storage_integration_test.go)

• Extracted test logic into reusable helper function
• Added TestStorageProviderUploadDownloadStability for multi-iteration testing
• Added 50ms stabilization wait after upload
• Improved error messages with detailed data comparison
• Made stability test opt-in via RUN_STABILITY_TESTS environment variable

Test Results After Fix

Regular Test:

go test -v -run TestStorageProviderUploadDownload
# PASS (44s)

Stability Test:

RUN_STABILITY_TESTS=1 STABILITY_TEST_ITERATIONS=3 \
  go test -v -run TestStorageProviderUploadDownloadStability
# PASS - 3/3 iterations successful (45s)

Benefits

1. Improved Reliability: Retry logic handles transient port conflicts
2. Better Debugging: Stability test helps identify race conditions
3. Opt-in Testing: Stability test skipped by default to keep CI fast
4. Configurable: Can adjust iteration count via environment variable

Usage

# Run regular tests
go test -v ./testing/...

# Enable stability testing with 10 iterations
RUN_STABILITY_TESTS=1 STABILITY_TEST_ITERATIONS=10 \
  go test -v -run TestStorageProviderUploadDownloadStability