Replace fixed timing waits in tests with condition-based polling #1943
Description
Problem
Tests use fixed tokio::time::sleep() calls that assume specific timing, making them flaky in CI environments with variable performance. Example: test_multiple_clients_subscription had a 40s startup wait that was insufficient for CI.
Root Causes
- Fixed waits don't adapt to actual system conditions
- CI environments have variable performance (slower than local)
- Network formation, contract propagation, and peer connections are non-deterministic
- No feedback loop - tests don't know when conditions are actually met
Solution Pattern
Replace fixed waits with:
- Condition polling loops - Check for actual readiness (e.g., "do nodes have ring connections?")
- Retry logic with exponential backoff - For operations that may initially fail
- Adaptive timeouts - Longer total timeout, shorter per-attempt timeout with retries
Example Fix (from PR #1937)
- Before:
sleep(40s)→ hope nodes are ready - After:
sleep(60s)+ retry GET up to 5x with exponential backoff (2s, 4s, 8s, 16s, 32s) - Result: Test adapts to actual conditions, passes reliably in 50-120s range
Action Items
- Audit test suite for
tokio::time::sleepusage - Identify which sleeps are waiting for conditions (vs intentional delays)
- Replace with proper readiness checks or retry logic
- Consider creating helper functions:
wait_for_connections(),retry_operation(), etc.
Files to Check
crates/core/tests/*.rs- Integration tests- Look for patterns like "wait for X seconds for Y to happen"
[AI-assisted debugging and comment]