Skip to content

Conversation

@sanity
Copy link
Collaborator

@sanity sanity commented Dec 8, 2025

Problem

CI tests have been flaky on self-hosted runners with high core counts (64 CPUs on technic). Multiple different tests fail intermittently with timeouts:

  • test_put_contract
  • test_put_contract_three_hop_returns_response
  • test_put_merge_persists_state
  • test_update_contract
  • test_three_node_network_connectivity
  • test_logger_capture*

Additionally, the six-peer-regression job fails with a linker error when building freenet fresh:

rust-lld: error: unable to find library -llzma

Root Cause

  1. Test thread contention: Cargo defaults to one test thread per CPU. With 64 CPUs, 64+ network tests run simultaneously, causing:

    • UDP port exhaustion/collisions
    • Async runtime starvation
    • Timing-sensitive operations failing their timeouts
  2. Library path not set: When six-peer-regression builds freenet from river's workspace (fresh build, no cache), the linker can't find liblzma because rust-lld doesn't search /usr/lib by default on Arch Linux.

Solution

  1. Limit test threads to 8 (--test-threads=8) - reduces parallel test execution to a manageable level
  2. Set LIBRARY_PATH and PKG_CONFIG_PATH for the six-peer-regression job

Testing

  • All tests pass consistently with thread limiting (verified on both nova and technic)
  • Currently running CI on the hop-by-hop-routing PR with these same fixes

[AI-assisted - Claude]

@sanity sanity enabled auto-merge December 8, 2025 22:55
Two fixes for CI flakiness:

1. Limit test threads to 8 (--test-threads=8)
   - On 64-core runners, 64+ parallel tests cause resource contention
   - Network tests timeout waiting for UDP responses due to scheduler starvation
   - All tests pass consistently with this limit

2. Add library paths for six-peer-regression job
   - LIBRARY_PATH and PKG_CONFIG_PATH for liblzma on Arch Linux
   - Needed when building freenet fresh from river's workspace

Also adds RUST_LOG for transport debugging to help diagnose future issues.

[AI-assisted - Claude]
@sanity sanity force-pushed the fix/ci-test-stability branch from 1f9183a to 975a8d2 Compare December 8, 2025 23:54
@sanity sanity added this pull request to the merge queue Dec 9, 2025
Merged via the queue into main with commit b64a85b Dec 9, 2025
8 checks passed
@sanity sanity deleted the fix/ci-test-stability branch December 9, 2025 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants