Skip to content

test(umbp): use kernel-assigned ports in distributed tests to fix CI flakiness#417

Merged
isytwu merged 1 commit into
ROCm:mainfrom
isytwu:fix/umbp-test-dynamic-ports
Jun 23, 2026
Merged

test(umbp): use kernel-assigned ports in distributed tests to fix CI flakiness#417
isytwu merged 1 commit into
ROCm:mainfrom
isytwu:fix/umbp-test-dynamic-ports

Conversation

@isytwu

@isytwu isytwu commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • This is a test/CI-infra fix, not an umbp behavior bug. The umbp production code is unchanged.
  • Distributed umbp integration tests (cross_node_smoke, pool_client_batch_put, peer_service) hard-coded their peer-service / server ports from fixed bases (52000 / 53000 / 50200).
  • PoolClient / PeerServiceServer bind that port directly and register it verbatim as the peer_address, so on the shared self-hosted CI runner the fixed ports collide with concurrent jobs or leftover servers -> bind fails -> remote AllocateSlot/CommitSlot RPCs unavailable -> cross-node Put/Get fail (ctest exit code 8). Non-deterministic, so a PR can pass CI yet the same commit fail once merged to main.
  • Fix: ask the kernel for a currently-free ephemeral port (bind :0 -> getsockname -> close), falling back to a PID-offset random base only if the probe is unavailable. Mirrors the existing AllocPort() in test_umbp_tags.cpp.

Test plan

  • In-container CI ctest filter -R '^(umbp_|test_dummy_ssd_tier|test_prefix_aware_eviction|test_ssd_tier)': 22/22 pass
  • Single run with old 52000-52059 range fully occupied: 18/18 pass (previously 18/18 fail)
  • Three concurrent test processes: each 18/18 pass

…red-CI flakiness

The distributed integration tests (cross_node_smoke, pool_client_batch_put,
peer_service) hard-coded their peer-service / server ports from fixed bases
(52000 / 53000 / 50200). PoolClient and PeerServiceServer bind that port
directly and register it verbatim as the peer_address, so on a shared
self-hosted CI runner the fixed ports collide with concurrent jobs or leftover
servers: the bind fails, remote AllocateSlot/CommitSlot RPCs become
unavailable, and the cross-node Put/Get cases fail (ctest exit code 8). This is
non-deterministic, which is why a PR can pass CI yet the same commit fail once
merged to main. The umbp production code is unaffected; this is purely a test
port-allocation issue.

Ask the kernel for a currently-free ephemeral port (bind :0 -> getsockname ->
close) instead, falling back to a PID-offset random base only if the probe is
unavailable. Mirrors the existing AllocPort() helper in test_umbp_tags.cpp.

Verified in-container with the CI ctest filter: 22/22 pass, including runs with
the old 52000+ range fully occupied and three concurrent test processes.
@isytwu isytwu self-assigned this Jun 22, 2026
@isytwu isytwu changed the title test(umbp): use kernel-assigned ports in distributed tests to fix shared-CI flakiness test(umbp): use kernel-assigned ports in distributed tests to fix CI flakiness Jun 22, 2026
@isytwu isytwu merged commit 6242de7 into ROCm:main Jun 23, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant