HPX Hierarchical Collectives Benchmark Results #7323
iemAnshuman
started this conversation in
General
Replies: 8 comments 10 replies
|
Correction: the MPI multi-node numbers look like they're on the TCP/ob1 fallback, not InfiniBand — 8.6 ms flat for a 1-element broadcast across 2 nodes is ~1000× IB latency, and MPI's intra-node path is fine (5 µs). So the HPX-vs-MPI multi-node comparison isn't fair yet; I need to force MPI onto UCX/IB and re-run before claiming any speedup. The HPX arity/hierarchical results (all_to_all, all_gather, etc.) are HPX-internal and stand. |
0 replies
|
@iemAnshuman would you have an idea what MPI does when used on one node? Is it that they go through shared memory instead of the networking stack? |
3 replies
|
Yes, exactly — on a single node OpenMPI doesn't touch the network stack at all. It uses a shared-memory transport (the sm/vader BTL, with CMA/xpmem for single-copy transfers between ranks), so an intra-node collective is just memory copies — hence the ~µs numbers. Off-node it switches to a network transport (UCX over InfiniBand here, rc_verbs/mlx4_0). One important correction to my earlier comment, though: I re-measured inter-node with completion-aligned timing (max elapsed across ranks), and the ~8.6 ms wasn't the network either — it was the benchmark recording only the root rank's call-return instead of collective completion. With correct timing, inter-node is also ~0.01 ms across Open MPI 5.0.5/5.0.7 (default or forced-UCX). So the real picture is: intra-node = shared memory, inter-node = InfiniBand, both fast — and the suspicious numbers were a timing-method bug, not the fabric. I'm fixing the benchmark timing and re-running the full sweep before posting any HPX-vs-MPI comparison. |
|
@iemAnshuman I am currently working on an improved collective benchmark with synchronized timing. I will open a PR soon. |
Could you please post the full corrected results? |
Corrected full collective benchmark resultsEnvironment
Validation
MPI coverage in this corrected run: Reported numbers below are milliseconds. For each repeated data point I use the median of the five per-run medians. For MPI, the table reports the best tuned algorithm among the valid algorithm IDs for that operation/size/node count. Across the selected comparable points below, MPI is faster in 63 cases and HPX/LCI is faster in 0 cases; 0 cases are within about 10%. HPX/LCI vs best Open MPI tuned algorithmBroadcast
Reduce
Gather
Scatter
All Gather
All Reduce
All To All
HPX/LCI barrier
Notes
|
2 replies
|
@iemAnshuman Thanks for this analysis. Looks like we have a long way to go here. OTOH, as you pointed out, in some sense this is a apples to oranges comparison. Is there a way to force MPI to go through the networking (not using IP communication buffers)? |
|
Yes, that makes sense. I think the current corrected table is the natural “best tuned MPI” baseline, but it is still not perfectly apples-to-apples because Open MPI can use shared-memory paths for ranks on the same node. If that looks valid, I’ll keep the current results as the normal MPI baseline and add a separate “network-forced MPI” comparison table rather than replacing the corrected results. |
|
@iemAnshuman Thanks for the comparison! I think this large deviation between HPX and MPI was introduced by the changes to the HPX and MPI benchmarks. With the old variants, performance was closer together (except on one node, where you just lose to MPI). We should rerun the old benchmarks and see which one got slower/faster. |
2 replies
|
From my previous results, it looks like the MPI benchmark is now considerably faster. HPX runtimes are similar to before. I need to investigate that ... |
|
Agreed. The corrected run suggests HPX/LCI is roughly stable, while MPI changed a lot after the timing fix and after comparing against the best valid tuned algorithms instead of only I’ll do a smaller controlled rerun rather than another full sweep: old vs corrected timing, |
|
@hkaiser @constracktor Quick update: I ran a compact transport diagnostic, but I do not want to present that as the final apples-to-apples benchmark. The diagnostic used Open MPI forced through UCX/IB:
This confirmed that the forced UCX/IB setup works on 1/2/4 nodes and that MPI stays in the microsecond / low-millisecond range. So the old ~8 ms MPI numbers were from the benchmark/timing issue, not from a TCP fallback. I am going to run a full transport-aligned sweep now, using the same corrected benchmark parameters as the full rerun: 1/2/4 nodes, 20 ranks per node, same sizes, same repetitions, and the valid Open MPI tuned algorithms. I will post the full HPX/LCI vs forced-UCX MPI comparison once that completes. |
2 replies
|
@hkaiser @constracktor I have a full transport-aligned rerun now. For MPI, I forced the Open MPI transport toward UCX/verbs as much as possible using:
The main restricted comparison is MPI with Run: Parsed records:
Ratio convention:
So in the restricted Primary restricted view by operation: MPI forced UCX algorithm=0 vs HPX best arity
This suggests the large gap is not explained simply by MPI using shared-memory or IP-buffer paths. I think the next thing to investigate is what changed between the old and corrected benchmark variants, especially timing methodology and any HPX-side overhead introduced by the benchmark changes. I can also share the full TSVs if useful. |
|
Also, please keep in mind that changes in HPX may have caused this (we have applied changes to the serialization layer, for instance). So please make sure you're using the same HPX commit as well. |
Full comparison run 20260704 (median ms; best-MPI = min over tuned algorithms; ratio = best-HPX / best-MPI)broadcast
reduce
scatter
gather
all_gather
all_reduce
all_to_all
|
0 replies
16-node HPX collectives benchmark@hkaiser @constracktor Results
Historical Medusa MPIThis uses the 168 common operation/payload points and maps old
MPI is much slower on Medusa than Buran here. My estimate is By collective
By layout
Fixed HPX modes
By payload
Complete 292-cell tableOld Medusa MPI is shown where a closest-layout historical point
Configuration
Layouts
Matrix and timing
Completion
TCP exclusion
MPI issues and fixes
Limits
Reproducibility
|
0 replies
16-node pairwise
|
| Item | Value |
|---|---|
| Cluster | Rostam/Buran |
| Nodes | 16 |
| Layouts | 1/2/4/8 localities per node |
| Total localities | 16/32/64/128 |
| Workers per locality | 8/4/2/1 |
| Payload | 4 B to 64 KiB per destination |
| HPX parcelport | LCI over IBV |
| LCI configuration | worker/global; packet=73728; npackets=8192 |
| Sampling | 3 warmups; 10 timed iterations; 5 repetitions |
| Pairwise selection | forced with pairwise_threshold=0 |
| Slurm job | 178549, completed with exit code 0 |
| Acceptance | 960/960 executions; no retries or failures |
Each result below is the median of five per-run medians. The comparison baseline is the fastest non-pairwise HPX result in the same campaign, selected independently for each cell from standard one-shot, standard reused, forced flat, and tree arities 2/3/4.
Summary
| Layout | Pairwise geomean | Best non-pairwise | Speedup | Cells won |
|---|---|---|---|---|
n16_l1_t8 |
426.05 us | 439.90 us | 1.03x | 3/8 |
n16_l2_t4 |
796.05 us | 1,083.94 us | 1.36x | 3/8 |
n16_l4_t2 |
1,557.00 us | 3,043.37 us | 1.95x | 4/8 |
n16_l8_t1 |
3,724.35 us | 12,993.21 us | 3.49x | 6/8 |
| Overall | 1,184.23 us | 2,083.81 us | 1.76x | 16/32 |
Pairwise is slower for the smallest payloads, which supports keeping a size-dependent selection rather than using it unconditionally. Its first winning payload is 4 KiB per destination at 16 and 32 localities, 1 KiB at 64 localities, and 64 B at 128 localities.
At 4 KiB per destination, pairwise is 1.14x, 2.79x, 5.25x, and 13.82x faster across the four layouts. At 64 KiB, the speedups increase to 5.24x, 10.25x, 15.39x, and 22.47x:
| Layout | Pairwise at 4 KiB | Speedup | Pairwise at 64 KiB | Speedup |
|---|---|---|---|---|
n16_l1_t8 |
424 us | 1.14x | 698 us | 5.24x |
n16_l2_t4 |
740 us | 2.79x | 1,530 us | 10.25x |
n16_l4_t2 |
1,381 us | 5.25x | 4,405 us | 15.39x |
n16_l8_t1 |
3,039 us | 13.82x | 15,427 us | 22.47x |
Native MPI was not rerun in this campaign, so these are HPX algorithm comparisons. The MPI results in the earlier post remain historical context, not a same-allocation comparison for this follow-up.
Executable SHA-256: 3a6e1c682da9af60ed14202ae0059c1bf9d0da21555f1c0c6b75fe673c60499f
1 reply
|
Interesting results! Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
@hkaiser @constracktor
Environment
Summary
Modes:
single_use/multi_use(flat),hierarchical_2/4/8(tree, by arity). Eight ops measured: broadcast, reduce, gather, scatter, all_reduce, all_gather, barrier, all_to_all. MPI baseline (OpenMPI tuned, algorithm=0) covers the four it implements. all_to_all is hierarchical-only (PR #7307).Takeaways:
multi_useis the best general-purpose HPX mode.HPX vs MPI Comparison
Broadcast
Reduce
Gather
Scatter
HPX-Only Operations
These operations were not benchmarked with MPI.
All-Reduce
All-Gather
Barrier
All-to-All
Hierarchical-only (no flat variant) — arity vs arity. Added by PR #7307.
Arity-8 vs arity-2 (binary tree) speedup:
Wider trees win everywhere; arity-8 is ~1.4–1.9x faster than binary at 2–4 nodes. (2-node/size-16 arity-2 = 10.5 ms is a low-loop outlier.)
Notes
All reactions