Fix collective mismatch hangs in multi-rank error paths

[nileshnegi]

Motivation

This PR builds on top of #269, so this PR will be rebased after the other is merged.

Three places used ERR_CHECK inside rank-specific guards preceding (or within loops containing) subsequent Broadcast collectives. A failure on one rank would cause it to early-return before the collective, leaving all other ranks blocked indefinitely.

Also, fixes segfault and improves error diagnostics in multi-rank error paths

Technical Details

• ExchangeMemory (POD path): capture hipSetDevice + hipMemExportToShareableHandle errors into exportErr and broadcast it after the handle broadcast, so all ranks see the failure and return together. Apply the same treatment to the import-side sequence (hipSetDevice, hipMemImport, hipMemAddressReserve, hipMemMap, hipMemSetAccess), broadcasting importErr before returning.
• BroadcastTfrResult: fix non-root path broadcasting only the first element of perIterCUs[i] regardless of setSize; replace the if-guard with a loop matching the root's per-element sends.
• NIC executor QP setup: replace ERR_CHECK inside the per-QP loop with captured ErrType values broadcast from srcMemRank and dstMemRank, respectively, so a TransitionQpToRtr/Rts failure on any rank does not leave others blocked on the next iteration's Broadcast when qpCount > 1.
• CheckMultiNodeConfigConsistency: null out vector members in DataOptions (fillPattern, fillCompress) and GfxOptions (cuMask, prefXccTable) before sizeof-broadcast. Broadcasting a struct containing std::vector transmits rank 0's heap pointers to other ranks; on scope exit the receiving rank calls free(remote_ptr) causing a segfault. These fields are permitted to differ across ranks and are not compared by the consistency check.
• NIC QP transition: replace bare ErrType broadcast with a POD QpTransitionResult {errType, rtrFailed} struct so the failing stage (RTR vs RTS) is preserved and reported accurately. Previously all failures reported "to RTS" regardless of which transition failed. Adds static_assert(is_trivially_copyable) to enforce broadcast safety.

Test Plan

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[Copilot]

Pull request overview

This PR prevents multi-rank hangs caused by rank-local failures returning early before subsequent collectives, ensuring all ranks observe the failure and exit together (notably in NIC QP setup and POD fabric-handle exchange). It also includes build-system/toolchain updates that improve ROCm/HIP (+ AMD-SMI) feature detection and configuration.

Changes:

• Synchronize error handling across ranks in NIC QP transition setup and POD fabric-handle export/import to avoid collective mismatch hangs.
• Fix BroadcastTfrResult non-root path to receive all CU pairs (not just the first element).
• Update CMake/Makefile/toolchain logic for ROCm path detection and compile-time API probes (HIP fabric + AMD-SMI fabric APIs), plus minor docs/changelog tweaks.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
toolchain-linux.cmake	Refactors ROCm/compiler auto-detection and default build-type flags in a Linux toolchain file.
src/header/TransferBench.hpp	Adds rank-synchronized error propagation around collectives; fixes CU broadcast loop; updates AMD-SMI BDF usage.
Makefile	Updates HIP compiler fallback selection and switches POD/AMD-SMI enablement to compile-time API probes.
CMakeLists.txt	Raises CMake minimum, improves ROCm/GPU target handling, and adds compile-time probes for POD/AMD-SMI support.
CHANGELOG.md	Normalizes preset naming/formatting entries.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.