Fixes for cuMem compilation and invalid device ordinal

[AtlantaPepsi]

Motivation

cuMem symbols and definitions are currently guarded by pod communication enablement. It's not a long term solution as these two are not always coupled and usage might diverge in future. The separation of these two also fixes existing linking error with cuMemcpyAsync or CUResult in absence of POD_COMM_ENABLED.

Technical Details

• Introduced a separate CUMEM_ENABLED macro for build process as well as header. POD_COMM_ENABLED will have to depend on cuMem enablement as well.
• Minor adjustment for error reporting for cuda runtime calls
• Eliminated unnecessary hipSetDevice: Previously in cuda + MNNVL update & pod presets #241 multiple hipSetDevice are added throughout cuMem allocation and release to make sure context is always initialized. Not all of them were needed, and unconditional invocation also caused error for non GPU executors.

Test Plan

Tested all combination of Makefile flags and made sure compilation/linking succeeded.
Previously on CI machines with more CPU NUMA nodes than GPU devices, certain sweeping presets such as p2p would fail, which is fixed now.

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[Copilot]

Pull request overview

This PR decouples CUDA driver API (cuMem / libcuda) enablement from pod-communication support, and removes unnecessary hipSetDevice calls that could trigger “invalid device ordinal” on non-GPU executors.

Changes:

• Introduce CUMEM_ENABLED as a separate build/header macro and make CUDA pod-comm depend on it.
• Adjust error reporting strings for CUDA runtime vs HIP runtime errors.
• Remove/guard some hipSetDevice calls to avoid invalid device selection in CPU-only contexts.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/header/TransferBench.hpp	Guards CUDA driver API usage behind CUMEM_ENABLED, tweaks error messages, and reduces hipSetDevice usage to avoid invalid device ordinal.
Makefile	Adds DISABLE_CUMEM / CUMEM_ENABLED logic and makes CUDA pod-comm conditional on cuMem availability.

Comments suppressed due to low confidence (1)

src/header/TransferBench.hpp:5090

• When CUMEM_ENABLED is not defined (e.g., TransferBenchCuda with DISABLE_CUMEM=1), this code falls back to hipMemcpyAsync(..., memcpyKind, ...), but memcpyKind is only declared under the HIP_PLATFORM_AMD && HIP_VERSION_MAJOR>=6 block. Under NVCC builds that block is not active, so memcpyKind is undefined and the CUDA build will fail to compile. Define an appropriate memcpy kind for the CUDA fallback path (e.g., device-to-device or default) and/or restructure the preprocessor guards so the fallback does not reference an undeclared variable.

                                  resources.numBytes, stream));
#else
          ERR_CHECK(hipMemcpyAsync(resources.dstMem[dstIdx], resources.srcMem[0], resources.numBytes,
                                   memcpyKind, stream));
#endif

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

This PR decouples CUDA driver API (cuMem/cuMemcpyAsync/CUresult) enablement from pod-communication enablement, and removes/guards several hipSetDevice calls that could trigger “invalid device ordinal” on non-GPU executors.

Changes:

• Add a standalone CUMEM_ENABLED build macro (and link -lcuda) for TransferBenchCuda, with CUDA pod-comm gated on cuMem availability.
• Switch the DMA executor’s CUDA driver copy path to be controlled by CUMEM_ENABLED instead of __NVCC__.
• Adjust error reporting strings for CUDA runtime vs HIP, and avoid hipSetDevice in CPU-memory initialization paths.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/header/TransferBench.hpp	Uses CUMEM_ENABLED to gate CUDA driver APIs, tweaks error messages, and reduces unconditional hipSetDevice usage to avoid invalid ordinals.
Makefile	Introduces DISABLE_CUMEM / CUMEM_ENABLED and makes CUDA pod-comm depend on cuMem enablement; moves -lcuda linkage to the cuMem feature gate.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.