[OpenMP][libomptarget] Use two SDMA engines

[jplehr]

Limit the use to two SDMA engines to resolve issues with certain cards.

[llvmbot]

@llvm/pr-subscribers-backend-amdgpu

Author: Jan Patrick Lehr (jplehr)

Changes

Limit the use to two SDMA engines to resolve issues with certain cards.

---

Full diff: https://github.com/llvm/llvm-project/pull/73633.diff

1 Files Affected:

• (modified) openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp (+2-2)

diff --git a/openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp b/openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp
index 208cce90d8e9277..b3eed60d2879d48 100644
--- a/openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp
+++ b/openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp
@@ -161,8 +161,8 @@ Error asyncMemCopy(bool UseMultipleSdmaEngines, void *Dst, hsa_agent_t DstAgent,
       Dst, DstAgent, Src, SrcAgent, Size, NumDepSignals, DepSignals,
       CompletionSignal, (hsa_amd_sdma_engine_id_t)LocalSdmaEngine,
       /*force_copy_on_sdma=*/true);
-  // Increment to use one of three SDMA engines: 0x1, 0x2, 0x4
-  LocalSdmaEngine = (LocalSdmaEngine << 1) % 7;
+  // Increment to use one of two SDMA engines: 0x1, 0x2
+  LocalSdmaEngine = (LocalSdmaEngine << 1) % 3;
   SdmaEngine.store(LocalSdmaEngine, std::memory_order_relaxed);
 
   return Plugin::check(S, "Error in hsa_amd_memory_async_copy_on_engine: %s");

[arsenm]

Keep the comment that lists all 3? Restrict the limit to only the relevant devices?

[jplehr]

Intention is to first unblock testing and then do better.

[jplehr]

After some more digging, I think limiting this to 2 SDMAs being used is actually the correct way to do it.

[jplehr]

The reason why I think this is right approach is that according to documentation, 2 SDMA engines are best suited for d2h/h2d transfers and other available engines are more suited for d2d transfers.
Since some devices only have those for host/device transfers, this is the safe thing to do, and using the engines best suited for those transfers.