Is cross-GPU thread_scope_system release/acquire on GPU memory officially guaranteed when cudaDevP2PAttrNativeAtomicSupported == 0?

[SteelWen]

Hi,

I have a question about a very specific case of cross-GPU synchronization.

• https://nvidia.github.io/cccl/unstable/libcudacxx/extended_api/memory_model.html
• https://en.cppreference.com/w/cpp/atomic/memory_order

According to the libcudacxx memory model documentation, for an object in GPU memory, a system-scope atomic operation is only guaranteed to be atomic across GPUs if cudaDevP2PAttrNativeAtomicSupported == 1 (or if only one GPU accesses it concurrently).

My question is specifically about plain store/load with memory_order_release / memory_order_acquire, not RMW atomics such as fetch_add, exchange, or compare_exchange.

Test setup

• x and y are both located in GPU0 global memory
• P0 runs on GPU0
• P1 runs on GPU4
• Peer access is enabled between GPU0 and GPU4
• nvidia-smi topo -p2p r reports GPU0 <-> GPU4 as OK
• nvidia-smi topo -p2p a reports GPU0 <-> GPU4 as NS

So this pair appears to support peer read/write, but not peer native atomics.

Test pattern

P0 does:

*(volatile int *)(px) = 1;
cuda::atomic_ref<int, cuda::thread_scope_system> flag(*py);
flag.store(1, cuda::memory_order_release);

P1 does:

cuda::atomic_ref<int, cuda::thread_scope_system> flag(*py);
int r0 = flag.load(cuda::memory_order_acquire);
int r1 = *(volatile int *)(px);

This is a basic MP-style litmus test.
The weak outcome I am checking is:

r0 == 1 && r1 == 0

Observed result

I ran:

• N_INSTANCES = 256
• N_ITERS = 100000

Total observations:

• 25600000

Histogram:

• r0=0, r1=0 : 1359292
• r0=0, r1=1 : 24189531
• r0=1, r1=1 : 51177
• r0=1, r1=0 : 0

So in this run, the weak MP outcome never appeared.

Main question

If cudaDevP2PAttrNativeAtomicSupported == 0, is cross-GPU

• store(..., cuda::memory_order_release) and
• load(..., cuda::memory_order_acquire)

on GPU memory officially:

• guaranteed,
• not guaranteed / implementation-defined,
• or unsupported?

I am mainly looking for an official clarification on the status of this case.

Follow-up question

If this case is not officially guaranteed, why can it still appear to pass reliably in a test like this?

More specifically, is the observed behavior more likely due to:

• a hardware/interconnect property for simple aligned cross-GPU store/load, or
• some driver/runtime support for release/acquire even when cudaDevP2PAttrNativeAtomicSupported == 0?

If there is any official documentation or guidance on this point, I would appreciate a reference.

I am attaching the test source and the log.

MP_system_diffGPU_G0_acq.cu.txt
MP_system_diffGPU_G0_acq.log

Thanks!

[gonzalobg]

Thanks for the issue.
Currently, like you've discovered, CUDA does not officially make this guarantee.
We'll review whether that could change and get back to you.
Leaving this open for now.