It mostly seems to work per nvidia-smi and p2pBandwidthLatencyTest, but...

[emerth]

A pair of MSI Shadow 5080.

nvidia-smi topo -p2p r
        GPU0    GPU1
 GPU0   X       OK
 GPU1   OK      X

nvidia-smi topo -p2p w
        GPU0    GPU1
 GPU0   X       OK
 GPU1   OK      X

nvidia-smi topo -p2p p
        GPU0    GPU1
 GPU0   X       OK
 GPU1   OK      X

But....

nvidia-smi topo -p2p a
        GPU0    GPU1
 GPU0   X       NS
 GPU1   NS      X

---

lspci -vvv
01:00.0 VGA compatible controller: NVIDIA Corporation Device 2c02 (rev a1) (prog-if 00 [VGA controller])
...
                DevCap2: Completion Timeout: Range AB, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Via message, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq+ OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
...
03:00.0 VGA compatible controller: NVIDIA Corporation Device 2c02 (rev a1) (prog-if 00 [VGA controller])
...
                DevCap2: Completion Timeout: Range AB, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Via message, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq+ OBFF Disabled,
                         AtomicOpsCtl: ReqEn-

It seems like atomics are half on, half off?

Pytorch script that uses all available GPU for CNN training: occasionally it actually uses both GPUs, most of the time it uses only GPU 1, and GPU 0 idle.

[emerth]

Closing: PEBKAC. Platform does not properly support atomics.