GPUDirect RDMA (dmabuf) READ from GPU BAR1 fails on Intel Arrow Lake-S / Z890 - writes work, reads fail with CQE error, no PCIe error anywhere; Z890 missing from chipset_info.c

[TensorRaya]

Summary

On Intel Arrow Lake-S (Core Ultra 9 285K, ASUS Pro WS Z890-ACE SE), any
NIC-initiated DMA READ of GPU BAR1 memory registered via the dmabuf path fails
deterministically. DMA WRITEs to the same BAR1 addresses work at full line rate.
The identical software stack on a Threadripper PRO node works in both directions
at 180 Gb/s.

We have reduced this to a single-node loopback reproducer (NIC port 1 reads GPU
memory served by NIC port 0 on the same machine) and traced it to the PCIe
transport between two CPU root ports. The kernel-side dmabuf mapping is provably
correct (verified with bpftrace: the NIC receives the exact BAR1 bus address).
No AER/DevSta error is logged on GPU, NIC, or either root port when the read
fails - the failure is silent at PCIe level and surfaces only as an RDMA
completion error.

Notably, chipset_info.c has entries with workaround flags for consumer Intel
platforms Z690 (Alder Lake) and Z790 (Raptor Lake), but no entry for Z890/Arrow
Lake. We tested the Z790-class rBAR workaround (BAR1 resized to 256M) - it does
not help, so this appears to be a different, new failure class.

System details

Component	Failing node ("atlas")	Working control ("nyx")
CPU	Intel Core Ultra 9 285K (Arrow Lake-S)	AMD Threadripper PRO 7975WX
Board	ASUS Pro WS Z890-ACE SE (BIOS 3002, 2026-01-29)	ASUS Pro WS WRX90E-SAGE
GPU	RTX PRO 4500 Blackwell (GB203GL), driver 580.159.03 (open modules)	RTX PRO 4000/4500 Blackwell
NIC	ConnectX-5 Ex MCX516A (FW 16.36.1010, dual 100G)	ConnectX-5 Ex (same FW)
Kernel	Ubuntu 7.0.0-15-generic, intel_iommu=off	same kernel family, iommu off
CUDA	13.1, perftest built from git HEAD (--use_cuda_dmabuf)	same
Topology	GPU at RP 0000:00:06.0, NIC at RP 0000:00:06.3 (both CPU root ports, no switch)	GPU+NIC behind WRX90 RPs

GPU BAR1: 32 GB resizable BAR at 0xa0_0000_0000 (above 4G).
nvidia-peermem not loadable (inbox kernel lacks ib_register_peer_memory_client);
all tests use the upstream dmabuf path (ib_umem_dmabuf).

Reproducer (single node, deterministic)

# Server: GPU memory via dmabuf on port 0
ib_read_bw -d rocep3s0f0 -x 3 --report_gbits --use_cuda=0 --use_cuda_dmabuf -D 3 -p 20001
# Client: reads from server's GPU memory via port 1 (loopback through switch)
ib_read_bw -d rocep3s0f1 -x 3 --report_gbits -D 3 -p 20001 10.99.8.1

Result on Arrow Lake, 100% of runs:

Failed status-remote operation error (11): wr_id 0 syndrom 0x89

Same command with ib_write_bw (NIC writes into GPU memory): ~59 Gb/s, no error.
Same commands with host RAM instead of GPU memory: 97.6 Gb/s, no error.
Same commands on the Threadripper PRO control node: 180 Gb/s, no error.

Evidence collected

1. Both verb directions fail identically when the NIC must READ GPU memory:

   • ib_read_bw, GPU memory on server (NIC = responder): client sees
     remote operation error, syndrome 0x89.
   • ib_write_bw, GPU memory on client (NIC = requester reading its local
     source buffer from VRAM): local protection error, syndrome 0x51.
   • Interpretation: same failure (NIC DMA read of BAR1), seen from the two
     verb roles. Writes into BAR1 always succeed.

2. The dmabuf mapping handed to the NIC is correct (bpftrace on
   dma_buf_map_attachment return): sg0: dma=0xa000600000, len=131072

   • exactly BAR1 base + allocation offset. reg_user_mr_dmabuf,
     ib_umem_dmabuf_map_pages, mlx5r_umr_update_mr_pas all return 0.

3. No PCIe error is recorded anywhere at failure time. GPU AER (UESta/CESta),
   NIC AER + Device Status (cleared before test), and both root ports
   (00:06.0/00:06.3) stay clean: no UR, no CA, no Completion Timeout, no
   Malformed TLP. The NIC appears to receive a formally valid completion and
   still fails the operation.

4. Exhaustive exclusion matrix (each tested, failure unchanged):

   • message size 64 B .. 128 KB (even a single 64-byte read fails)
   • QP types RC, DC, XRC; rdma_cm and non-CM connection setup
   • IOMMU off (kernel cmdline), ACS disabled on all root ports
   • NoSnoop / RelaxedOrdering toggled off on NIC and GPU (setpci)
   • NVreg_RegistryDwords="ForceP2P=0x11;PeerMappingOverride=1" (clean reboot)
   • BAR1 resized 32G -> 256M (sysfs resource1_resize) - rules out the
     Z790-class HAS_RESIZABLE_BAR_ISSUE workaround as a fix
   • cold vs warm GPU (8.5 GB allocated + active CUDA context)
   • two independent perftest builds (23.04 and git HEAD)
   • NIC FW config identical to working node (mstconfig byte-identical:
     MAX_ACC_OUT_READ=44, P2P_ORDERING_MODE=DEVICE_DEFAULT, etc.)
   • NIC FW health: devlink health clean, no fwtrace faults, AER clean

5. Sporadic NVRM allocation errors on the failing platform only (possibly
   related, not 1:1 correlated with the CQE failure):
   NVRM: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal @ mem_desc.c:1359 via nv_gpu_ops.c:4968
   (NV01_MEMORY_SYSTEM alloc). Never seen on the Threadripper control node.

6. chipset_info.c has no Arrow Lake / Z890 entry. Z690 and Z790 entries
   exist and set PDB_PROP_CL_HAS_RESIZABLE_BAR_ISSUE. Since the failure here
   is independent of BAR1 size, Z890 appears to need a different workaround
   class (or an erratum-level fix from Intel).

7. Reproduced on a second, independent Arrow Lake system ("vega"): Core
   Ultra 7 265K host, RTX PRO 4000 Blackwell, two single-port ConnectX-5 Ex
   (separate cards), running as a Q35 VM with GPU+NIC passthrough (VFIO) on the
   ARL host. Same loopback test: syndrome 0x89 on the GDR read; host-RAM read
   on the same path: 97.5 Gb/s clean. Different CPU SKU, different card
   layout, virtualized: identical failure signature. This makes a
   board/SKU-specific defect unlikely and points at the platform (root
   complex) itself.

Working hypothesis

The Arrow Lake-S root complex does not correctly route NIC-initiated non-posted
(read) TLPs between two CPU root ports when the target is a peer BAR above 4G,
while posted (write) TLPs on the same path route fine. The returned completion
is formally valid at PCIe level (no AER anywhere) but the data/completion
semantics are broken, causing the CX-5 to report a fatal MR/operation error.

Prior art: linux-rdma/perftest#289 shows the same signature (syndrome 0x51,
one direction fails) on a Z690 consumer platform; @drossetti suggested
ACS P2P redirection as the usual cause - excluded here (ACS fully disabled).
That case was never root-caused; the reporter gave up.

Questions for NVIDIA

1. Is GPUDirect RDMA (dmabuf path) expected to work across two root ports on
   consumer/workstation Intel platforms, or is a PCIe-switch topology the only
   supported configuration on these chipsets?
2. Is there a known Arrow Lake / Z890 erratum for inter-root-port non-posted
   P2P routing? Should Z890 get a chipset_info.c entry?
3. Are there driver-level knobs (registry keys, dmabuf mapping variants) that
   change how BAR1 pages are exposed for peer DMA reads that we haven't tried?
4. Any suggestions to capture the failing completion at the NIC (CX-5) level -
   e.g. vendor syndrome decode of 0x89/0x51 in this context - to pin down
   whether the completion carries poisoned data, wrong byte count, or wrong tag?

Attachments

Evidence bundle (lspci topology + -vv dumps, mstconfig, failing runs on both
ARL systems, Threadripper control, bpftrace script + output):
https://gist.github.com/TensorRaya/37c896bc9887767543d55e946e761e6d

---

[TensorRaya]

Evidence bundle is now attached as a public gist: https://gist.github.com/TensorRaya/37c896bc9887767543d55e946e761e6d

Contents:

• atlas-lspci.txt - PCIe topology (lspci -t) plus full lspci -vv of GPU and NIC on the failing node
• atlas-mstconfig.txt - complete ConnectX-5 firmware configuration dump (identical to the working control node)
• atlas-failing-run.log / vega-failing-run.log - full perftest output of the failing loopback runs on both Arrow Lake systems (syndrome 0x89)
• nyx-control-pass.txt - Threadripper PRO control result (180 Gb/s, no errors, same software stack)
• gdr-dmabuf-trace.bt + bpftrace-output.txt - the bpftrace script used to verify the dmabuf sg_table handed to the NIC, and its output showing the correct BAR1 bus address at failure time

Happy to run additional instrumented tests on either system if that helps narrow it down.

[drossetti]

@TensorRaya

1. Is GPUDirect RDMA (dmabuf path) expected to work across two root ports on consumer/workstation Intel platforms, or is a PCIe-switch topology the onlysupported configuration on these chipsets?

Intel client CPUs are known to only support PCIe P2P writes.
As you mentioned, one workaround is to connect your peer PCIe devices via a PCIe switch.

2. Is there a known Arrow Lake / Z890 erratum for inter-root-port non-posted P2P routing?

I do not know. I suggest to work with your vendor.

Should Z890 get a chipset_info.c entry?

Possibly, though I don't believe that particular chipset DB is designed to capture support for P2P features.

3. Are there driver-level knobs (registry keys, dmabuf mapping variants) that change how BAR1 pages are exposed for peer DMA reads that we haven't tried?

The HW limitation is fundamental. There are no easy solutions.

4. Any suggestions to capture the failing completion at the NIC (CX-5) level - e.g. vendor syndrome decode of 0x89/0x51 in this context - to pin down whether the completion carries poisoned data, wrong byte count, or wrong tag?

A PCIe bus analyzer may be able to show something, though it depends on how the CPU root complex handles the P2P read.

[TensorRaya]

Thank you @drossetti — that settles it, and it's genuinely valuable to have "Intel client CPUs are known to only support PCIe P2P writes" stated explicitly somewhere public and searchable. As far as we can tell, this limitation isn't documented in any public Intel client datasheet, so this issue is now effectively the reference for it.

Closing the loop for future readers who land here with the same symptom (IBV_WC_LOC_PROT_ERR, vendor syndrome 0x89/0x51, writes fine / reads failing between root ports on Intel client platforms — Alder Lake through Arrow Lake):

• Root cause: fundamental root-complex limitation on Intel client CPUs — non-posted P2P (reads) between root ports is not supported. Not a driver bug, not fixable in software/BIOS.
• Prior art: one direction bandwidth testing fail with GPUdirect linux-rdma/perftest#289 shows the identical failure on Z690 (Sep 2024, never root-caused — this is the same issue).
• Workarounds: (1) place GPU and NIC behind a shared PCIe switch, or (2) use a workstation/server platform — we verified Threadripper PRO (WRX90) routes P2P reads at full speed with the identical software stack.
• Practical impact for NCCL users: minimal. NCCL detects the failing read path and falls back to host staging; in our 100 GbE fabric the throughput difference is negligible for collectives.

We'll leave our reproducer, bpftrace scripts and the full exclusion matrix in the linked gist for anyone who needs to confirm the diagnosis on their own hardware. From our side this can be closed — though a note in the GPUDirect docs about client-platform limitations would save the next person three days. Thanks again.

[TensorRaya]

@drossetti One follow-up question, if you don't mind — less about the bug, more about whether fixing it would even matter for setups like ours. I couldn't find solid public guidance on this, so your input would be valuable for others landing here from search.

Setup recap: consumer Intel (Arrow Lake) nodes, ConnectX-5 2×100G RoCEv2, RTX PRO (Blackwell) GPUs, GPU and NIC on different root ports. The fabric is a UniFi ECS-Aggregation (48×25G SFP28 + 6×100G QSFP28) running SONiC, configured lossless for RoCE: PFC on priority 3, DSCP-based classification (CS3), ECN/WRED, PFC watchdog. P2P writes work (~59 Gbit/s into GPU BAR1 measured with perftest), P2P reads fail as established above, so the send side cannot use GPUDirect and NCCL falls back to staging through host memory (pinned buffers, CPU/RAM in the path).

Our numbers with that host-staging send path (NCCL 2.30.7, 3 nodes over the lossless 100G fabric):

• point-to-point: 94.8-95.2 Gbit/s uni-directional, ~188 Gbit/s full duplex on one cable pair (both directions genuinely simultaneous, i.e. combined send+receive at wire speed)
• dual-rail bidirectional: 214-220 Gbit/s per node pair
• all_gather/all_reduce busbw: 91-93 Gbit/s at 512MB messages
• point-to-point latency ~2.4 µs
• The receive side does use GDR (P2P writes work), so this is a mixed path: NIC→GPU direct on receive, GPU→host→NIC on send.

The question: for GPU-to-GPU transfers, how much is actually left on the table by staging the send side through CPU/RAM instead of reading GPU memory directly — at 100G per rail? Even full-duplex traffic (simultaneous send+receive, ~188G through one NIC) currently sustains wire speed with the staged send path. Is the expected win from working P2P reads mainly latency, CPU utilization, or memory-bandwidth pressure under concurrent compute load — or does send-side GDR only start to matter at 400G-class NICs where the staging copy itself becomes the bottleneck? We're trying to decide whether moving to a platform with working P2P reads (server chipset or PCIe switch) is worth real money for a homelab-scale cluster, or whether host-staged sends at these numbers are already close to what full GDR would deliver. If the honest answer is "at your scale you'd barely notice", that's just as valuable to have on record.