[GSD-13226] zeContextMakeMemoryResident on a peer device allocates a same-sized host (GTT) buffer — 55 GiB of host RAM for a 2-GPU inference server, and it turns out not to be needed

[TSUMUGI-XE]

Filing this at the request of @pbalcer in intel/llvm#22873, who noted that
"it's not expected that an equivalent host physical allocation is created for every GPU device
allocation."

First — thank you. The explanation in that thread (that the runtime makes memory resident on all
devices to enable efficient P2P) is what told us where to look, and it turned out to be actionable
on our side immediately. More on that in section 4, because I think the result is the most useful
thing I can contribute here.

1. What happens

Making an allocation owned by device A resident on device B causes a host-side allocation of the
same size, accounted as GPU-driver memory (GTT). It is not visible in VmRSS, ps, or free's
process accounting.

Raw Level Zero, no SYCL involved. 1 GiB allocated on device 0, per-device /proc/<pid>/fdinfo:

after zeMemAllocDevice(1 GiB, dev0)     0000:0e:00.0  VRAM 0.00  GTT 0.00
                                        0000:12:00.0  VRAM 0.00  GTT 0.00
after MakeMemoryResident(ptr, dev0)     0000:0e:00.0  VRAM 1.00  GTT 0.00
                                        0000:12:00.0  VRAM 0.00  GTT 0.00
after MakeMemoryResident(ptr, dev1)     0000:0e:00.0  VRAM 1.00  GTT 0.00
                                        0000:12:00.0  VRAM 0.00  GTT 1.00   <-- appears here

Control arms, same program, varying only the residency target:

ctx devices | MakeMemoryResident on | device1 GTT | nr_gpu_active
------------+-----------------------+-------------+---------------
     1      | dev0                  |   0.00 GiB  |   +0.00 GiB
     2      | dev0 only             |   0.00 GiB  |   +0.00 GiB
     2      | dev0 + dev1           |   1.00 GiB  |   +1.00 GiB

zeMemAllocDevice is lazily committed, so zeContextMakeMemoryResident is required before any of
this is observable — an earlier version of my test omitted it and reported a clean "no mirror" for
both arms, which was wrong.

2. Environment

• 2× Intel Arc Pro B70 (Battlemage / BMG G31), under a PLX switch, ACS redirect off
• Linux 7.1.1 (xe)
• Reproduces on compute-runtime 26.18.38308, 26.22.38646 and 26.27.39122 (latest) — same
  +8.07 GiB for 4 GiB×2, verified the intended .so was loaded via /proc/self/maps, not just installed
• P2P works well on this box: 28.6 GB/s device-to-device, ze_peer agrees

3. Scale

Host cost tracks the total device allocation, 1:1. On a tensor-parallel inference server across the
two GPUs (Qwen3-Next-80B-A3B, -tp 2, model entirely in VRAM):

--gpu-memory-utilization 0.90  ->  55.1 GiB host RAM
                          0.80  ->  50.7 GiB host RAM
   (proportional to the VRAM pool, not to model size)

On a 123 GiB host that is about half the machine. It is hard to diagnose because it does not appear
in the usual counters; the two witnesses that do see it are
/proc/<pid>/fdinfo/<drm fd> → drm-total-gtt, and /sys/devices/system/node/node0/vmstat →
nr_gpu_active. nr_foll_pin_* and Mlocked stay flat, so it is driver-owned system memory rather
than pinned user pages.

4. The part I think matters: we skipped peer residency entirely, and nothing needed it

Following the explanation in #22873, I wrote a small LD_PRELOAD shim that records the owning device
of each zeMemAllocDevice result and skips zeContextMakeMemoryResident whenever the target device
is not the owner (returning ZE_RESULT_SUCCESS without calling through).

Then I ran the same 2-GPU tensor-parallel server twice, identical in every respect except one
environment variable that enables the skip:

	residency as-is	peer residency skipped
server starts	yes, ~140 s	yes, ~135 s
host RAM (nr_gpu_active) peak	55.12 GiB	1.55 GiB
model loading	20.58 GiB / 67.8 s	20.58 GiB / 66.3 s
available KV cache	6.14 GiB	6.14 GiB
KV cache size	508,450 tokens	507,904 tokens
max concurrency @32k	15.52x	15.50x
collectives (xccl, world_size=2)	OK	OK
generated text (2 prompts, greedy)	—	byte-identical

53.6 GiB of host RAM disappears and nothing else changes — same VRAM, same KV cache, same startup
time, same collective init, identical output. A separate microbenchmark agrees on the P2P side:
device-to-device copies still run at 28.59 / 28.58 GB/s with peer residency skipped, i.e. exactly the
same as with it.

I want to be careful about what this does and does not show. It is one workload, one smoke test, and
this application never calls the SYCL explicit peer-access API — so I am not claiming peer residency
is never necessary. What it does show is that on this hardware, for a workload that actively uses
P2P, the host-side allocation backing peer residency was not required for any of the functionality it
appears to exist for. Which lines up with your "not expected" — it looks like the host buffer is a
side effect rather than the mechanism.

5. Why we cannot just use the SYCL-side mitigation

@pbalcer pointed at intel/llvm#21889 and the
SYCL_UR_L0_RESTRICT_USM_RESIDENCY_TO_P2P environment variable. That is clearly the right direction,
and having read the PR I expect it would help us, since it ties residency to explicit
urUsmP2PEnablePeerAccessExp calls rather than to hardware P2P capability, and our stack never makes
those calls.

Unfortunately we cannot reach it yet. #21889 merged into the sycl branch on 2026-05-18 and is in the
nightlies, and the variable is documented in sycl/doc/EnvironmentVariables.md — but it is not in any
released oneAPI DPC++ we can install. strings over
libur_adapter_level_zero*.so in our runtime finds no occurrence of it.

So for anyone on a released toolchain, the driver-side behaviour is currently the only place this can
be fixed — which is part of why I think it is worth addressing here even with the SYCL-side change
already in flight.

6. Related reports

• llama.cpp issue #22116 — same behaviour from a different application, with per-device
  drm-total-gtt numbers. Closed as not_planned.
• [XPU] Use Level Zero zeMemAllocDevice to avoid host memory shadowing pytorch/pytorch#180145 — same symptom; a zeMemAllocDevice substitution was proposed and closed with
  "driver update is the real fix". Worth noting that the verification there used VmRSS, which moves
  by ~5 MiB while ~8 GiB of host RAM is consumed, so it cannot distinguish fixed from not-fixed.

7. Happy to help

I can share the reproducers (raw Level Zero, SYCL/PyTorch, and the residency-skipping shim), and I am
glad to run variants on this box — different allocation sizes, more than two devices, explicit
per-device contexts, or any instrumented driver build you want tested. Two B70s under a PLX switch
with working P2P is a reasonably convenient configuration to poke at.

Thanks again for the pointer in #22873 — it turned a mystery we had been chasing for days into
something we could act on the same evening.