Does CUDA VMM teardown serialize across independent GPUs?

[number-eleven11]

When several processes release CUDA virtual memory concurrently, each on its own GPU, the time to finish teardown can be substantially longer than it is for a single process. Is this expected with the open GPU kernel modules?

The workload can be expressed using only the public CUDA Driver API:

for each worker, on a distinct GPU:
    create a CUDA context and reserve a virtual address range
    allocate N device memory handles with cuMemCreate
    map each handle once with cuMemMap and grant access to the local GPU
    synchronize the context and wait at a host barrier
    start timer
    for each mapping:
        cuMemUnmap(address, size)
        cuMemRelease(handle)
    stop timer

Compare the elapsed interval from the first worker starting teardown to the last worker finishing, with one worker and then with several workers. Hold the number and size of allocations per GPU constant across those runs. No peer access, handle exchange, or GPU work is needed during teardown.

The workaround we tested is to keep the total allocated bytes constant, use fewer and larger VMM mappings, and suballocate logical buffers inside them. That reduces the number of unmap/release calls and shortened teardown.

Is there shared serialization between otherwise independent devices on this path? If so, is there a supported way to batch teardown while preserving allocation granularity, or is coarsening the mappings the recommended workaround?

[alok-108]

Yes, this cross-GPU serialization is expected in the current open-gpu-kernel-modules architecture.

Where the cross-GPU serialization comes from

Even though each worker has its own process, CUDA context, and GPU with no peer access, CUDA VMM teardown still enters global, driver-wide kernel locks rather than purely per-GPU locks:

1. Global RM API Lock (g_resServ / rmapiFreeWithSecInfo):
   When cuMemRelease drops the final reference on a cuMemCreate physical allocation handle, it frees the RM memory object via NV_ESC_RM_FREE → rmapiFreeWithSecInfo() → serverFreeResourceTree(&g_resServ, &freeParams) (src/nvidia/src/kernel/rmapi/alloc_free.c). Resource Server alloc/free operations on g_resServ contend on the global RM API lock across all GPUs in the host.
2. UVM external unmap & deferred teardown (kernel-open/nvidia-uvm/uvm_map_external.c):
   cuMemUnmap invokes uvm_api_unmap_external() / uvm_unmap_external(), which unmaps the GPU page tables and runs uvm_deferred_free_object_list() (plus any underlying RM calls), adding kernel/RM lock contention when many small handles are torn down concurrently across processes.

Because each cuMemUnmap + cuMemRelease pair triggers separate user↔kernel transitions that serialize on the global RM API lock, running K workers concurrently scales roughly with K × N lock acquisitions rather than staying constant per GPU.

Is there a batched VMM teardown API?

• No: The public CUDA Driver VMM API has no batched cuMemUnmap / cuMemRelease call—every cuMemCreate handle and cuMemMap mapping is torn down individually.
• Coarsening mappings and suballocating (keeping fewer, larger cuMemCreate/cuMemMap backing chunks and suballocating logical buffers inside them) is the recommended workaround to reduce handle count and global RM lock contention.
• If your workload creates/destroys allocations repeatedly, pooling and reusing cuMemCreate handles (or unmapping/reusing slabs instead of calling cuMemRelease on every logical free) also avoids hitting NV_ESC_RM_FREE under the global RM API lock in the hot path.