Misc. bug: UMA detection incorrectly limits available memory on AMD APUs with large TTM allocations

[AmesianX]

Name and Version

$ ./llama-server --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
version: 7464 (0a0bba0)
built with Clang 20.0.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

./llama-server -m /home/user/Models/lmstudio-community/gpt-oss-120b-GGUF/gpt-oss-120b-MXFP4-00001-of-00002.gguf \
    -t 4 \
    -c 131072 \
    -n 8192 \
    --parallel 10 \
    --cont-batching \
    --jinja \
    --reasoning-format deepseek \
    --chat-template-kwargs '{"reasoning_effort": "low", "reasoning_format": "auto"}' \
    --n-gpu-layers 999 \
    --swa-full \
    --flash-attn on \
    -b 2048 \
    -ub 2048 \
    --no-mmap \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --cache-ram 0 \
    --cache-reuse 256 \
    --defrag-thold 0.1 \
    --temp 1.0 \
    --top-p 1.0 \
    --top-k 0 \
    --min-p 0 \
    --host 0.0.0.0 \
    --port 8888 \
    --api-key "okokok"

Problem description & steps to reproduce

Summary

The UMA (Unified Memory Architecture) detection introduced in PR #17368 incorrectly identifies AMD APUs (such as Strix Halo) as UMA systems and limits the available GPU memory to system RAM's MemAvailable, ignoring the much larger TTM (Translation Table Manager) memory allocation that AMD APUs can utilize.

Environment

• Hardware: AMD Strix Halo APU (gfx1151)
• GPU Memory: 96GB TTM allocation via kernel boot parameters
• System RAM: 32GB
• OS: Ubuntu Linux with kernel 6.14.0
• ROCm Version: 7.9.0
• llama.cpp Version: Latest master (commit 0a0bba0)

Kernel Boot Parameters

amdgpu.gttsize=98304 ttm.pages_limit=25000000 ttm.page_pool_size=12000000

Problem Description

After PR #17368 (DGX Spark: UMA support), the ggml_backend_cuda_device_get_memory() function checks prop.integrated > 0 to detect UMA systems. When detected as UMA, it reads available memory from /proc/meminfo instead of using cudaMemGetInfo().

The issue: AMD APUs report integrated = true, but they can utilize significantly more memory than system RAM through TTM. In my case:

• System RAM available: ~27GB
• Actual GPU memory available via TTM: 96GB

This causes the new llama_params_fit auto-fitting logic to incorrectly calculate that there isn't enough memory, resulting in:

1. Context size being reduced from 131072 to 4096
2. Model layers being offloaded to CPU unnecessarily
3. Significant performance degradation

Logs

Before patch (incorrect behavior):

llama_params_fit_impl: projected to use 64909 MiB of device memory vs. 98304 MiB of free device memory
llama_params_fit_impl: cannot fulfill margin of 1024 MiB, need to reduce device memory by 37767 MiB
llama_params_fit_impl: context size reduced from 131072 to 4096
llama_params_fit_impl: ROCm0 (AMD Radeon Graphics): 36 layers (21 overflowing), 27049 MiB used

After patch (correct behavior):

llama_params_fit_impl: projected to use 64909 MiB of device memory vs. 98304 MiB of free device memory
llama_params_fit_impl: will leave 33092 >= 1024 MiB of free device memory, no changes needed

Root Cause

In ggml/src/ggml-cuda/ggml-cuda.cu, the ggml_backend_cuda_device_get_memory() function:

#if defined(__linux__)
    cudaDeviceProp prop;
    CUDA_CHECK(cudaGetDeviceProperties(&prop, ctx->device));

    bool uma_env = getenv("GGML_CUDA_ENABLE_UNIFIED_MEMORY") != nullptr;
    bool is_uma = prop.integrated > 0 || uma_env;  // <-- AMD APUs hit this

    if (is_uma) {
        // Uses /proc/meminfo which returns ~27GB instead of 96GB TTM
        if (ggml_backend_cuda_get_available_uma_memory(&available_memory_kb, &free_swap_kb) && available_memory_kb > 0) {
            *free = (size_t)available_memory_kb * 1024;
        }
    }
#endif

Proposed Fix

Skip the UMA memory detection for ROCm/HIP builds, as AMD APUs with TTM can report accurate memory via hipMemGetInfo():

-#if defined(__linux__)
+#if defined(__linux__) && !defined(GGML_USE_HIP)
     // Check if this is a UMA (Unified Memory Architecture) system
+    // Note: Skip this for ROCm/HIP - AMD APUs with large TTM allocations (e.g., Strix Halo)
+    // report integrated=true but can use much more memory via TTM than system RAM
     cudaDeviceProp prop;
     CUDA_CHECK(cudaGetDeviceProperties(&prop, ctx->device));
     ...
-#endif // defined(__linux__)
+#endif // defined(__linux__) && !defined(GGML_USE_HIP)

Alternative Solutions

1. Environment variable override: Add an env var like GGML_CUDA_DISABLE_UMA_DETECTION to skip UMA detection
2. TTM-aware detection for AMD: Check /sys/class/drm/card*/device/mem_info_vram_total or similar for AMD GPUs
3. Compare reported vs actual: If cudaMemGetInfo returns significantly more than MemAvailable, prefer cudaMemGetInfo

Verification

With the proposed patch applied:

• amd-smi shows 65GB+ GPU memory usage (correct)
• Model loads with full 128K context
• All 36 layers stay on GPU
• No unnecessary CPU offloading

Related

• PR DGX Spark: UMA support #17368: DGX Spark: UMA support
• PR fix UMA detection on discrete GPUs. #17537: cuda: fix UMA detection on discrete GPUs

System Information

$ amd-smi
+------------------------------------------------------------------------------+
| AMD-SMI 26.1.0      ROCm version: 7.9.0                                      |
|-------------------------------------+----------------------------------------|
| 0000:c5:00.0    AMD Radeon Graphics |                          65XXX/98304 MB |
+-------------------------------------+----------------------------------------+

$ amd-ttm
Current TTM pages limit: 25000000 pages (95.37 GB)
Total system memory: 30.99 GB

$ cat /proc/cmdline
... amdgpu.gttsize=98304 ttm.pages_limit=25000000 ...

First Bad Commit

No response

Relevant log output

[Djip007]

I think the use of /proc/meminfo :: MemAvailable is correct if you define GGML_CUDA_ENABLE_UNIFIED_MEMORY

If you use export GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON ggml-hip backend did not use VRAM/TTM, the memory is allocated on system RAM.

so you have 2 case:

• 1: use export GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON in that case:
  • do not change TTM it is not use,
  • set VRAM as low as possible (512Mo...)
• 2: if you do not use GGML_CUDA_ENABLE_UNIFIED_MEMORY env :
  • 2.1: with never kernel (can remember which one... >6.10/6.11 I think)
    • set VRAM as low as possible (512Mo...)
    • set GTT/TTM as you want.
      => in that case the memory available is VRAM+TTM.
  • 2.2: with older kernel
    • TTM/GTT is not use when gpu memory alloc.
    • Set VRAM as high as needed (96G for max)

So my advise in your case with 6.14 kernel, remove all you kernel param (This isn't even the correct way to do it for recent OS versions (Fedora 43 + systemd), for example.) set VRAM as low as possible, and run llama.cpp+romc like that:

GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON ./llama-server [...]

good luck ;)

[alex-mashin]

* 1: use `export GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON` in that case:
  * do not change TTM it is not use,
  * set VRAM as low as possible (512Mo...)
* 2: if you do not use `GGML_CUDA_ENABLE_UNIFIED_MEMORY` env :
  * 2.1: with never kernel (can remember which one... >6.10/6.11 I think)
    * set VRAM as low as possible (512Mo...)
    * set GTT/TTM as you want.
      => in that case the memory available is VRAM+TTM.

Is there any performance difference between 1 and 2.1?