llama-server and llama-cli hang/crash during RPC tensor upload for large models, while llama-bench succeeds

[visorcraft]

llama-server and llama-cli hang/crash during RPC tensor upload for large models, while llama-bench succeeds

Summary

When loading large models (>100 GiB) over RPC split across two hosts, llama-bench completes successfully (~3-4 minutes), but llama-server hangs indefinitely at the load_tensors phase (never reaches healthy), and llama-cli crashes with a HIP/HSA runtime fault at the same point.

Small models (~46 GiB) load and serve correctly over the same RPC path in all three tools.

Workaround found: -dio (direct I/O) flag resolves the issue completely. See below.

Environment

• Build: llama.cpp master 39e4b1dc (b8102)
• Compiler: GCC 15.2.1
• OS: Fedora 43, kernel 6.18.10-200.fc43.x86_64
• Hardware: 2x AMD Ryzen AI Max+ 395 (Strix Halo, gfx1151), 128 GB RAM each
• ROCm: Tested across ROCm 6.4.2, 7.2, and 7.0 nightlies (see testing matrix below)
• Interconnect: USB4/Thunderbolt direct link between hosts (~9.4 Gbps measured)
• Build flags: -DGGML_HIP=ON -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release -DAMDGPU_TARGETS=gfx1151

Reproduction

Setup

Host A runs rpc-server:

rpc-server -H <host-a-ip> -p 50052

Host B (where model files reside) runs bench/server/cli with --rpc <host-a-ip>:50052 -ts 1/1.

Models tested

| Model | Size | RPC split | llama-bench | llama-server | llama-cli |
|---|---|---|---|---|---|---|
| Qwen3-Coder-Next-Q4_K_M | ~46 GiB | 1/1 | ✅ | ✅ | ✅ |
| MiniMax-M2.5-REAP-139B-A10B-Q8_0 | ~138 GiB | 1/1 | ✅ | ❌ hangs | ❌ crash |
| Qwen3.5-397B-A17B-UD-Q4_K_XL | ~205 GiB | 1/1 | ✅ (tested previously) | ❌ hangs | not tested |

llama-bench (WORKS) — 218 seconds

llama-bench \
  -m /path/to/MiniMax-M2.5-REAP-139B-A10B-Q8_0-00001-of-00004.gguf \
  -ngl 99 -fa on --rpc <host-a-ip>:50052 -ts 1/1 -p 64 -n 16

Output:

| model                          |       size |     params | backend    | ngl | ts           |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------ | --------------: | -------------------: |
| minimax-m2 230B.A10B Q8_0      | 137.77 GiB |   139.15 B | ROCm,RPC   |  99 | 1.00/1.00    |            pp64 |       117.15 ± 10.13 |
| minimax-m2 230B.A10B Q8_0      | 137.77 GiB |   139.15 B | ROCm,RPC   |  99 | 1.00/1.00    |            tg16 |         14.75 ± 0.09 |

llama-server (HANGS without -dio) — never reaches healthy

# ❌ This hangs indefinitely:
llama-server \
  -m /path/to/MiniMax-M2.5-REAP-139B-A10B-Q8_0-00001-of-00004.gguf \
  --host 0.0.0.0 --port 8081 -ngl 99 -fa on \
  --rpc <host-a-ip>:50052 -ts 1/1 --ctx-size 4096 --no-warmup

Server log output stops at:

load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
load_tensors: offloading output layer to GPU
load_tensors: offloading 61 repeating layers to GPU
load_tensors: offloaded 63/63 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   622.76 MiB
load_tensors:        ROCm0 model buffer size = 68285.33 MiB
load_tensors: RPC0[<host-a-ip>:50052] model buffer size = 72173.39 MiB
.......................................

The dots continue appearing very slowly but /health returns 503 indefinitely (tested up to 30+ minutes). Process stays alive at ~90% CPU.

Also tested with --no-mmap — same behavior (hangs at same point, log shows mmap = false).

llama-cli (CRASHES without -dio) — HIP/HSA fault

Same arguments as server but using llama-cli -p "Hello" -n 5. Crashes after ~309 seconds with:

terminate called without an active exception

Stack trace (abbreviated):

libhsa-runtime64.so.1(+0x4e735)
libhsa-runtime64.so.1(+0x491ae)
libhsa-runtime64.so.1(+0x590cc)
libamdhip64.so.6(+0x43d19b)
libamdhip64.so.6(+0x44b48a)
libamdhip64.so.6(+0x44e0e2)
libamdhip64.so.6(+0x45a26f)
...
libggml-hip.so.0(+0x2082c0)
libllama.so.0(_ZN18llama_model_loader13load_all_dataEP12ggml_context...+0x6ef)
libllama.so.0(_ZN11llama_model12load_tensorsER18llama_model_loader+0x4243e)

The crash occurs inside llama_model::load_tensors → llama_model_loader::load_all_data → HIP memory operations.

✅ Workaround: -dio (direct I/O)

Adding the -dio flag to llama-server or llama-cli completely resolves the issue. The server loads the full 138 GiB model over RPC in ~190-196 seconds and reaches health 200 normally.

# ✅ This works:
ROCBLAS_USE_HIPBLASLT=1 llama-server \
  -m /path/to/MiniMax-M2.5-REAP-139B-A10B-Q8_0-00001-of-00004.gguf \
  --host 0.0.0.0 --port 8081 -ngl 99 -fa on \
  --rpc <host-a-ip>:50052 -ts 1/1 --ctx-size 4096 --no-warmup -dio

Log with -dio shows direct_io = true and ROCm_Host buffer instead of CPU_Mapped:

load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = true)
...
load_tensors:        ROCm0 model buffer size = 68285.33 MiB
load_tensors:    ROCm_Host model buffer size =   622.76 MiB
load_tensors: RPC0[<host-a-ip>:50052] model buffer size = 72173.39 MiB

Full -dio testing matrix

Tested with MiniMax-M2.5-REAP-139B-A10B-Q8_0 (138 GiB) over RPC:

ROCm Version	Source	Without -dio	With -dio	Load time	Speed (t/s)
6.4.2 (HIP 6.4.43484-9999)	Host-built llama.cpp	❌ hangs forever	✅ healthy	~196s	15.85 tg, 75.17 pp
7.2	kyuz0/amd-strix-halo-toolboxes container	❌ hangs forever	✅ healthy	~190s	15.88 tg, 49.05 pp
7.0 nightlies	kyuz0/amd-strix-halo-toolboxes container	❌ hangs forever	✅ healthy	~191s	15.83 tg, 71.71 pp

-dio resolves the issue on all tested ROCm versions. The root cause appears to be in the mmap→HIP/HSA tensor upload path, not in the ROCm runtime itself.

Key observations

1. llama-bench succeeds without -dio for the exact same model + RPC config + build, suggesting different model loading codepaths between bench and server/cli.
2. Small models work fine in all three tools over the same RPC link without -dio (tested with ~46 GiB model).
3. The issue appears specific to large models (>~100 GiB total across RPC split) where each device needs to receive 60-100+ GiB of tensor data.
4. --no-warmup and --no-mmap alone do not help — only -dio resolves it.
5. The behavior reproduces identically inside containerized environments (kyuz0/amd-strix-halo-toolboxes) — not specific to host builds.
6. -dio works on all tested ROCm versions (6.4.2, 7.2, 7.0 nightlies).
7. Hardware is Strix Halo UMA (unified memory architecture) — CPU and GPU share 128 GB LPDDR5X. The mmap path likely causes a HIP/HSA allocation collision when both the local mmap and the HIP runtime try to manage the same physical memory pages during large RPC tensor transfers.

Root cause hypothesis

On UMA systems (Strix Halo), the default mmap loading path in llama-server/llama-cli creates a conflict: the kernel maps model files into the process address space via mmap, then the HIP runtime attempts to allocate and transfer these tensors to GPU buffers (local and remote RPC). For large models (>100 GiB), this exhausts the HIP/HSA address space or triggers a page fault loop, causing the hang or crash.

-dio bypasses mmap entirely — it reads model data directly from disk into pre-allocated GPU buffers, avoiding the mmap↔HIP collision.

llama-bench likely uses a simpler or differently-ordered allocation pattern in its loading codepath that avoids triggering this collision even without -dio.

[pwilkin]

Please do not, per the Contributing guidelines, post LLM-generated issue reports. It's of way more use to us when the report actually contains the raw data - crash dumps, stack traces etc. than an LLM generated solution proposal.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.