Rocm 7.2 Chaos with Memory

[damianhunziker]

Hi!

Im trying to get Rocm running on Strix Halo GMKTEc Evo-x2 with Ubuntu 24.4, Kernel 6.18.7-061807-generic and the firmware of 26.1.26.

I have set the UMA Memory in Bios to 2 GB as its using the CPU memory anyhow. Now I can let run big models like GPT OSS 120b gpt-oss-120b-Q8_0. However when i run it it shows in htop memory usage of 65 GB and in nvtop it doesnt shows the GPU is using memory but the GPU is running on 100%. Is this how it should be? It doesnt seems to be faster at least not with QWEN 80b. The pp seems to be the exact same speed as with Vulkan and the tg is much slower. It seems weird that its not using the GPU memory and gemini tells me that it would now shift the data around between CPU and GPU and therefor would be slower. Would you please tell me if that is how it should be and can i somehow optimize it so i get the full speed rocm should provide?

Otherwise would you please tell me which setup i need to use to get the full ROCM speed in tg and pp? The tg of QWEN 80 b is now one third of Vulkan, the pp is the same...

[Server] POST /api/v1/chat/completions - 200
[Server PRE-ROUTE] POST /api/v1/chat/completions
[Server DEBUG] No tools in request
[Server] Auto-loading model: extra.gpt-oss-120b-Q8_0.gguf
[Router] Effective settings: ctx_size=4096, llamacpp_backend=rocm, llamacpp_args=-fa on --no-mmap -c 81920 --mlock -ub 1024 -fa on -sm row --temp 0.2 --top-k 20 --top-p 0.95 --min-p 0 -n 16384 --no-context-shift --flash-attn on --no-mmap -ub 1024 --no-kv-offload --no-warmup
[Router] Loading model: extra.gpt-oss-120b-Q8_0.gguf (checkpoint: /data/models/gpt-oss-120b-Q8_0.gguf, recipe: llamacpp, type: llm, device: gpu)
[Router] Slot limit reached for type llm, evicting LRU: extra.Qwen_Qwen3-Next-80B-A3B-Instruct-Q8_0-00001-of-00003.gguf
[Router] Evicting model: extra.Qwen_Qwen3-Next-80B-A3B-Instruct-Q8_0-00001-of-00003.gguf
[LlamaCpp] Unloading model...
srv operator(): operator(): cleaning up before exit...
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | - ROCm0 (8060S Graphics) | 128000 = 37729 + (81274 = 80562 + 0 + 712) + 8995 |
llama_memory_breakdown_print: | - Host | 2897 = 315 + 2221 + 360 |
[ProcessManager] Process terminated, waiting for GPU driver cleanup...
[LlamaCpp] Unloading model...
[Router] Evicted model: extra.Qwen_Qwen3-Next-80B-A3B-Instruct-Q8_0-00001-of-00003.gguf
[Router] Creating LlamaCpp backend
[Router] Starting backend (this may take a moment)...
[LlamaCpp] Loading model: extra.gpt-oss-120b-Q8_0.gguf
[LlamaCpp] Per-model settings: ctx_size=4096, llamacpp_backend=rocm, llamacpp_args=-fa on --no-mmap -c 81920 --mlock -ub 1024 -fa on -sm row --temp 0.2 --top-k 20 --top-p 0.95 --min-p 0 -n 16384 --no-context-shift --flash-attn on --no-mmap -ub 1024 --no-kv-offload --no-warmup
[LlamaCpp] Using rocm version from config: b1162
[LlamaCpp] Found llama-server at: /home/damian/.cache/lemonade/bin/llama/rocm/llama-server
[LlamaCpp] Using GGUF: /data/models/gpt-oss-120b-Q8_0.gguf
llama-server will use port: 8001
[LlamaCpp] Using backend: rocm
[LlamaCpp] Use GPU: true
[LlamaCpp] Adding custom arguments: -fa on --no-mmap -c 81920 --mlock -ub 1024 -fa on -sm row --temp 0.2 --top-k 20 --top-p 0.95 --min-p 0 -n 16384 --no-context-shift --flash-attn on --no-mmap -ub 1024 --no-kv-offload --no-warmup
[LlamaCpp] Starting llama-server...
[LlamaCpp] Setting LD_LIBRARY_PATH=/home/damian/.cache/lemonade/bin/llama/rocm:/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu:
Waiting for llama-server to be ready...
ggml_cuda_init: found 1 ROCm devices:
Device 0: Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
DEPRECATED: argument '-fa' specified multiple times, use comma-separated values instead (only last value will be used)
DEPRECATED: argument '--no-mmap' specified multiple times, use comma-separated values instead (only last value will be used)
DEPRECATED: argument '-ub' specified multiple times, use comma-separated values instead (only last value will be used)
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 1 (ad8d85b) with Clang 22.0.0 for Linux
system info: n_threads = 16, n_threads_batch = 16, total_threads = 32

system_info: n_threads = 16 (n_threads_batch = 16) / 32 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : LLAMAFILE = 1 | REPACK = 1 |

Running without SSL
init: using 31 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model '/data/models/gpt-oss-120b-Q8_0.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected to use 60648 MiB of device memory vs. 123415 MiB of free device memory
llama_params_fit_impl: will leave 62766 >= 1024 MiB of free device memory, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.47 seconds
llama_model_load_from_file_impl: using device ROCm0 (Radeon 8060S Graphics) (0000:c5:00.0) - 123415 MiB free
llama_model_loader: loaded meta data with 39 key-value pairs and 687 tensors from /data/models/gpt-oss-120b-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = gpt-oss
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Gpt Oss 120b
llama_model_loader: - kv 3: general.basename str = gpt-oss
llama_model_loader: - kv 4: general.size_label str = 120B
llama_model_loader: - kv 5: general.license str = apache-2.0
llama_model_loader: - kv 6: general.tags arr[str,2] = ["vllm", "text-generation"]
llama_model_loader: - kv 7: gpt-oss.block_count u32 = 36
llama_model_loader: - kv 8: gpt-oss.context_length u32 = 131072
llama_model_loader: - kv 9: gpt-oss.embedding_length u32 = 2880
llama_model_loader: - kv 10: gpt-oss.feed_forward_length u32 = 2880
llama_model_loader: - kv 11: gpt-oss.attention.head_count u32 = 64
llama_model_loader: - kv 12: gpt-oss.attention.head_count_kv u32 = 8
llama_model_loader: - kv 13: gpt-oss.rope.freq_base f32 = 150000.000000
llama_model_loader: - kv 14: gpt-oss.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 15: gpt-oss.expert_count u32 = 128
llama_model_loader: - kv 16: gpt-oss.expert_used_count u32 = 4
llama_model_loader: - kv 17: gpt-oss.attention.key_length u32 = 64
llama_model_loader: - kv 18: gpt-oss.attention.value_length u32 = 64
llama_model_loader: - kv 19: gpt-oss.attention.sliding_window u32 = 128
llama_model_loader: - kv 20: gpt-oss.expert_feed_forward_length u32 = 2880
llama_model_loader: - kv 21: gpt-oss.rope.scaling.type str = yarn
llama_model_loader: - kv 22: gpt-oss.rope.scaling.factor f32 = 32.000000
llama_model_loader: - kv 23: gpt-oss.rope.scaling.original_context_length u32 = 4096
llama_model_loader: - kv 24: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 25: tokenizer.ggml.pre str = gpt-4o
llama_model_loader: - kv 26: tokenizer.ggml.tokens arr[str,201088] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 27: tokenizer.ggml.token_type arr[i32,201088] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 28: tokenizer.ggml.merges arr[str,446189] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 29: tokenizer.ggml.bos_token_id u32 = 199998
llama_model_loader: - kv 30: tokenizer.ggml.eos_token_id u32 = 200002
llama_model_loader: - kv 31: tokenizer.ggml.padding_token_id u32 = 199999
llama_model_loader: - kv 32: tokenizer.chat_template str = {#-\n In addition to the normal input...
llama_model_loader: - kv 33: general.quantization_version u32 = 2
llama_model_loader: - kv 34: general.file_type u32 = 7
llama_model_loader: - kv 35: quantize.imatrix.file str = gpt-oss-120b/gpt-oss-120b.dat
llama_model_loader: - kv 36: quantize.imatrix.dataset str = calibration_datav3.txt
llama_model_loader: - kv 37: quantize.imatrix.entries_count u32 = 288
llama_model_loader: - kv 38: quantize.imatrix.chunks_count u32 = 122
llama_model_loader: - type f32: 433 tensors
llama_model_loader: - type q8_0: 146 tensors
llama_model_loader: - type mxfp4: 108 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q8_0
print_info: file size = 59.02 GiB (4.34 BPW)
load: 0 unused tokens
load: setting token '<|message|>' (200008) attribute to USER_DEFINED (16), old attributes: 8
load: setting token '<|start|>' (200006) attribute to USER_DEFINED (16), old attributes: 8
load: setting token '<|constrain|>' (200003) attribute to USER_DEFINED (16), old attributes: 8
load: setting token '<|channel|>' (200005) attribute to USER_DEFINED (16), old attributes: 8
load: printing all EOG tokens:
load: - 199999 ('<|endoftext|>')
load: - 200002 ('<|return|>')
load: - 200007 ('<|end|>')
load: - 200012 ('<|call|>')
load: special_eog_ids contains both '<|return|>' and '<|call|>', or '<|calls|>' and '<|flush|>' tokens, removing '<|end|>' token from EOG list
load: special tokens cache size = 21
load: token to piece cache size = 1.3332 MB
print_info: arch = gpt-oss
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 131072
print_info: n_embd = 2880
print_info: n_embd_inp = 2880
print_info: n_layer = 36
print_info: n_head = 64
print_info: n_head_kv = 8
print_info: n_rot = 64
print_info: n_swa = 128
print_info: is_swa_any = 1
print_info: n_embd_head_k = 64
print_info: n_embd_head_v = 64
print_info: n_gqa = 8
print_info: n_embd_k_gqa = 512
print_info: n_embd_v_gqa = 512
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 2880
print_info: n_expert = 128
print_info: n_expert_used = 4
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 2
print_info: rope scaling = yarn
print_info: freq_base_train = 150000.0
print_info: freq_scale_train = 0.03125
print_info: freq_base_swa = 150000.0
print_info: freq_scale_swa = 0.03125
print_info: n_ctx_orig_yarn = 4096
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: model type = 120B
print_info: model params = 116.83 B
print_info: general.name = Gpt Oss 120b
print_info: n_ff_exp = 2880
print_info: vocab type = BPE
print_info: n_vocab = 201088
print_info: n_merges = 446189
print_info: BOS token = 199998 '<|startoftext|>'
print_info: EOS token = 200002 '<|return|>'
print_info: EOT token = 199999 '<|endoftext|>'
print_info: PAD token = 199999 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: EOG token = 199999 '<|endoftext|>'
print_info: EOG token = 200002 '<|return|>'
print_info: EOG token = 200012 '<|call|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = true)
load_tensors: offloading output layer to GPU
load_tensors: offloading 35 repeating layers to GPU
load_tensors: offloaded 37/37 layers to GPU
load_tensors: ROCm0 model buffer size = 58246.00 MiB
load_tensors: ROCm0_Split model buffer size = 1605.70 MiB
load_tensors: ROCm_Host model buffer size = 586.82 MiB
Still waiting for llama-server...
.
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|return|> logit bias = -inf
common_init_result: added <|call|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max = 4
llama_context: n_ctx = 81920
llama_context: n_ctx_seq = 81920
llama_context: n_batch = 2048
llama_context: n_ubatch = 1024
llama_context: causal_attn = 1
llama_context: flash_attn = enabled
llama_context: kv_unified = true
llama_context: freq_base = 150000.0
llama_context: freq_scale = 0.03125
llama_context: n_ctx_seq (81920) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_context: ROCm_Host output buffer size = 3.07 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 81920 cells
llama_kv_cache: CPU KV buffer size = 2880.00 MiB
llama_kv_cache: size = 2880.00 MiB ( 81920 cells, 18 layers, 4/1 seqs), K (f16): 1440.00 MiB, V (f16): 1440.00 MiB
llama_kv_cache_iswa: creating SWA KV cache, size = 1536 cells
llama_kv_cache: CPU KV buffer size = 54.00 MiB
llama_kv_cache: size = 54.00 MiB ( 1536 cells, 18 layers, 4/1 seqs), K (f16): 27.00 MiB, V (f16): 27.00 MiB
sched_reserve: reserving ...
sched_reserve: ROCm0 compute buffer size = 796.75 MiB
sched_reserve: ROCm_Host compute buffer size = 337.29 MiB
sched_reserve: graph nodes = 2024
sched_reserve: graph splits = 74
sched_reserve: reserve took 77.65 ms, sched copies = 1
srv load_model: initializing slots, n_slots = 4
slot load_model: id 0 | task -1 | new slot, n_ctx = 81920
slot load_model: id 1 | task -1 | new slot, n_ctx = 81920
slot load_model: id 2 | task -1 | new slot, n_ctx = 81920
slot load_model: id 3 | task -1 | new slot, n_ctx = 81920
srv load_model: prompt cache is enabled, size limit: 8192 MiB
srv load_model: use --cache-ram 0 to disable the prompt cache
srv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
init: chat template, example_format: '<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2026-01-31

Reasoning: medium

Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions

You are a helpful assistant<|end|><|start|>user<|message|>Hello<|end|><|start|>assistant<|channel|>final<|message|>Hi there<|end|><|start|>user<|message|>How are you?<|end|><|start|>assistant'
srv init: init: chat template, thinking = 0
main: model loaded
main: server is listening on http://127.0.0.1:8001
main: starting the main loop...
srv update_slots: all slots are idle
llama-server is ready!
[LlamaCpp] Model loaded on port 8001
[Router] Backend started successfully
[Router] Model loaded successfully. Total loaded: 1
[Server] Model loaded successfully: extra.gpt-oss-120b-Q8_0.gguf
[Server] POST /api/v1/chat/completions - srv params_from_: Chat format: GPT-OSS
slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> ?min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
slot update_slots: id 3 | task 0 | new prompt, n_ctx_slot = 81920, n_keep = 16, task.n_tokens = 74
slot update_slots: id 3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 10, batch.n_tokens = 10, progress = 0.135135
slot update_slots: id 3 | task 0 | n_tokens = 10, memory_seq_rm [10, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 74, batch.n_tokens = 64, progress = 1.000000
slot update_slots: id 3 | task 0 | prompt done, n_tokens = 74, batch.n_tokens = 64
slot init_sampler: id 3 | task 0 | init sampler, took 0.01 ms, tokens: text = 74, total = 74
slot print_timing: id 3 | task 0 |
prompt eval time = 1183.59 ms / 74 tokens ( 15.99 ms per token, 62.52 tokens per second)
eval time = 5395.71 ms / 228 tokens ( 23.67 ms per token, 42.26 tokens per second)
total time = 6579.29 ms / 302 tokens
slot release: id 3 | task 0 | stop processing: n_tokens = 301, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
200 OK
[Server DEBUG] Response message does NOT contain tool_calls
[Server DEBUG] Message content: Ich bin ein KI‑Modell der GPT‑4‑Familie. OpenAI hat die genaue Größe von GPT‑4 nicht veröffentlicht – die offizielle Angabe zu den Parameter‑Zahlen bleibt also vertraulich. Allerdings

=== Telemetry ===
Input tokens: 74
Output tokens: 228
TTFT (s): 1.18
TPS: 42.26

[Server] POST /api/v1/chat/completions - 200

[danielholanda]

@benrichard-amd This seems like a ROCm issue. Can you please take a look?

[benrichard-amd]

Hi @damianhunziker

I have set the UMA Memory in Bios to 2 GB as its using the CPU memory anyhow. Now I can let run big models like GPT OSS 120b gpt-oss-120b-Q8_0. However when i run it it shows in htop memory usage of 65 GB and in nvtop it doesnt shows the GPU is using memory but the GPU is running on 100%. Is this how it should be?

Yes. On Strix Halo the VRAM cutout set in the BIOS is not directly accessible by the OS, and it is permanent. GTT (Graphics Translation Table)-backed memory is allocated from system memory when the GPU needs it and can be returned to the system. This is what you are seeing. Ultimately the VRAM and system RAM are on the same physical memory, so the performance difference is minimal.

See here: https://github.com/ROCm/TheRock/blob/main/docs/faq.md#gfx1151-strix-halo-specific-questions

I think nvtop mght be looking at VRAM, not GTT. If you run amd-smi or rocm-smi --showmeminfo GTT, you will see the memory allocation.

As for Vulkan vs ROCm performance: we have many tickets regarding this and are working to improve. If you have a specific example, feel free to create a ticket for it.

Let me know if you have any other questions.

Thanks!

[iz0eyj]

@benrichard-amd
As a side note: on this same Strix Halo system I am currently running comparative benchmarks between the Vulkan and ROCm backends using Qwen3 30B and Qwen3‑Next 80B, both in 8‑bit GGUF.
On this hardware, the 30B model fits entirely in the effective VRAM cutout, while the 80B model necessarily spills over the 48 GB VRAM region and uses a large amount of GTT‑backed system memory (as confirmed by rocm-smi --showmeminfo GTT VRAM).
If you are interested, I can share detailed per‑backend measurements (prompt and generation tokens/sec, context length scaling, GPU utilization, and VRAM vs GTT memory usage) so you have more data points on how ROCm behaves in these “beyond VRAM” scenarios compared to Vulkan on gfx1150/gfx1151.

From the server logs I also see a significant difference in how Vulkan vs ROCm place the 80B model in memory and how the compute graph is split (Vulkan uses a single large device buffer and only 2 graph splits, whereas ROCm splits the model across device and host and generates 90+ graph splits for the same batch). If needed I can provide the full log excerpts for further analysis.

[damianhunziker]

I now just installed Fedora 43 with the 6.18 kernel and the Jan 26 Firmware, works much better, and I can use the kyuz0 toolboxes directly without translating them to distroboxes, works very good, set 2GB UMA in Bios and the ttm grub parameter, installed rocm7.2 on the host and radv. Can now use all the backends without the need to change memory parameters in the Bios. It shows it correctly in nvtop and htop 123 GB available. Problem solved for me.