Eval bug: Incorrect outputs when running inference in multiple nodes (Mac)

[jlcaam-bit]

Name and Version

b6791 and previous

Operating systems

Mac

GGML backends

Metal

Hardware

Models

Multiple****

Problem description & steps to reproduce

My setup has two different nodes: MacMini M2 Pro + MacMini M1. When running inference on multi-node at some point the output starts repeating one token until the end of the context window.

Only in multi-node inference, no issue in single node.

Happens with different models.

Using llama-server or llama-cli. ( + rpc-server )

Bug in all versions starting at 6473. 6471 and previous are fine.

Examples

Running in dual-node the result is wrong:

./rpc-server -H 10.1.2.2:50052 (in the second machine)

./llama-cli --model /var/lib/modelostemp/gpt-oss-20b-mxfp4.gguf --temp 1 --top_k 20 --top_p 1.0 --min_p 0 --seed 1234 --jinja --ctx-size 7000 --no-mmap --rpc 10.1.2.2:50052

> tell me about Julio Cesar
<|channel|>analysis<|message|>User says: "tell me about Julio Cesar". This is ambiguous: could refer to historical Julius (Roman) or other person named Julio Cesar e.g., footballer. Usually people ask general about historical. Need to ask clarifying? The instruction: "If you are unsure what user is asking for... ask for clarification." Here unclear: Julio Cesar could be the Emperor? It might also be the Brazilian footballer? Could also be the Spanish singer? Let's see: "Julio César" with accent; many people. In Brazil, famous footballers: Julio Cesar de Souza Júnior (born 1985), Brazilian goalkeeper. There's also "Julio César" the singer? Or "Julio César" in Spanish maybe meaning "Julius Caesar" but spelled Spanish. Likely user wants info about the Roman general Julius Caesar. But the name is spelled Julio with J. In Spanish/Portuguese, Julius Caesar is Julio César. So user might want about Roman. But there is also the footballer. Without clarifying, risk misinterpretation. The instruction: if uncertain, ask for clarification. So I should ask: specify which Julio Cesar.<|end|><|start|>assistant<|channel|>final<|message|>I’m happy to help—but could you specify which Julio César you’re interested in?  
- The Roman general/ statesman (often written “Julius Caesar” in English).  
- The Brazilian football goalkeeper Julio César de Souza Júnior (born 1985).  
- Another public figure named Julio César (e.g., an artist, politician, etc.).

Just let me know which one (or if you’d like information on all of them!) so I can give you the most accurate and useful details.

> the emperor
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
(omitted)
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

main: context full and context shift is disabled => stopping

Running on an single node the result is okay:

./llama-cli --model /var/lib/modelostemp/gpt-oss-20b-mxfp4.gguf --temp 1 --top_k 20 --top_p 1.0 --min_p 0 --seed 1234 --jinja --ctx-size 7000 --no-mmap --n-cpu-moe 1

does not show the problem.

Same problem with other models:

./llama-cli --model /var/lib/modelostemp/Qwen3-4B-2507/Qwen3-4B-Instruct-2507-Q8_0.gguf --temp 0 --top_k 20 --top_p 0.95 --seed 12345 --ctx-size 4000 --rpc 10.1.2.2:50052

> tell me about Julio Cesar
It seems like you might be referring to **Julio César**, but there are a few notable people with that name. Here are the most prominent ones:

---

### 1. **Julio César (the Roman Emperor)**
This is likely the most historically significant figure named Julio César.

- **Full Name**: Gaius Julius Caesar (commonly known as Julius Caesar)
- **Lived**: 100 BCE – 44 BCE
- **Role**: Roman statesman, general, and dictator
- **Key Facts**:
  - One of the most influential figures in Roman history.
  - Expanded the Roman Republic through military conquests (e.g., Gaul).
  - Introduced major reforms in law, calendar (Julian calendar), and governance.
  - Assassinated in 44 BCE by a group of senators who feared his growing power.
  - His assassination led to the rise of the Roman Empire and eventually Augustus (Octavian).

> Note: The name "Julio César" is a Spanish/Portuguese version of "Julius Caesar".

---

### 2. **Julio César (the boxer)000000000000000000000000000000000
(omitted)
00000000000000000000000000000000000000000000000000000

main: context full and context shift is disabled => stopping

./llama-cli --model /var/lib/modelostemp/DeepSeek-R1-0528-Qwen3-8B-Q8_0.gguf --flash_attn on --ctx-size 4000 --rpc 10.1.2.2:50052

> tell me about Julio Cesar
<think>
Okay, the user asked "tell me about Julio Cesar". I think they meant Julius Caesar, the famous Roman general and statesman. They probably want a general overview, but I should consider what level of detail they might need.

Let me start by outlining the key points about Julius Caesar. He was a pivotal figure in Roman history, known for his military conquests and political reforms. I need to structure this information clearly but also engagingly. Since the user might be a student or someone with casual interest, I should include both historical significance and interesting anecdotes to keep it captivating.

The user might not just want facts; they could be interested in understanding why Julius Caesar is still relevant today. So, highlighting his lasting legacies like the Julian calendar and his influence on Western literature through "The Brutus" would be useful. Also, his assassination is a major event, so explaining its causes and consequences is necessary.

I should check if the user has any specific interests. Maybe they’re researching for a project or just curious. Either way, providing a well-rounded summary covering his life, achievements, and end is best. Avoiding too much detail unless asked, keeping it concise but informative. 

Including his rise to power, reforms@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

First Bad Commit

b6473

Relevant log output

x

[chodorenko]

I confirm on MacOS 26.0.1 i retrieve at the same problem

RPC server log

chodorenko@Mikhails-MacBook-Air bin % ./rpc-server -d Metal -c -H 169.254.97.200

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
WARNING: Host ('169.254.97.200') is != '127.0.0.1'
Never expose the RPC server to an open network!
This is an experimental feature and is not secure!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 0.018 sec
ggml_metal_device_init: GPU name: Apple M1
ggml_metal_device_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 12713.12 MB
Starting RPC server v3.0.0
endpoint : 169.254.97.200:50052
local cache : /Users/chodorenko/Library/Caches/llama.cpp/rpc/
Devices:
Metal: Apple M1 (12124 MiB, 12123 MiB free)
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1
ggml_metal_init: picking default device: Apple M1
ggml_metal_init: use bfloat = true
ggml_metal_init: use fusion = true
ggml_metal_init: use concurrency = true
ggml_metal_init: use graph optimize = true
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Client connection closed
Accepted client connection
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
Null buffer for tensor passed to init_tensor function
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_rms_norm_f32_4', name = 'kernel_rms_norm_f32_4'
ggml_metal_library_compile_pipeline: loaded kernel_rms_norm_f32_4 0x1054c7990 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_mul_row_c4_fuse_1', name = 'kernel_mul_row_c4_fuse_1'
ggml_metal_library_compile_pipeline: loaded kernel_mul_row_c4_fuse_1 0x1054c84d0 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_mul_mv_q8_0_f32', name = 'kernel_mul_mv_q8_0_f32_nsg=4'
ggml_metal_library_compile_pipeline: loaded kernel_mul_mv_q8_0_f32_nsg=4 0x1054c92d0 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_add_row_c4_fuse_1', name = 'kernel_add_row_c4_fuse_1'
ggml_metal_library_compile_pipeline: loaded kernel_add_row_c4_fuse_1 0x1054c9c10 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_rope_neox_f32', name = 'kernel_rope_neox_f32'
ggml_metal_library_compile_pipeline: loaded kernel_rope_neox_f32 0x1054ca410 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_set_rows_f16_i64', name = 'kernel_set_rows_f16_i64'
ggml_metal_library_compile_pipeline: loaded kernel_set_rows_f16_i64 0x1054ca8d0 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_cpy_f32_f16', name = 'kernel_cpy_f32_f16'
ggml_metal_library_compile_pipeline: loaded kernel_cpy_f32_f16 0x1054cad90 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_flash_attn_ext_vec_f16_dk64_dv64', name = 'kernel_flash_attn_ext_vec_f16_dk64_dv64_mask=1_sink=1_bias=0_scap=0_kvpad=0_ns10=512_ns20=512_nsg=1_nwg=32'
ggml_metal_library_compile_pipeline: loaded kernel_flash_attn_ext_vec_f16_dk64_dv64_mask=1_sink=1_bias=0_scap=0_kvpad=0_ns10=512_ns20=512_nsg=1_nwg=32 0x1054cb390 | th_max = 768 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_flash_attn_ext_vec_reduce', name = 'kernel_flash_attn_ext_vec_reduce_dv=64_nwg=32'
ggml_metal_library_compile_pipeline: loaded kernel_flash_attn_ext_vec_reduce_dv=64_nwg=32 0xbfede8000 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_mul_mv_ext_q8_0_f32_r1_2', name = 'kernel_mul_mv_ext_q8_0_f32_r1_2_nsg=2_nxpsg=16'
ggml_metal_library_compile_pipeline: loaded kernel_mul_mv_ext_q8_0_f32_r1_2_nsg=2_nxpsg=16 0xbfede8300 | th_max = 832 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_add_fuse_1', name = 'kernel_add_fuse_1'
ggml_metal_library_compile_pipeline: loaded kernel_add_fuse_1 0xbfede8600 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_mul_mv_f32_f32_4', name = 'kernel_mul_mv_f32_f32_4_nsg=4'
ggml_metal_library_compile_pipeline: loaded kernel_mul_mv_f32_f32_4_nsg=4 0xbfede8900 | th_max = 768 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_argsort_f32_i32_desc', name = 'kernel_argsort_f32_i32_desc'
ggml_metal_library_compile_pipeline: loaded kernel_argsort_f32_i32_desc 0xbfede8c00 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_get_rows_f32', name = 'kernel_get_rows_f32'
ggml_metal_library_compile_pipeline: loaded kernel_get_rows_f32 0xbfede8f00 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_soft_max_f32_4', name = 'kernel_soft_max_f32_4'
ggml_metal_library_compile_pipeline: loaded kernel_soft_max_f32_4 0xbfede9200 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_mul_mv_id_mxfp4_f32', name = 'kernel_mul_mv_id_mxfp4_f32_nsg=2'
ggml_metal_library_compile_pipeline: loaded kernel_mul_mv_id_mxfp4_f32_nsg=2 0xbfede9500 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_add_id', name = 'kernel_add_id'
ggml_metal_library_compile_pipeline: loaded kernel_add_id 0xbfede9800 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_swiglu_oai_f32', name = 'kernel_swiglu_oai_f32'
ggml_metal_library_compile_pipeline: loaded kernel_swiglu_oai_f32 0xbfede9b00 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_mul_fuse_1', name = 'kernel_mul_fuse_1'
ggml_metal_library_compile_pipeline: loaded kernel_mul_fuse_1 0xbfede9e00 | th_max = 1024 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_mul_mm_q8_0_f32', name = 'kernel_mul_mm_q8_0_f32_bci=0_bco=1'
ggml_metal_library_compile_pipeline: loaded kernel_mul_mm_q8_0_f32_bci=0_bco=1 0xbfedea100 | th_max = 896 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_mul_mm_f32_f32', name = 'kernel_mul_mm_f32_f32_bci=0_bco=1'
ggml_metal_library_compile_pipeline: loaded kernel_mul_mm_f32_f32_bci=0_bco=1 0xbfedea400 | th_max = 832 | th_width = 32
ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_flash_attn_ext_vec_f16_dk64_dv64', name = 'kernel_flash_attn_ext_vec_f16_dk64_dv64_mask=1_sink=1_bias=0_scap=0_kvpad=0_ns10=512_ns20=512_nsg=2_nwg=32'
ggml_metal_library_compile_pipeline: loaded kernel_flash_attn_ext_vec_f16_dk64_dv64_mask=1_sink=1_bias=0_scap=0_kvpad=0_ns10=512_ns20=512_nsg=2_nwg=32 0xbfedea700 | th_max = 768 | th_width = 32
Client connection closed

llama-cli log

chodorenko@Chodorenko-M15-2 bin % ./llama-cli --model ~/18/gpt-oss-20b-Q8_0.gguf --rpc 169.254.97.200:50052
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 0.049 sec
ggml_metal_device_init: GPU name: Apple M3
ggml_metal_device_init: GPU family: MTLGPUFamilyApple9 (1009)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 19069.67 MB
build: 6865 (1c1409e) with Apple clang version 17.0.0 (clang-1700.3.19.1) for x86_64-apple-darwin25.0.0
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device RPC0 (169.254.97.200:50052) (unknown id) - 12123 MiB free
llama_model_load_from_file_impl: using device Metal (Apple M3) (unknown id) - 18185 MiB free
llama_model_loader: loaded meta data with 37 key-value pairs and 459 tensors from /Users/chodorenko/18/gpt-oss-20b-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = gpt-oss
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Gpt-Oss-20B
llama_model_loader: - kv 3: general.basename str = Gpt-Oss-20B
llama_model_loader: - kv 4: general.quantized_by str = Unsloth
llama_model_loader: - kv 5: general.size_label str = 20B
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 8: general.tags arr[str,2] = ["vllm", "text-generation"]
llama_model_loader: - kv 9: gpt-oss.block_count u32 = 24
llama_model_loader: - kv 10: gpt-oss.context_length u32 = 131072
llama_model_loader: - kv 11: gpt-oss.embedding_length u32 = 2880
llama_model_loader: - kv 12: gpt-oss.feed_forward_length u32 = 2880
llama_model_loader: - kv 13: gpt-oss.attention.head_count u32 = 64
llama_model_loader: - kv 14: gpt-oss.attention.head_count_kv u32 = 8
llama_model_loader: - kv 15: gpt-oss.rope.freq_base f32 = 150000,000000
llama_model_loader: - kv 16: gpt-oss.attention.layer_norm_rms_epsilon f32 = 0,000010
llama_model_loader: - kv 17: gpt-oss.expert_count u32 = 32
llama_model_loader: - kv 18: gpt-oss.expert_used_count u32 = 4
llama_model_loader: - kv 19: gpt-oss.attention.key_length u32 = 64
llama_model_loader: - kv 20: gpt-oss.attention.value_length u32 = 64
llama_model_loader: - kv 21: gpt-oss.attention.sliding_window u32 = 128
llama_model_loader: - kv 22: gpt-oss.expert_feed_forward_length u32 = 2880
llama_model_loader: - kv 23: gpt-oss.rope.scaling.type str = yarn
llama_model_loader: - kv 24: gpt-oss.rope.scaling.factor f32 = 32,000000
llama_model_loader: - kv 25: gpt-oss.rope.scaling.original_context_length u32 = 4096
llama_model_loader: - kv 26: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 27: tokenizer.ggml.pre str = gpt-4o
llama_model_loader: - kv 28: tokenizer.ggml.tokens arr[str,201088] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 29: tokenizer.ggml.token_type arr[i32,201088] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 30: tokenizer.ggml.merges arr[str,446189] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 31: tokenizer.ggml.bos_token_id u32 = 199998
llama_model_loader: - kv 32: tokenizer.ggml.eos_token_id u32 = 200002
llama_model_loader: - kv 33: tokenizer.ggml.padding_token_id u32 = 200017
llama_model_loader: - kv 34: tokenizer.chat_template str = {# Chat template fixes by Unsloth #}\n...
llama_model_loader: - kv 35: general.quantization_version u32 = 2
llama_model_loader: - kv 36: general.file_type u32 = 7
llama_model_loader: - type f32: 289 tensors
llama_model_loader: - type q8_0: 98 tensors
llama_model_loader: - type mxfp4: 72 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q8_0
print_info: file size = 11,27 GiB (4,63 BPW)
load: printing all EOG tokens:
load: - 199999 ('<|endoftext|>')
load: - 200002 ('<|return|>')
load: - 200007 ('<|end|>')
load: - 200012 ('<|call|>')
load: special_eog_ids contains both '<|return|>' and '<|call|>' tokens, removing '<|end|>' token from EOG list
load: special tokens cache size = 21
load: token to piece cache size = 1,3332 MB
print_info: arch = gpt-oss
print_info: vocab_only = 0
print_info: n_ctx_train = 131072
print_info: n_embd = 2880
print_info: n_layer = 24
print_info: n_head = 64
print_info: n_head_kv = 8
print_info: n_rot = 64
print_info: n_swa = 128
print_info: is_swa_any = 1
print_info: n_embd_head_k = 64
print_info: n_embd_head_v = 64
print_info: n_gqa = 8
print_info: n_embd_k_gqa = 512
print_info: n_embd_v_gqa = 512
print_info: f_norm_eps = 0,0e+00
print_info: f_norm_rms_eps = 1,0e-05
print_info: f_clamp_kqv = 0,0e+00
print_info: f_max_alibi_bias = 0,0e+00
print_info: f_logit_scale = 0,0e+00
print_info: f_attn_scale = 0,0e+00
print_info: n_ff = 2880
print_info: n_expert = 32
print_info: n_expert_used = 4
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 2
print_info: rope scaling = yarn
print_info: freq_base_train = 150000,0
print_info: freq_scale_train = 0,03125
print_info: n_ctx_orig_yarn = 4096
print_info: rope_finetuned = unknown
print_info: model type = 20B
print_info: model params = 20,91 B
print_info: general.name = Gpt-Oss-20B
print_info: n_ff_exp = 2880
print_info: vocab type = BPE
print_info: n_vocab = 201088
print_info: n_merges = 446189
print_info: BOS token = 199998 '<|startoftext|>'
print_info: EOS token = 200002 '<|return|>'
print_info: EOT token = 200007 '<|end|>'
print_info: PAD token = 200017 '<|reserved_200017|>'
print_info: LF token = 198 'Ċ'
print_info: EOG token = 199999 '<|endoftext|>'
print_info: EOG token = 200002 '<|return|>'
print_info: EOG token = 200012 '<|call|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors: CPU_Mapped model buffer size = 586,82 MiB
load_tensors: Metal_Mapped model buffer size = 11536,18 MiB
load_tensors: RPC0[169.254.97.200:50052] model buffer size = 4317,72 MiB
.................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = auto
llama_context: kv_unified = false
llama_context: freq_base = 150000,0
llama_context: freq_scale = 0,03125
llama_context: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3
ggml_metal_init: picking default device: Apple M3
ggml_metal_init: use bfloat = true
ggml_metal_init: use fusion = true
ggml_metal_init: use concurrency = true
ggml_metal_init: use graph optimize = true
llama_context: CPU output buffer size = 0,77 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 4096 cells
llama_kv_cache: Metal KV buffer size = 56,00 MiB
llama_kv_cache: RPC0[169.254.97.200:50052] KV buffer size = 40,00 MiB
llama_kv_cache: size = 96,00 MiB ( 4096 cells, 12 layers, 1/1 seqs), K (f16): 48,00 MiB, V (f16): 48,00 MiB
llama_kv_cache_iswa: creating SWA KV cache, size = 768 cells
llama_kv_cache: Metal KV buffer size = 10,50 MiB
llama_kv_cache: RPC0[169.254.97.200:50052] KV buffer size = 7,50 MiB
llama_kv_cache: size = 18,00 MiB ( 768 cells, 12 layers, 1/1 seqs), K (f16): 9,00 MiB, V (f16): 9,00 MiB
llama_context: Flash Attention was auto, set to enabled
llama_context: RPC0[169.254.97.200:50052] compute buffer size = 97,02 MiB
llama_context: Metal compute buffer size = 398,38 MiB
llama_context: CPU compute buffer size = 15,15 MiB
llama_context: graph nodes = 1352
llama_context: graph splits = 3
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|return|> logit bias = -inf
common_init_from_params: added <|call|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 4
main: chat template is available, enabling conversation mode (disable it with -no-cnv)
main: chat template example:
<|start|>system<|message|>You are a helpful assistant<|end|><|start|>user<|message|>Hello<|end|><|start|>assistant<|message|>Hi there<|return|><|start|>user<|message|>How are you?<|end|><|start|>assistant

system_info: n_threads = 4 (n_threads_batch = 4) / 8 | Metal : EMBED_LIBRARY = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | REPACK = 1 |

main: interactive mode on.
sampler seed: 2315456133
sampler params:
repeat_last_n = 64, repeat_penalty = 1,000, frequency_penalty = 0,000, presence_penalty = 0,000
dry_multiplier = 0,000, dry_base = 1,750, dry_allowed_length = 2, dry_penalty_last_n = 4096
top_k = 40, top_p = 0,950, min_p = 0,050, xtc_probability = 0,000, xtc_threshold = 0,100, typical_p = 1,000, top_n_sigma = -1,000, temp = 0,800
mirostat = 0, mirostat_lr = 0,100, mirostat_ent = 5,000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
generate: n_ctx = 4096, n_batch = 2048, n_predict = -1, n_keep = 0

== Running in interactive mode. ==

• Press Ctrl+C to interject at any time.
• Press Return to return control to the AI.
• To return control without starting a new line, end your input with '/'.
• If you want to submit another line, end your input with ''.
• Not using system message. To change it, set a different value via -sys PROMPT

let me history King Arthur
<|channel|>analysis<|message|>User: "let me history King Arthur". They likely want a brief history. We can give overview.<|end|><|start|>assistant<|channel|>final<|message|>King Arthur—A Mix of History, Myth, and Legend

Time	Key Events	Sources	What We Know
5th‑6th c. AD	Britain is shaken by the Anglo‑Saxon invasions. Local leaders defend against Saxon raids.	Gildas (c. 6 c.), Historia Brittonum (c. 8 c.)	No concrete record of a single “Arthur.” The era’s chaos produced many local war‑lords.
c. 530–560 AD	The Eddas mentions “Arthurus” in a Latin account of the Battle of the Quin?	Early Middle Ages (c. 530 c.)	This is a fictitious mention.
7th‑8th c. AD	Arthurus is Arcturus – the Arthurian legend.	*Arthurian@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

main: context full and context shift is disabled => stopping

llama_perf_sampler_print: sampling time = 1040,03 ms / 4096 runs ( 0,25 ms per token, 3938,34 tokens per second)
llama_perf_context_print: load time = 17781,43 ms
llama_perf_context_print: prompt eval time = 751,50 ms / 11 tokens ( 68,32 ms per token, 14,64 tokens per second)
llama_perf_context_print: eval time = 369494,66 ms / 4084 runs ( 90,47 ms per token, 11,05 tokens per second)
llama_perf_context_print: total time = 394866,23 ms / 4095 tokens
llama_perf_context_print: graphs reused = 4068
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | - RPC0 (169.254.97.200:50052) | 12124 = 7661 + ( 4462 = 4317 + 47 + 97) + 0 |
llama_memory_breakdown_print: | - Metal (Apple M3) | 18186 = 2992 + (12001 = 11536 + 66 + 398) + 3192 |
llama_memory_breakdown_print: | - Host | 601 = 586 + 0 + 15 |
ggml_metal_free: deallocating

[chodorenko]

@rgerganov @slaren

May be you can confirm the bug and fix it ?

[ggerganov]

I think this is a known issue with Metal + RPC (#16276 (review)). Can you confirm that adding -fa off fixes the issue?

[chodorenko]

I think this is a known issue with Metal + RPC (#16276 (review)). Can you confirm that adding -fa off fixes the issue?

Big Thanks, its params fix error