Eval bug: --n-cpu-moe does not evenly split on multi-GPU system

[mallorbc]

Name and Version

Using cuda docker image:
docker run ghcr.io/ggml-org/llama.cpp:server-cuda --version
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
version: 6097 (9515c61)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

Ryzen 5950x with two RTX 3090s and 128GB of DDR4 RAM

Models

I tried multiple models but the clearest one is Qwen3-235B-A22B-IQ4_XS.

https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF

Problem description & steps to reproduce

I was trying out the new, cpu-moe and n-cpu-moe flags to help speed up performance running large MOE models, or at the very least, run them with longer context.

The flags definitely work in helping me run the models better, but the current implementation does the evenly divide the model across my two GPUs.

Instead, it will fill the VRAM of one of the GPUs, and if I try to offload more parameters to the GPU, it will sooner run out of memory on one of the GPUs before utilizing the second GPU.

Assuming I'm not doing anything wrong and assuming this is a bug, upon resolving the bug, I will be able to use more MOE layers on the GPU, allowing me to run the model faster.

command I am running:

docker run --rm --gpus all \
  -v /datadrive/data/text-generation-webui/user_data/models:/models \
  -v /datadrive/data/slot-cache:/slot-cache \
  -p 5000:5000 \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf \
  -c 8192 \
  -ngl 999 \
  --flash-attn \
  --parallel 1 \
  --cont-batching \
  --mlock \
  --threads 14 \
  --threads-batch 28 \
  --slot-save-path /slot-cache \
  --host 0.0.0.0 \
  --port 5000 \
  --jinja --n-cpu-moe 80 --cache-type-k q8_0 --cache-type-v q8_0 --no-mmap

Vram utilization:

I would expect the VRAM usage to be split evenly or nearly evenly across both GPUs. This would either leave me VRAM for other models or allow me to decrease the --n-cpu-moe flag value

The -cpu-moe flag seems to be working fine. It offloads all moe layers to cpu and only attention layers are on GPUs. The amount of Vram utilization is then dependent on model size, but mainly context size

Working command for running with 8k context with --cpu-moe:

docker run --rm --gpus all \
  -v /datadrive/data/text-generation-webui/user_data/models:/models \
  -v /datadrive/data/slot-cache:/slot-cache \
  -p 5000:5000 \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf \
  -c 8192 \
  -ngl 999 \
  --flash-attn \
  --parallel 1 \
  --cont-batching \
  --mlock \
  --threads 14 \
  --threads-batch 28 \
  --slot-save-path /slot-cache \
  --host 0.0.0.0 \
  --port 5000 \
  --jinja --cpu-moe --cache-type-k q8_0 --cache-type-v q8_0

Vram utilization:

Working command for running with 64k context with --cpu-moe:

docker run --rm --gpus all \
  -v /datadrive/data/text-generation-webui/user_data/models:/models \
  -v /datadrive/data/slot-cache:/slot-cache \
  -p 5000:5000 \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf \
  -c 65536 \
  -ngl 999 \
  --flash-attn \
  --parallel 1 \
  --cont-batching \
  --mlock \
  --threads 14 \
  --threads-batch 28 \
  --slot-save-path /slot-cache \
  --host 0.0.0.0 \
  --port 5000 \
  --jinja --cpu-moe --cache-type-k q8_0 --cache-type-v q8_0

Vram utilization (it increased as expected and is split evenly):

First Bad Commit

No response

Relevant log output

Nothing really.  The model seems to operate normally.  I will copy logs for the --n-cpu-moe, just in case:


docker run --rm --gpus all \
  -v /datadrive/data/text-generation-webui/user_data/models:/models \
  -v /datadrive/data/slot-cache:/slot-cache \
  -p 5000:5000 \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf \
  -c 8192 \
  -ngl 999 \
  --flash-attn \
  --parallel 1 \
  --cont-batching \
  --mlock \
  --threads 14 \
  --threads-batch 28 \
  --slot-save-path /slot-cache \
  --host 0.0.0.0 \
  --port 5000 \
  --jinja --n-cpu-moe 80 --cache-type-k q8_0 --cache-type-v q8_0 --no-mmap
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
build: 6097 (9515c613) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
system info: n_threads = 14, n_threads_batch = 28, total_threads = 32

system_info: n_threads = 14 (n_threads_batch = 28) / 32 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 0.0.0.0, port: 5000, http threads: 31
main: loading model
srv    load_model: loading model '/models/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090) - 23964 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 3090) - 23982 MiB free
llama_model_loader: additional 2 GGUFs metadata loaded.
llama_model_loader: loaded meta data with 45 key-value pairs and 1131 tensors from /models/Qwen3-235B-A22B-IQ4_XS-00001-of-00003.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3moe
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen3-235B-A22B
llama_model_loader: - kv   3:                           general.basename str              = Qwen3-235B-A22B
llama_model_loader: - kv   4:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   5:                         general.size_label str              = 235B-A22B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen3-235...
llama_model_loader: - kv   8:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   9:                   general.base_model.count u32              = 1
llama_model_loader: - kv  10:                  general.base_model.0.name str              = Qwen3 235B A22B
llama_model_loader: - kv  11:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  12:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen3-235...
llama_model_loader: - kv  13:                               general.tags arr[str,2]       = ["unsloth", "text-generation"]
llama_model_loader: - kv  14:                       qwen3moe.block_count u32              = 94
llama_model_loader: - kv  15:                    qwen3moe.context_length u32              = 40960
llama_model_loader: - kv  16:                  qwen3moe.embedding_length u32              = 4096
llama_model_loader: - kv  17:               qwen3moe.feed_forward_length u32              = 12288
llama_model_loader: - kv  18:              qwen3moe.attention.head_count u32              = 64
llama_model_loader: - kv  19:           qwen3moe.attention.head_count_kv u32              = 4
llama_model_loader: - kv  20:                    qwen3moe.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  21:  qwen3moe.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  22:                 qwen3moe.expert_used_count u32              = 8
llama_model_loader: - kv  23:              qwen3moe.attention.key_length u32              = 128
llama_model_loader: - kv  24:            qwen3moe.attention.value_length u32              = 128
llama_model_loader: - kv  25:                      qwen3moe.expert_count u32              = 128
llama_model_loader: - kv  26:        qwen3moe.expert_feed_forward_length u32              = 1536
llama_model_loader: - kv  27:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  28:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  29:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  30:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  31:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  33:            tokenizer.ggml.padding_token_id u32              = 151654
llama_model_loader: - kv  34:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  35:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  36:               general.quantization_version u32              = 2
llama_model_loader: - kv  37:                          general.file_type u32              = 30
llama_model_loader: - kv  38:                      quantize.imatrix.file str              = Qwen3-235B-A22B-GGUF/imatrix_unsloth.dat
llama_model_loader: - kv  39:                   quantize.imatrix.dataset str              = unsloth_calibration_Qwen3-235B-A22B.txt
llama_model_loader: - kv  40:             quantize.imatrix.entries_count i32              = 744
llama_model_loader: - kv  41:              quantize.imatrix.chunks_count i32              = 685
llama_model_loader: - kv  42:                                   split.no u16              = 0
llama_model_loader: - kv  43:                        split.tensors.count i32              = 1131
llama_model_loader: - kv  44:                                split.count u16              = 3
llama_model_loader: - type  f32:  471 tensors
llama_model_loader: - type q4_K:    9 tensors
llama_model_loader: - type q5_K:   94 tensors
llama_model_loader: - type q6_K:    1 tensors
llama_model_loader: - type iq4_xs:  556 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = IQ4_XS - 4.25 bpw
print_info: file size   = 116.88 GiB (4.27 BPW)
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch             = qwen3moe
print_info: vocab_only       = 0
print_info: n_ctx_train      = 40960
print_info: n_embd           = 4096
print_info: n_layer          = 94
print_info: n_head           = 64
print_info: n_head_kv        = 4
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 16
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 12288
print_info: n_expert         = 128
print_info: n_expert_used    = 8
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 40960
print_info: rope_finetuned   = unknown
print_info: model type       = 235B.A22B
print_info: model params     = 235.09 B
print_info: general.name     = Qwen3-235B-A22B
print_info: n_ff_exp         = 1536
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 11 ','
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151654 '<|vision_pad|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
warning: failed to mlock 102852722688-byte buffer (after previously locking 0 bytes): Cannot allocate memory
Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root).
warning: failed to mlock 350060544-byte buffer (after previously locking 0 bytes): Cannot allocate memory
Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root).
load_tensors: offloading 94 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 95/95 layers to GPU
load_tensors:    CUDA_Host model buffer size = 98088.00 MiB
load_tensors:        CUDA0 model buffer size =  1846.55 MiB
load_tensors:        CUDA1 model buffer size = 19416.48 MiB
load_tensors:          CPU model buffer size =   333.84 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 8192
llama_context: n_ctx_per_seq = 8192
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 1
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     0.58 MiB
llama_kv_cache_unified:      CUDA0 KV buffer size =   408.00 MiB
llama_kv_cache_unified:      CUDA1 KV buffer size =   391.00 MiB
llama_kv_cache_unified: size =  799.00 MiB (  8192 cells,  94 layers,  1/1 seqs), K (q8_0):  399.50 MiB, V (q8_0):  399.50 MiB
llama_context:      CUDA0 compute buffer size =   848.25 MiB
llama_context:      CUDA1 compute buffer size =   304.75 MiB
llama_context:  CUDA_Host compute buffer size =    24.01 MiB
llama_context: graph nodes  = 5835
llama_context: graph splits = 275 (with bs=512), 162 (with bs=1)
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: added <|fim_pad|> logit bias = -inf
common_init_from_params: added <|repo_name|> logit bias = -inf
common_init_from_params: added <|file_sep|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 8192
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 8192
main: model loaded
main: chat template, chat_template: {%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0].role == 'system' %}
        {{- messages[0].content + '\n\n' }}
    {%- endif %}
    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
    {%- if messages[0].role == 'system' %}
        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
    {%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for forward_message in messages %}
    {%- set index = (messages|length - 1) - loop.index0 %}
    {%- set message = messages[index] %}
    {%- set current_content = message.content if message.content is defined and message.content is not none else '' %}
    {%- set tool_start = '<tool_response>' %}
    {%- set tool_start_length = tool_start|length %}
    {%- set start_of_message = current_content[:tool_start_length] %}
    {%- set tool_end = '</tool_response>' %}
    {%- set tool_end_length = tool_end|length %}
    {%- set start_pos = (current_content|length) - tool_end_length %}
    {%- if start_pos < 0 %}
        {%- set start_pos = 0 %}
    {%- endif %}
    {%- set end_of_message = current_content[start_pos:] %}
    {%- if ns.multi_step_tool and message.role == "user" and not(start_of_message == tool_start and end_of_message == tool_end) %}
        {%- set ns.multi_step_tool = false %}
        {%- set ns.last_query_index = index %}
    {%- endif %}
{%- endfor %}
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
    {%- elif message.role == "assistant" %}
        {%- set m_content = message.content if message.content is defined and message.content is not none else '' %}
        {%- set content = m_content %}
        {%- set reasoning_content = '' %}
        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}
            {%- set reasoning_content = message.reasoning_content %}
        {%- else %}
            {%- if '</think>' in m_content %}
                {%- set content = (m_content.split('</think>')|last).lstrip('\n') %}
                {%- set reasoning_content = (m_content.split('</think>')|first).rstrip('\n') %}
                {%- set reasoning_content = (reasoning_content.split('<think>')|last).lstrip('\n') %}
            {%- endif %}
        {%- endif %}
        {%- if loop.index0 > ns.last_query_index %}
            {%- if loop.last or (not loop.last and (not reasoning_content.strip() == '')) %}
                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
            {%- else %}
                {{- '<|im_start|>' + message.role + '\n' + content }}
            {%- endif %}
        {%- else %}
            {{- '<|im_start|>' + message.role + '\n' + content }}
        {%- endif %}
        {%- if message.tool_calls %}
            {%- for tool_call in message.tool_calls %}
                {%- if (loop.first and content) or (not loop.first) %}
                    {{- '\n' }}
                {%- endif %}
                {%- if tool_call.function %}
                    {%- set tool_call = tool_call.function %}
                {%- endif %}
                {{- '<tool_call>\n{"name": "' }}
                {{- tool_call.name }}
                {{- '", "arguments": ' }}
                {%- if tool_call.arguments is string %}
                    {{- tool_call.arguments }}
                {%- else %}
                    {{- tool_call.arguments | tojson }}
                {%- endif %}
                {{- '}\n</tool_call>' }}
            {%- endfor %}
        {%- endif %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" %}
        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
            {{- '<|im_start|>user' }}
        {%- endif %}
        {{- '\n<tool_response>\n' }}
        {{- message.content }}
        {{- '\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
    {%- if enable_thinking is defined and enable_thinking is false %}
        {{- '<think>\n\n</think>\n\n' }}
    {%- endif %}
{%- endif %}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://0.0.0.0:5000 - starting the main loop
srv  update_slots: all slots are idle

[mallorbc]

#15125 seems like a related issue

[abc-nix]

I think it is expected. (EDIT: because it offloads layers in blocks, not looking at what GPU has each layer, so it reduces the number of layers in order).

If you use ngl 999, it will split the layers evenly between all the GPU devices (in this case, 2 devices). You need to use the --tensor_split option to divide the layers correctly so that one GPU doesn't overflow (OOM).

In your example, 2 GPU devices with 24 GBs of VRAM each. The model is Qwen3-235B-A22B-IQ4_XS (aprox 125 GBs) with a total amount of 95 layers. Quick Maths:

• 1 layer is about 1.3 GBs.
• 1 GPU with 24 GBs VRAM could house about 18 full layers.
• You could offload 17-18 full layers to GPU2.
• All other non-expert layers could go to GPU1 (and maybe 10-13 of them can be full layers).

Try with:

--tensor-split 77,18 \
--n-cpu-moe 64

[slaren]

As the documentation states, --n-cpu-moe N keeps the MoE tensors of the first N layers in the CPU. If you are using multiple GPUs, this means that it will affect the first GPU first.

I understand that this behavior can be confusing, and it would be better if it removed a proportional number of layers from each GPU. However, doing that would be a lot more complicated than the current implementation, and it may not happen for a while. Meanwhile, you can also do this by using --override-tensor.

[mallorbc]

I think it is expected. (EDIT: because it offloads layers in blocks, not looking at what GPU has each layer, so it reduces the number of layers in order).

If you use ngl 999, it will split the layers evenly between all the GPU devices (in this case, 2 devices). You need to use the --tensor_split option to divide the layers correctly so that one GPU doesn't overflow (OOM).

In your example, 2 GPU devices with 24 GBs of VRAM each. The model is Qwen3-235B-A22B-IQ4_XS (aprox 125 GBs) with a total amount of 95 layers. Quick Maths:

• 1 layer is about 1.3 GBs.
• 1 GPU with 24 GBs VRAM could house about 18 full layers.
• You could offload 17-18 full layers to GPU2.
• All other non-expert layers could go to GPU1 (and maybe 10-13 of them can be full layers).

Try with:

--tensor_split 77/18 \
--n-cpu-moe 64

This seems to be the solution. Memory utilization is much higher. And because I can offload more layers to the GPU, I get about 25% faster generation(around 4.5 tokens/second to around 5.5) and can use the full context window of the model

It might be worthwhile to have this behavior be controlled by a flag one day, but there are higher priority things in the works, I am sure.

Thank you both for your help.