Which version of LM Studio?
LM Studio 0.4.0
Which operating system?
Windows 10 Workstation
What is the bug?
NVIDIA RTX3080 20G,512G RAM.After upgrading CUDA 12 llama.cpp (Windows) to version1.104.2 and 2.0, when loading DeepSeek R1 671b Q4-K-M, the log shows the following warning: "sched_reserve: layer 0 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support). sched_reserve: Flash Attention was auto, set to disabled." However, with the exact same parameter configuration in the previous version CUDA 12 llama.cpp (Windows) 1.103.2, this issue did not occur, and the processing speed was several times faster.
Screenshots
capture_20260130233736264.bmp
Logs
2026-01-30 22:05:02 [DEBUG]
[LM Studio] GPU Configuration:
Strategy: evenly
Priority: []
Disabled GPUs: []
Limit weight offload to dedicated GPU Memory: OFF
Offload KV Cache to GPU: OFF
2026-01-30 22:05:02 [DEBUG]
[LM Studio] Live GPU memory info (source 'LMS Core'):
GPU 0: NVIDIA GeForce RTX 3080 (Used: 2.67 GB, Total: 21.47 GB, Free: 18.80 GB)
2026-01-30 22:05:02 [DEBUG]
[LM Studio] Model load size estimate with raw num offload layers 'max' and context length '16384':
Model: 416.56 GB
Context: 20.84 GB
Total: 437.40 GB
2026-01-30 22:05:02 [DEBUG]
[LM Studio] Not using full context length for VRAM overflow calculations due to single GPU setup. Instead, using '8192' as context length for the calculation. Original context length: '16384'.
[LM Studio] Strict GPU VRAM cap is OFF: GPU offload layers will not be checked for adjustment
2026-01-30 22:05:02 [DEBUG]
[LM Studio] Resolved GPU config options:
Num Offload Layers: max
Num CPU Expert Layers: 71
Main GPU: 0
Tensor Split: [0]
Disabled GPUs: []
2026-01-30 22:05:02 [DEBUG]
LlamaV4::load called with model path:...
LlamaV4::load config: n_parallel=1 n_ctx=16384 kv_unified=false
2026-01-30 22:05:02 [DEBUG]
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes
2026-01-30 22:05:02 [DEBUG]
DeepSeek2 architecture detected, using AUTO flash attention mode
2026-01-30 22:05:02 [DEBUG]
srv load_model: loading model '...'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3080) (0000:03:00.0) - 19271 MiB free
llama_model_loader: ------------------------ Adding override for key 'deepseek2.expert_used_count'
2026-01-30 22:05:02 [DEBUG]
llama_model_loader: loaded meta data with 52 key-value pairs and 1025 tensors from ... (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = deepseek2
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = DeepSeek V3 Bf16
llama_model_loader: - kv 3: general.finetune str = abliterated
llama_model_loader: - kv 4: general.basename str = DeepSeek-V3
llama_model_loader: - kv 5: general.size_label str = 256x20B
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: general.base_model.count u32 = 1
llama_model_loader: - kv 8: general.base_model.0.name str = DeepSeek V3
llama_model_loader: - kv 9: general.base_model.0.version str = V3
llama_model_loader: - kv 10: general.base_model.0.organization str = Deepseek Ai
llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/deepseek-ai/De...
llama_model_loader: - kv 12: general.tags arr[str,4] = ["deepseek_v3", "bf16", "Safetensors"...
llama_model_loader: - kv 13: deepseek2.block_count u32 = 61
llama_model_loader: - kv 14: deepseek2.context_length u32 = 163840
llama_model_loader: - kv 15: deepseek2.embedding_length u32 = 7168
llama_model_loader: - kv 16: deepseek2.feed_forward_length u32 = 18432
llama_model_loader: - kv 17: deepseek2.attention.head_count u32 = 128
llama_model_loader: - kv 18: deepseek2.attention.head_count_kv u32 = 128
llama_model_loader: - kv 19: deepseek2.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 20: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 21: deepseek2.expert_used_count u32 = 8
llama_model_loader: - kv 22: deepseek2.leading_dense_block_count u32 = 3
llama_model_loader: - kv 23: deepseek2.vocab_size u32 = 129280
llama_model_loader: - kv 24: deepseek2.attention.q_lora_rank u32 = 1536
llama_model_loader: - kv 25: deepseek2.attention.kv_lora_rank u32 = 512
llama_model_loader: - kv 26: deepseek2.attention.key_length u32 = 192
llama_model_loader: - kv 27: deepseek2.attention.value_length u32 = 128
llama_model_loader: - kv 28: deepseek2.expert_feed_forward_length u32 = 2048
llama_model_loader: - kv 29: deepseek2.expert_count u32 = 256
llama_model_loader: - kv 30: deepseek2.expert_shared_count u32 = 1
llama_model_loader: - kv 31: deepseek2.expert_weights_scale f32 = 2.500000
llama_model_loader: - kv 32: deepseek2.expert_weights_norm bool = true
llama_model_loader: - kv 33: deepseek2.expert_gating_func u32 = 2
llama_model_loader: - kv 34: deepseek2.rope.dimension_count u32 = 64
llama_model_loader: - kv 35: deepseek2.rope.scaling.type str = yarn
llama_model_loader: - kv 36: deepseek2.rope.scaling.factor f32 = 40.000000
llama_model_loader: - kv 37: deepseek2.rope.scaling.original_context_length u32 = 4096
2026-01-30 22:05:02 [DEBUG]
llama_model_loader: - kv 38: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000
llama_model_loader: - kv 39: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 40: tokenizer.ggml.pre str = deepseek-v3
2026-01-30 22:05:02 [DEBUG]
llama_model_loader: - kv 41: tokenizer.ggml.tokens arr[str,129280] = ["<｜begin▁of▁sentence｜>", "<�...
2026-01-30 22:05:02 [DEBUG]
llama_model_loader: - kv 42: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
2026-01-30 22:05:02 [DEBUG]
llama_model_loader: - kv 43: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
llama_model_loader: - kv 44: tokenizer.ggml.bos_token_id u32 = 0
llama_model_loader: - kv 45: tokenizer.ggml.eos_token_id u32 = 1
llama_model_loader: - kv 46: tokenizer.ggml.padding_token_id u32 = 1
llama_model_loader: - kv 47: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 48: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 49: tokenizer.chat_template str = {% if not add_generation_prompt is de...
llama_model_loader: - kv 50: general.quantization_version u32 = 2
llama_model_loader: - kv 51: general.file_type u32 = 15
llama_model_loader: - type f32: 361 tensors
llama_model_loader: - type q4_K: 606 tensors
llama_model_loader: - type q6_K: 58 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 376.65 GiB (4.82 BPW)
validate_override: Using metadata override ( int) 'deepseek2.expert_used_count' = 8
load_hparams: ----------------------- n_expert_used = 8 n_expert_groups = 0
2026-01-30 22:05:02 [DEBUG]
load: 0 unused tokens
2026-01-30 22:05:02 [DEBUG]
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load: - 1 ('<｜end▁of▁sentence｜>')
2026-01-30 22:05:02 [DEBUG]
load: special tokens cache size = 818
2026-01-30 22:05:02 [DEBUG]
load: token to piece cache size = 0.8223 MB
print_info: arch = deepseek2
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 163840
print_info: n_embd = 7168
print_info: n_embd_inp = 7168
print_info: n_layer = 61
print_info: n_head = 128
print_info: n_head_kv = 128
print_info: n_rot = 64
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 192
print_info: n_embd_head_v = 128
print_info: n_gqa = 1
print_info: n_embd_k_gqa = 24576
print_info: n_embd_v_gqa = 16384
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 18432
print_info: n_expert = 256
print_info: n_expert_used = 8
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 0
print_info: rope scaling = yarn
print_info: freq_base_train = 10000.0
print_info: freq_scale_train = 0.025
print_info: n_ctx_orig_yarn = 4096
print_info: rope_yarn_log_mul = 1.0000
print_info: rope_finetuned = unknown
print_info: model type = 671B
print_info: model params = 671.03 B
print_info: general.name = DeepSeek V3 Bf16
print_info: n_layer_dense_lead = 3
print_info: n_lora_q = 1536
print_info: n_lora_kv = 512
print_info: n_embd_head_k_mla = 0
print_info: n_embd_head_v_mla = 0
print_info: n_ff_exp = 2048
print_info: n_expert_shared = 1
print_info: expert_weights_scale = 2.5
print_info: expert_weights_norm = 1
print_info: expert_gating_func = sigmoid
print_info: vocab type = BPE
print_info: n_vocab = 129280
print_info: n_merges = 127741
print_info: BOS token = 0 '<｜begin▁of▁sentence｜>'
print_info: EOS token = 1 '<｜end▁of▁sentence｜>'
print_info: EOT token = 1 '<｜end▁of▁sentence｜>'
print_info: PAD token = 1 '<｜end▁of▁sentence｜>'
print_info: LF token = 201 'Ċ'
print_info: FIM PRE token = 128801 '<｜fim▁begin｜>'
print_info: FIM SUF token = 128800 '<｜fim▁hole｜>'
print_info: FIM MID token = 128802 '<｜fim▁end｜>'
print_info: EOG token = 1 '<｜end▁of▁sentence｜>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
2026-01-30 22:05:41 [DEBUG]
load_tensors: offloading output layer to GPU
load_tensors: offloading 60 repeating layers to GPU
load_tensors: offloaded 62/62 layers to GPU
load_tensors: CPU_Mapped model buffer size = 384956.77 MiB
load_tensors: CUDA0 model buffer size = 9460.52 MiB
2026-01-30 22:09:40 [DEBUG]
common_init_result: added <｜end▁of▁sentence｜> logit bias = -inf
2026-01-30 22:09:40 [DEBUG]
llama_context: constructing llama_context
llama_context: setting new yarn_attn_factor = 1.0000 (mscale == 1.0, mscale_all_dim = 1.0)
llama_context: n_seq_max = 1
llama_context: n_ctx = 16384
llama_context: n_ctx_seq = 16384
llama_context: n_batch = 1024
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = auto
llama_context: kv_unified = false
llama_context: freq_base = 10000.0
llama_context: freq_scale = 0.025
llama_context: n_ctx_seq (16384) < n_ctx_train (163840) -- the full capacity of the model will not be utilized
2026-01-30 22:09:40 [DEBUG]
llama_context: CUDA_Host output buffer size = 0.49 MiB
2026-01-30 22:09:40 [DEBUG]
llama_kv_cache: CPU KV buffer size = 78080.00 MiB
2026-01-30 22:10:30 [DEBUG]
llama_kv_cache: size = 78080.00 MiB ( 16384 cells, 61 layers, 1/1 seqs), K (f16): 46848.00 MiB, V (f16): 31232.00 MiB
sched_reserve: reserving ...
2026-01-30 22:10:30 [DEBUG]
sched_reserve: layer 0 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support)
sched_reserve: Flash Attention was auto, set to disabled
To Reproduce
Use DeepSeek R1 671b Q4-K-M
GPU Offload= 61
Offload KV cache to GPU Memory= No
Number of layers to force the experts into CPU=61
Flash Attention=Yes
Then load to Reproduce
Which version of LM Studio?
LM Studio 0.4.0
Which operating system?
Windows 10 Workstation
What is the bug?
NVIDIA RTX3080 20G,512G RAM.After upgrading CUDA 12 llama.cpp (Windows) to version1.104.2 and 2.0, when loading DeepSeek R1 671b Q4-K-M, the log shows the following warning: "sched_reserve: layer 0 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support). sched_reserve: Flash Attention was auto, set to disabled." However, with the exact same parameter configuration in the previous version CUDA 12 llama.cpp (Windows) 1.103.2, this issue did not occur, and the processing speed was several times faster.
Screenshots
capture_20260130233736264.bmp
Logs
2026-01-30 22:05:02 [DEBUG]
[LM Studio] GPU Configuration:
Strategy: evenly
Priority: []
Disabled GPUs: []
Limit weight offload to dedicated GPU Memory: OFF
Offload KV Cache to GPU: OFF
2026-01-30 22:05:02 [DEBUG]
[LM Studio] Live GPU memory info (source 'LMS Core'):
GPU 0: NVIDIA GeForce RTX 3080 (Used: 2.67 GB, Total: 21.47 GB, Free: 18.80 GB)
2026-01-30 22:05:02 [DEBUG]
[LM Studio] Model load size estimate with raw num offload layers 'max' and context length '16384':
Model: 416.56 GB
Context: 20.84 GB
Total: 437.40 GB
2026-01-30 22:05:02 [DEBUG]
[LM Studio] Not using full context length for VRAM overflow calculations due to single GPU setup. Instead, using '8192' as context length for the calculation. Original context length: '16384'.
[LM Studio] Strict GPU VRAM cap is OFF: GPU offload layers will not be checked for adjustment
2026-01-30 22:05:02 [DEBUG]
[LM Studio] Resolved GPU config options:
Num Offload Layers: max
Num CPU Expert Layers: 71
Main GPU: 0
Tensor Split: [0]
Disabled GPUs: []
2026-01-30 22:05:02 [DEBUG]
LlamaV4::load called with model path:...
LlamaV4::load config: n_parallel=1 n_ctx=16384 kv_unified=false
2026-01-30 22:05:02 [DEBUG]
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes
2026-01-30 22:05:02 [DEBUG]
DeepSeek2 architecture detected, using AUTO flash attention mode
2026-01-30 22:05:02 [DEBUG]
srv load_model: loading model '...'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3080) (0000:03:00.0) - 19271 MiB free
llama_model_loader: ------------------------ Adding override for key 'deepseek2.expert_used_count'
2026-01-30 22:05:02 [DEBUG]
llama_model_loader: loaded meta data with 52 key-value pairs and 1025 tensors from ... (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = deepseek2
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = DeepSeek V3 Bf16
llama_model_loader: - kv 3: general.finetune str = abliterated
llama_model_loader: - kv 4: general.basename str = DeepSeek-V3
llama_model_loader: - kv 5: general.size_label str = 256x20B
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: general.base_model.count u32 = 1
llama_model_loader: - kv 8: general.base_model.0.name str = DeepSeek V3
llama_model_loader: - kv 9: general.base_model.0.version str = V3
llama_model_loader: - kv 10: general.base_model.0.organization str = Deepseek Ai
llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/deepseek-ai/De...
llama_model_loader: - kv 12: general.tags arr[str,4] = ["deepseek_v3", "bf16", "Safetensors"...
llama_model_loader: - kv 13: deepseek2.block_count u32 = 61
llama_model_loader: - kv 14: deepseek2.context_length u32 = 163840
llama_model_loader: - kv 15: deepseek2.embedding_length u32 = 7168
llama_model_loader: - kv 16: deepseek2.feed_forward_length u32 = 18432
llama_model_loader: - kv 17: deepseek2.attention.head_count u32 = 128
llama_model_loader: - kv 18: deepseek2.attention.head_count_kv u32 = 128
llama_model_loader: - kv 19: deepseek2.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 20: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 21: deepseek2.expert_used_count u32 = 8
llama_model_loader: - kv 22: deepseek2.leading_dense_block_count u32 = 3
llama_model_loader: - kv 23: deepseek2.vocab_size u32 = 129280
llama_model_loader: - kv 24: deepseek2.attention.q_lora_rank u32 = 1536
llama_model_loader: - kv 25: deepseek2.attention.kv_lora_rank u32 = 512
llama_model_loader: - kv 26: deepseek2.attention.key_length u32 = 192
llama_model_loader: - kv 27: deepseek2.attention.value_length u32 = 128
llama_model_loader: - kv 28: deepseek2.expert_feed_forward_length u32 = 2048
llama_model_loader: - kv 29: deepseek2.expert_count u32 = 256
llama_model_loader: - kv 30: deepseek2.expert_shared_count u32 = 1
llama_model_loader: - kv 31: deepseek2.expert_weights_scale f32 = 2.500000
llama_model_loader: - kv 32: deepseek2.expert_weights_norm bool = true
llama_model_loader: - kv 33: deepseek2.expert_gating_func u32 = 2
llama_model_loader: - kv 34: deepseek2.rope.dimension_count u32 = 64
llama_model_loader: - kv 35: deepseek2.rope.scaling.type str = yarn
llama_model_loader: - kv 36: deepseek2.rope.scaling.factor f32 = 40.000000
llama_model_loader: - kv 37: deepseek2.rope.scaling.original_context_length u32 = 4096
2026-01-30 22:05:02 [DEBUG]
llama_model_loader: - kv 38: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000
llama_model_loader: - kv 39: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 40: tokenizer.ggml.pre str = deepseek-v3
2026-01-30 22:05:02 [DEBUG]
llama_model_loader: - kv 41: tokenizer.ggml.tokens arr[str,129280] = ["<｜begin▁of▁sentence｜>", "<�...
2026-01-30 22:05:02 [DEBUG]
llama_model_loader: - kv 42: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
2026-01-30 22:05:02 [DEBUG]
llama_model_loader: - kv 43: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
llama_model_loader: - kv 44: tokenizer.ggml.bos_token_id u32 = 0
llama_model_loader: - kv 45: tokenizer.ggml.eos_token_id u32 = 1
llama_model_loader: - kv 46: tokenizer.ggml.padding_token_id u32 = 1
llama_model_loader: - kv 47: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 48: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 49: tokenizer.chat_template str = {% if not add_generation_prompt is de...
llama_model_loader: - kv 50: general.quantization_version u32 = 2
llama_model_loader: - kv 51: general.file_type u32 = 15
llama_model_loader: - type f32: 361 tensors
llama_model_loader: - type q4_K: 606 tensors
llama_model_loader: - type q6_K: 58 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 376.65 GiB (4.82 BPW)
validate_override: Using metadata override ( int) 'deepseek2.expert_used_count' = 8
load_hparams: ----------------------- n_expert_used = 8 n_expert_groups = 0
2026-01-30 22:05:02 [DEBUG]
load: 0 unused tokens
2026-01-30 22:05:02 [DEBUG]
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load: - 1 ('<｜end▁of▁sentence｜>')
2026-01-30 22:05:02 [DEBUG]
load: special tokens cache size = 818
2026-01-30 22:05:02 [DEBUG]
load: token to piece cache size = 0.8223 MB
print_info: arch = deepseek2
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 163840
print_info: n_embd = 7168
print_info: n_embd_inp = 7168
print_info: n_layer = 61
print_info: n_head = 128
print_info: n_head_kv = 128
print_info: n_rot = 64
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 192
print_info: n_embd_head_v = 128
print_info: n_gqa = 1
print_info: n_embd_k_gqa = 24576
print_info: n_embd_v_gqa = 16384
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 18432
print_info: n_expert = 256
print_info: n_expert_used = 8
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 0
print_info: rope scaling = yarn
print_info: freq_base_train = 10000.0
print_info: freq_scale_train = 0.025
print_info: n_ctx_orig_yarn = 4096
print_info: rope_yarn_log_mul = 1.0000
print_info: rope_finetuned = unknown
print_info: model type = 671B
print_info: model params = 671.03 B
print_info: general.name = DeepSeek V3 Bf16
print_info: n_layer_dense_lead = 3
print_info: n_lora_q = 1536
print_info: n_lora_kv = 512
print_info: n_embd_head_k_mla = 0
print_info: n_embd_head_v_mla = 0
print_info: n_ff_exp = 2048
print_info: n_expert_shared = 1
print_info: expert_weights_scale = 2.5
print_info: expert_weights_norm = 1
print_info: expert_gating_func = sigmoid
print_info: vocab type = BPE
print_info: n_vocab = 129280
print_info: n_merges = 127741
print_info: BOS token = 0 '<｜begin▁of▁sentence｜>'
print_info: EOS token = 1 '<｜end▁of▁sentence｜>'
print_info: EOT token = 1 '<｜end▁of▁sentence｜>'
print_info: PAD token = 1 '<｜end▁of▁sentence｜>'
print_info: LF token = 201 'Ċ'
print_info: FIM PRE token = 128801 '<｜fim▁begin｜>'
print_info: FIM SUF token = 128800 '<｜fim▁hole｜>'
print_info: FIM MID token = 128802 '<｜fim▁end｜>'
print_info: EOG token = 1 '<｜end▁of▁sentence｜>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
2026-01-30 22:05:41 [DEBUG]
load_tensors: offloading output layer to GPU
load_tensors: offloading 60 repeating layers to GPU
load_tensors: offloaded 62/62 layers to GPU
load_tensors: CPU_Mapped model buffer size = 384956.77 MiB
load_tensors: CUDA0 model buffer size = 9460.52 MiB
2026-01-30 22:09:40 [DEBUG]
common_init_result: added <｜end▁of▁sentence｜> logit bias = -inf
2026-01-30 22:09:40 [DEBUG]
llama_context: constructing llama_context
llama_context: setting new yarn_attn_factor = 1.0000 (mscale == 1.0, mscale_all_dim = 1.0)
llama_context: n_seq_max = 1
llama_context: n_ctx = 16384
llama_context: n_ctx_seq = 16384
llama_context: n_batch = 1024
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = auto
llama_context: kv_unified = false
llama_context: freq_base = 10000.0
llama_context: freq_scale = 0.025
llama_context: n_ctx_seq (16384) < n_ctx_train (163840) -- the full capacity of the model will not be utilized
2026-01-30 22:09:40 [DEBUG]
llama_context: CUDA_Host output buffer size = 0.49 MiB
2026-01-30 22:09:40 [DEBUG]
llama_kv_cache: CPU KV buffer size = 78080.00 MiB
2026-01-30 22:10:30 [DEBUG]
llama_kv_cache: size = 78080.00 MiB ( 16384 cells, 61 layers, 1/1 seqs), K (f16): 46848.00 MiB, V (f16): 31232.00 MiB
sched_reserve: reserving ...
2026-01-30 22:10:30 [DEBUG]
sched_reserve: layer 0 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support)
sched_reserve: Flash Attention was auto, set to disabled
To Reproduce
Use DeepSeek R1 671b Q4-K-M
GPU Offload= 61
Offload KV cache to GPU Memory= No
Number of layers to force the experts into CPU=61
Flash Attention=Yes
Then load to Reproduce