Name and Version
I used the following two models:
https://huggingface.co/froggeric/Qwen3.6-27B-MTP-GGUF/tree/main
https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF/tree/main
However, an error occurred during startup. The error log is as follows:
print_info: freq_base_train = 10000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: mrope sections = [11, 11, 10, 0]
print_info: model type = ?B
print_info: model params = 27.32 B
print_info: general.name = Qwen3.6 27B
print_info: vocab type = BPE
print_info: n_vocab = 248320
print_info: n_merges = 247587
print_info: BOS token = 248044 '<|endoftext|>'
print_info: EOS token = 248046 '<|im_end|>'
print_info: EOT token = 248046 '<|im_end|>'
print_info: PAD token = 248044 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 248060 '<|fim_prefix|>'
print_info: FIM SUF token = 248062 '<|fim_suffix|>'
print_info: FIM MID token = 248061 '<|fim_middle|>'
print_info: FIM PAD token = 248063 '<|fim_pad|>'
print_info: FIM REP token = 248064 '<|repo_name|>'
print_info: FIM SEP token = 248065 '<|file_sep|>'
print_info: EOG token = 248044 '<|endoftext|>'
print_info: EOG token = 248046 '<|im_end|>'
print_info: EOG token = 248063 '<|fim_pad|>'
print_info: EOG token = 248064 '<|repo_name|>'
print_info: EOG token = 248065 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
llama_model_load: error loading model: invalid vector subscript
�[0mllama_model_load_from_file_impl: failed to load model
�[0msrv load_model: failed to load MTP head from 'Qwen3.6-27B-Q4_K_M-mtp.gguf'
�[0msrv operator (): operator (): cleaning up before exit...
main: exiting due to model loading error
�[0mPress any key to continue . . .
The graphics card I'm using is an RTX 4080 Super with 16GB of VRAM with CUDA 12.4, and I have 32GB of system RAM.Below are my startup configuration parameters:
@echo off
echo.
cd /d "D:\mtp-turboquant"
set CUDA_SCALE_LAUNCH_QUEUES=4x
.\llama-server.exe ^
-m Qwen3.6-27B-Q4_K_M-mtp.gguf ^
--mmproj mmproj-Qwen3.6-27B-f16.gguf ^
-fa on ^
--spec-type mtp --spec-draft-n-max 2 ^
-ctk q8_0 -ctv turbo3 ^
--n-gpu-layers 99 ^
--temp 0.1 --top-k 20 ^
--flash-attn on ^
--reasoning off ^
--no-mmap --mlock ^
--reasoning-format deepseek ^
--repeat-penalty 1.15 ^
--presence-penalty 1.5 ^
--frequency-penalty 0.0 ^
-np 1 ^
-t 8 ^
--host 0.0.0.0 --port 8189 ^
-c 156888 -b 2048 -ub 512 ^
--jinja ^
--chat-template-file "3.6_chat_template-v10.jinja"
pause
My compilation script is:
@echo off
cd /d "D:\Cmake\llamacpp-MTP-TurboQuant"
cmake -B build ^
-G "Visual Studio 17 2022" -A x64 -T "cuda=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.4" ^
-DGGML_CUDA=ON ^
-DCMAKE_CUDA_ARCHITECTURES="89" ^
-DGGML_CUDA_FA_ALL_QUANTS=ON ^
-DGGML_NATIVE=OFF ^
-DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 14 --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
So, please help me see where I went wrong?
Operating systems
Windows
GGML backends
CUDA
Hardware
14700KF+RTX4080Super+32GB RAM
D:\mtp-turboquant>nvidia-smi
Sun May 24 22:07:27 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 551.78 Driver Version: 551.78 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4080 ... WDDM | 00000000:01:00.0 Off | N/A |
| 30% 29C P8 14W / 320W | 582MiB / 16376MiB | 8% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 5664 C+G ...5n1h2txyewy\ShellExperienceHost.exe N/A |
| 0 N/A N/A 6784 C+G C:\Windows\explorer.exe N/A |
| 0 N/A N/A 7376 C+G ....Search_cw5n1h2txyewy\SearchApp.exe N/A |
| 0 N/A N/A 8536 C+G ...CBS_cw5n1h2txyewy\TextInputHost.exe N/A |
| 0 N/A N/A 8904 C+G ...ky-borders-v1.3.1\tacky-borders.exe N/A |
| 0 N/A N/A 10620 C+G D:\v2rayN-windows-64\v2rayN.exe N/A |
| 0 N/A N/A 10704 C+G D:\Chrome++\App\chrome.exe N/A |
| 0 N/A N/A 11188 C+G ...ms@opencode-aidesktop\OpenCode.exe N/A |
+-----------------------------------------------------------------------------------------+
Models
Qwen3.6-27B-Q4_K_M-mtp.gguf
Problem description & steps to reproduce
failed to load MTP head from 'Qwen3.6-27B-Q4_K_M-mtp.gguf'
First Bad Commit
No response
Relevant log output
Logs
Name and Version
I used the following two models:
https://huggingface.co/froggeric/Qwen3.6-27B-MTP-GGUF/tree/main
https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF/tree/main
However, an error occurred during startup. The error log is as follows:
print_info: freq_base_train = 10000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: mrope sections = [11, 11, 10, 0]
print_info: model type = ?B
print_info: model params = 27.32 B
print_info: general.name = Qwen3.6 27B
print_info: vocab type = BPE
print_info: n_vocab = 248320
print_info: n_merges = 247587
print_info: BOS token = 248044 '<|endoftext|>'
print_info: EOS token = 248046 '<|im_end|>'
print_info: EOT token = 248046 '<|im_end|>'
print_info: PAD token = 248044 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 248060 '<|fim_prefix|>'
print_info: FIM SUF token = 248062 '<|fim_suffix|>'
print_info: FIM MID token = 248061 '<|fim_middle|>'
print_info: FIM PAD token = 248063 '<|fim_pad|>'
print_info: FIM REP token = 248064 '<|repo_name|>'
print_info: FIM SEP token = 248065 '<|file_sep|>'
print_info: EOG token = 248044 '<|endoftext|>'
print_info: EOG token = 248046 '<|im_end|>'
print_info: EOG token = 248063 '<|fim_pad|>'
print_info: EOG token = 248064 '<|repo_name|>'
print_info: EOG token = 248065 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
llama_model_load: error loading model: invalid vector subscript
�[0mllama_model_load_from_file_impl: failed to load model
�[0msrv load_model: failed to load MTP head from 'Qwen3.6-27B-Q4_K_M-mtp.gguf'
�[0msrv operator (): operator (): cleaning up before exit...
main: exiting due to model loading error
�[0mPress any key to continue . . .
The graphics card I'm using is an RTX 4080 Super with 16GB of VRAM with CUDA 12.4, and I have 32GB of system RAM.Below are my startup configuration parameters:
@echo off
echo.
cd /d "D:\mtp-turboquant"
set CUDA_SCALE_LAUNCH_QUEUES=4x
.\llama-server.exe ^
-m Qwen3.6-27B-Q4_K_M-mtp.gguf ^
--mmproj mmproj-Qwen3.6-27B-f16.gguf ^
-fa on ^
--spec-type mtp --spec-draft-n-max 2 ^
-ctk q8_0 -ctv turbo3 ^
--n-gpu-layers 99 ^
--temp 0.1 --top-k 20 ^
--flash-attn on ^
--reasoning off ^
--no-mmap --mlock ^
--reasoning-format deepseek ^
--repeat-penalty 1.15 ^
--presence-penalty 1.5 ^
--frequency-penalty 0.0 ^
-np 1 ^
-t 8 ^
--host 0.0.0.0 --port 8189 ^
-c 156888 -b 2048 -ub 512 ^
--jinja ^
--chat-template-file "3.6_chat_template-v10.jinja"
pause
My compilation script is:
@echo off
cd /d "D:\Cmake\llamacpp-MTP-TurboQuant"
cmake -B build ^
-G "Visual Studio 17 2022" -A x64 -T "cuda=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.4" ^
-DGGML_CUDA=ON ^
-DCMAKE_CUDA_ARCHITECTURES="89" ^
-DGGML_CUDA_FA_ALL_QUANTS=ON ^
-DGGML_NATIVE=OFF ^
-DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 14 --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
So, please help me see where I went wrong?
Operating systems
Windows
GGML backends
CUDA
Hardware
14700KF+RTX4080Super+32GB RAM
D:\mtp-turboquant>nvidia-smi
Sun May 24 22:07:27 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 551.78 Driver Version: 551.78 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4080 ... WDDM | 00000000:01:00.0 Off | N/A |
| 30% 29C P8 14W / 320W | 582MiB / 16376MiB | 8% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 5664 C+G ...5n1h2txyewy\ShellExperienceHost.exe N/A |
| 0 N/A N/A 6784 C+G C:\Windows\explorer.exe N/A |
| 0 N/A N/A 7376 C+G ....Search_cw5n1h2txyewy\SearchApp.exe N/A |
| 0 N/A N/A 8536 C+G ...CBS_cw5n1h2txyewy\TextInputHost.exe N/A |
| 0 N/A N/A 8904 C+G ...ky-borders-v1.3.1\tacky-borders.exe N/A |
| 0 N/A N/A 10620 C+G D:\v2rayN-windows-64\v2rayN.exe N/A |
| 0 N/A N/A 10704 C+G D:\Chrome++\App\chrome.exe N/A |
| 0 N/A N/A 11188 C+G ...ms@opencode-aidesktop\OpenCode.exe N/A |
+-----------------------------------------------------------------------------------------+
Models
Qwen3.6-27B-Q4_K_M-mtp.gguf
Problem description & steps to reproduce
failed to load MTP head from 'Qwen3.6-27B-Q4_K_M-mtp.gguf'
First Bad Commit
No response
Relevant log output
Logs