0.00.135.607 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.135.615 I device_info:
0.00.278.084 I - CUDA0 : NVIDIA GeForce RTX 4060 (8187 MiB, 7107 MiB free)
0.00.278.103 I - CPU : Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (24530 MiB, 11688 MiB free)
0.00.278.208 I system_info: n_threads = 4 (n_threads_batch = 4) / 4 | CUDA : ARCHS = 750,800,860,890,900,1200,1210 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.278.220 W srv llama_server: embeddings enabled with n_batch (2048) > n_ubatch (512)
0.00.278.224 W srv llama_server: setting n_batch = n_ubatch = 512 to avoid assertion failure
0.00.278.228 I srv llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.278.318 I srv init: running without SSL
0.00.278.370 I srv init: using 8 threads for HTTP server
0.00.278.650 I srv start: binding port with default address family
0.00.283.340 I srv llama_server: loading model
0.00.283.358 I srv load_model: loading model '.\models\unsloth\gemma-4-12B-it-qat-GGUF\gemma-4-12B-it-qat-UD-Q4_K_XL.gguf'
0.00.799.112 I srv load_model: [mtmd] estimated worst-case memory usage of mmproj is 354.46 MiB
0.02.285.755 E llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this is normal during memory fitting)
0.02.372.946 W srv load_model: [spec] failed to measure draft model memory: failed to create llama_context from model
0.02.372.986 I common_init_result: fitting params to device memory ...
0.02.372.987 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.04.633.037 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort
0.05.716.945 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.05.717.789 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.05.782.790 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.11.291.285 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.11.752.913 I srv load_model: loading draft model '.\models\unsloth\gemma-4-12B-it-qat-GGUF\mtp-gemma-4-12B-it.gguf'
0.12.828.249 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.12.828.998 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.12.921.572 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.13.053.534 E llama_model_load: error loading model: invalid vector subscript
0.13.053.546 E llama_model_load_from_file_impl: failed to load model
0.13.053.549 E srv load_model: failed to load draft model, '.\models\unsloth\gemma-4-12B-it-qat-GGUF\mtp-gemma-4-12B-it.gguf'
0.13.053.577 I srv operator(): operator(): cleaning up before exit...
0.13.054.524 E srv llama_server: exiting due to model loading error
Name and Version
F:\aimodels>.\llama\llama-cli --version
version: 9587 (d2e22ed)
built with Clang 20.1.8 for Windows x86_64
Operating systems
Windows
GGML backends
CUDA
Hardware
4060
Models
https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF/tree/main
Problem description & steps to reproduce
.\llama\llama-server^
-m .\models\unsloth\gemma-4-12B-it-qat-GGUF\gemma-4-12B-it-qat-UD-Q4_K_XL.gguf^
--mmproj .\models\unsloth\gemma-4-12B-it-qat-GGUF\mmproj-BF16.gguf^
--model-draft .\models\unsloth\gemma-4-12B-it-qat-GGUF\mtp-gemma-4-12B-it.gguf^
--spec-type draft-mtp^
--spec-draft-n-max 2^
-ngl 99^
-c 0^
-t 4^
--flash-attn on^
--n-predict -1^
--batch-size 2048^
--host 0.0.0.0^
--port 18080^
--embedding^
--temp 0.3^
--top-p 0.9^
--repeat-penalty 1.1^
--cache-type-k q4_0^
--cache-type-v q4_0^
--jinja
First Bad Commit
No response
Relevant log output
Logs