Skip to content

Eval bug: GGML_ASSERT(offset + size <= ggml_nbytes(tensor) && "tensor read out of bounds") failed in Qwen3 VL Embedding #18910

Description

@fgst666

Name and Version

version: 7767 (10c98cb)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

GGML backends

CPU

Hardware

10th core 10400

Models

qwen3 VL embedding 2B
https://huggingface.co/DevQuasar/Qwen.Qwen3-VL-Embedding-2B-GGUF/blob/main/Qwen.Qwen3-VL-Embedding-2B.Q4_K_M.gguf

Problem description & steps to reproduce

llama-server.exe -m Qwen3-VL-Embedding-2B.Q4_K_M.gguf --embedding --pooling last -c 256 -ub 256 --host 0.0.0.0 --port 8081 --cache-ram 0

error occurred in warming up, then program quit
……
llama_kv_cache: size = 28.00 MiB ( 256 cells, 28 layers, 4/1 seqs), K (f16): 14.00 MiB, V (f16): 14.00 MiB
sched_reserve: reserving ...
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: CPU compute buffer size = 150.63 MiB
sched_reserve: graph nodes = 988
sched_reserve: graph splits = 1
sched_reserve: reserve took 4.79 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
�[0mD:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:271: GGML_ASSERT(offset + size <= ggml_nbytes(tensor) && "tensor read out of bounds") failed

If use --no-warmup, the error occurred at the first request, and the error is the same as above.

First Bad Commit

No response

Relevant log output

llama-server.exe -m G:\ai\Qwen3-VL-Embedding-2B.Q4_K_M.gguf --embedding --pooling last -c 256 -ub 256 --host 0.0.0.0 --port 8081 --cache-ram 0 >llamalog.txt
load_backend: loaded RPC backend from W:\ai\llama-b7767-bin-win-cpu-x64\ggml-rpc.dll
load_backend: loaded CPU backend from W:\ai\llama-b7767-bin-win-cpu-x64\ggml-cpu-haswell.dll
main: embeddings enabled with n_batch (2048) > n_ubatch (256)
main: setting n_batch = n_ubatch = 256 to avoid assertion failure
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 7767 (10c98cb) with Clang 19.1.5 for Windows x86_64
system info: n_threads = 6, n_threads_batch = 6, total_threads = 12

system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

Running without SSL
init: using 11 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model 'G:\ai\Qwen3-VL-Embedding-2B.Q4_K_M.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: no devices with dedicated memory found
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.30 seconds
llama_model_loader: direct I/O is enabled, disabling mmap
llama_model_loader: loaded meta data with 37 key-value pairs and 310 tensors from G:\ai\Qwen3-VL-Embedding-2B.Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen3vl
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Qwen.Qwen3 VL Embedding 2B
llama_model_loader: - kv 3: general.basename str = Qwen.Qwen3-VL-Embedding
llama_model_loader: - kv 4: general.size_label str = 2B
llama_model_loader: - kv 5: general.license str = apache-2.0
llama_model_loader: - kv 6: general.base_model.count u32 = 1
llama_model_loader: - kv 7: general.base_model.0.name str = Qwen3 VL 2B Instruct
llama_model_loader: - kv 8: general.base_model.0.organization str = Qwen
llama_model_loader: - kv 9: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen3-VL-...
llama_model_loader: - kv 10: general.tags arr[str,2] = ["transformers", "multimodal embedding"]
llama_model_loader: - kv 11: qwen3vl.block_count u32 = 28
llama_model_loader: - kv 12: qwen3vl.context_length u32 = 262144
llama_model_loader: - kv 13: qwen3vl.embedding_length u32 = 2048
llama_model_loader: - kv 14: qwen3vl.feed_forward_length u32 = 6144
llama_model_loader: - kv 15: qwen3vl.attention.head_count u32 = 16
llama_model_loader: - kv 16: qwen3vl.attention.head_count_kv u32 = 8
llama_model_loader: - kv 17: qwen3vl.rope.dimension_sections arr[i32,4] = [24, 20, 20, 0]
llama_model_loader: - kv 18: qwen3vl.rope.freq_base f32 = 5000000.000000
llama_model_loader: - kv 19: qwen3vl.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 20: qwen3vl.attention.key_length u32 = 128
llama_model_loader: - kv 21: qwen3vl.attention.value_length u32 = 128
llama_model_loader: - kv 22: qwen3vl.n_deepstack_layers u32 = 3
llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2
llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,151936] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["臓 臓", "臓臓 臓臓", "i n", "臓 t",...
llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151643
llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 30: tokenizer.ggml.eot_token_id u32 = 151645
llama_model_loader: - kv 31: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 32: tokenizer.ggml.add_eos_token bool = true
llama_model_loader: - kv 33: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 34: tokenizer.chat_template str = {%- set default_system_message = 'Rep...
llama_model_loader: - kv 35: general.quantization_version u32 = 2
llama_model_loader: - kv 36: general.file_type u32 = 15
llama_model_loader: - type f32: 113 tensors
llama_model_loader: - type q4_K: 168 tensors
llama_model_loader: - type q6_K: 29 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 1.03 GiB (5.12 BPW)
load: 0 unused tokens
load: printing all EOG tokens:
load: - 151643 ('<|endoftext|>')
load: - 151645 ('<|im_end|>')
load: - 151662 ('<|fim_pad|>')
load: - 151663 ('<|repo_name|>')
load: - 151664 ('<|file_sep|>')
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch = qwen3vl
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd = 2048
print_info: n_embd_inp = 8192
print_info: n_layer = 28
print_info: n_head = 16
print_info: n_head_kv = 8
print_info: n_rot = 128
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 2
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 6144
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 40
print_info: rope scaling = linear
print_info: freq_base_train = 5000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: mrope sections = [24, 20, 20, 0]
print_info: model type = 1.7B
print_info: model params = 1.72 B
print_info: general.name = Qwen.Qwen3 VL Embedding 2B
print_info: vocab type = BPE
print_info: n_vocab = 151936
print_info: n_merges = 151387
print_info: BOS token = 151643 '<|endoftext|>'
print_info: EOS token = 151643 '<|endoftext|>'
print_info: EOT token = 151645 '<|im_end|>'
print_info: PAD token = 151643 '<|endoftext|>'
print_info: LF token = 198 '膴'
print_info: FIM PRE token = 151659 '<|fim_prefix|>'
print_info: FIM SUF token = 151661 '<|fim_suffix|>'
print_info: FIM MID token = 151660 '<|fim_middle|>'
print_info: FIM PAD token = 151662 '<|fim_pad|>'
print_info: FIM REP token = 151663 '<|repo_name|>'
print_info: FIM SEP token = 151664 '<|file_sep|>'
print_info: EOG token = 151643 '<|endoftext|>'
print_info: EOG token = 151645 '<|im_end|>'
print_info: EOG token = 151662 '<|fim_pad|>'
print_info: EOG token = 151663 '<|repo_name|>'
print_info: EOG token = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = true)
load_tensors: offloading output layer to GPU
load_tensors: offloading 27 repeating layers to GPU
load_tensors: offloaded 29/29 layers to GPU
load_tensors: CPU model buffer size = 404.68 MiB
load_tensors: CPU_REPACK model buffer size = 645.75 MiB
..............................................................................
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|im_end|> logit bias = -inf
common_init_result: added <|fim_pad|> logit bias = -inf
common_init_result: added <|repo_name|> logit bias = -inf
common_init_result: added <|file_sep|> logit bias = -inf
llama_init_from_model: model default pooling_type is [0], but [3] was specified
llama_context: constructing llama_context
llama_context: n_seq_max = 4
llama_context: n_ctx = 256
llama_context: n_ctx_seq = 256
llama_context: n_batch = 256
llama_context: n_ubatch = 256
llama_context: causal_attn = 1
llama_context: flash_attn = auto
llama_context: kv_unified = true
llama_context: freq_base = 5000000.0
llama_context: freq_scale = 1
llama_context: n_ctx_seq (256) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context: CPU output buffer size = 2.35 MiB
llama_kv_cache: CPU KV buffer size = 28.00 MiB
llama_kv_cache: size = 28.00 MiB ( 256 cells, 28 layers, 4/1 seqs), K (f16): 14.00 MiB, V (f16): 14.00 MiB
sched_reserve: reserving ...
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: CPU compute buffer size = 150.63 MiB
sched_reserve: graph nodes = 988
sched_reserve: graph splits = 1
sched_reserve: reserve took 3.79 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
D:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:271: GGML_ASSERT(offset + size <= ggml_nbytes(tensor) && "tensor read out of bounds") failed

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions