Skip to content

MTMD Vision extremely slow: mtmd_encode_chunk() takes ~82 seconds per image slice (llama-server only, llama-cli is fast) #22582

Description

@sswtodo

ISSUE DESCRIPTION
I encountered a severe performance problem when using llama-server with MTMD and a Vision model (Qwen3.5‑9B‑Vision). The same model and the same image run extremely fast in llama-cli, but become unusably slow in llama-server, where Vision encoding takes over 82 seconds before any tokens are generated.

This problem affects only llama-server (MTMD path).
llama-cli processes the same image almost instantly.

Image Used for Test
7.44 Mb , ~4k resolution
https://freenaturestock.com/bird-calling-from-a-tree/

ENVIRONMENT
CPU: AMD Ryzen 9 7950X
GPU: 7900 xtx
OS: FreeBSD/Linux (both checked, same delay)
llama.cpp: latest commit (fresh clone today)
Model:
Qwen3.5-9B-Q4_K_M.gguf
mmproj-Qwen3.5-9B-BF16.gguf

COMMAND USED

llama-server \
  --jinja \
  --threads 30 \
  --gpu-layers 999 \
  --ctx-size 131072 \
  --batch-size 2048 \
  --ubatch-size 512 \
  --temperature 0.7 \
  --top-k 20 \
  --top-p 0.8 \
  --min-p 0.0 \
  --repeat-penalty 1.0 \
  --presence-penalty 1.5 \
  --flash-attn off \
  --reasoning off \
  --reasoning-budget 0 \
  --host 0.0.0.0 \
  --port 8080 \
  --mmproj mmproj-Qwen3.5-9B-BF16.gguf \
  -m Qwen3.5-9B-Q4_K_M.gguf

LOG OUTPUT (critical part)
encoding image slice...
image slice encoded in 82047 ms

Full Context log

srv  process_chun: processing image...
encoding image slice...

image slice encoded in 82047 ms
decoding image batch 1/2, n_tokens_batch = 2048
image decoded (batch 1/2) in 945 ms
decoding image batch 2/2, n_tokens_batch = 2008
image decoded (batch 2/2) in 821 ms
srv  process_chun: image processed in 83813 ms

Root Cause
mtmd_encode_chunk(ctx, chunk) -> called from mtmd_helper_eval_chunk_single()

This function performs the Vision encoding step for MTMD, and in the current implementation it appears to run the full Vision encoder (mmproj + vision tower) on CPU in BF16 for every image slice sequentially without GPU acceleration This explains why llama-cli is fast (different pipeline, GPU-friendly), while llama-server (MTMD) is extremely slow.

Version
version: 8994 (aab68217b)

Actual Behaviour
llama-cli: fast, image processed almost instantly
llama-server (MTMD): 82 seconds spent inside mtmd_encode_chunk()
Vision inference becomes unusable

Expected Behaviour
Vision encoding in llama-server should be comparable to llama-cli or at least not take 82 seconds per image MTMD Vision path should use GPU when available BF16 mmproj should not run on CPU if GPU layers are enabled.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions