Repository navigation
b12503
Pre-releaseggml-openvino: expand model support, optimize NPU,GPU and memory usage (#32172)
- ggml-openvino: fix SWA mask detection
When the SWA cache size matches the full cache size, identify SWA
layers by the number of layers using each mask instead of cache size;
the larger group is SWA, and a tie leaves the layers unclassified.
- ggml-openvino: fix SWA KV cache write indices
Give SWA KV cache index tensors unique input names (a _swa suffix, like
the masks) to prevent them from being merged with the full-attention
cache indices.
- ggml-openvino: classify SWA layers only when the mask shows the window
Two attention masks do not imply SWA: DeepSeek sparse attention uses two
full-attention masks with the same name. When cache extents match,
classify the larger mask group as SWA only once its mask has fewer keys
per row than the other mask. Keep the classification per KV cache buffer
so new sequences reuse the same graph, and include swa_layers when
checking graph reuse.
- ggml-openvino: classify SWA layers from the first graph
Waiting for the window to appear in the mask caused a mid-sequence graph
recompile (~9 s for Gemma-4 at -c 1024 after position 512).
SWA classification already gives each mask group its own mask, KV write
indices, and attention size, which is correct whether or not the mask is
a sliding window. Stateful mask rebuild recovers the window from the
mask, or the causal mask for full attention. Classify the larger of two
mask groups from the first graph and drop the per-KV-buffer bookkeeping.
- ggml-openvino: mark SWA fallback as a model-level heuristic
Add a TODO above the equal-extent fallback to document that the larger
mask group is currently assumed to be the SWA group. Since stateless
graphs only require the two groups to remain distinct, this heuristic
may only be necessary for stateful execution and could potentially be
restricted to that path.
- ggml-openvino: derive attention sizes from the KQ mask shape
Dynamic CPU/GPU graphs previously passed n_seq_active and
attention_size(_swa) as separate inputs. Derive these values directly
from the KQ mask shape using ShapeOf + Gather inside the model. This
allows the GPU plugin to infer the Q reshape and KV slice shapes from
shape inference rather than runtime input values. Q's head size also
becomes static, enabling the fused SDPA kernel.
GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK=1 restores the previous
separate-input behavior.
The fused SDPA path also requires two supporting changes:
- Clamp the GPU mask to -30000. A key block that is entirely -inf for
a row produces NaNs in the fused kernel. - Restate the head count and head size after the KV slice. The plugin
loses all slice dimensions when the end index is dynamic, causing
SDPA to fall back to the reference kernel.
Additional graph and quantization improvements:
- Skip the identity reshape of rows in SET_ROWS.
- Share a single Split for one-element view slices in dynamic GPU
graphs (e.g. Gemma-4 per-layer embeddings), replacing one
StridedSlice per layer. - Add GGML_OPENVINO_REQUANT_EMBD=native (off by default) to preserve
the file's native quantization for token_embd/output instead of
requantizing to Q8_0_C. - Include both environment settings in the graph and model cache keys
and document them.
- ggml-openvino: only reuse Split outputs for exact chunk-aligned views
The dynamic GPU path reused a Split output for any single-element view
by deriving the chunk index from offset / stride. This produced
incorrect results for views that were not chunk-aligned or whose
remaining dimensions did not preserve the source strides, as seen in
test-backend-ops CONT with use_view_slice.
Restrict Split reuse to cases where the view exactly matches a split
chunk, meaning the offset is aligned to the split stride and the
remaining non-unit dimensions preserve the original source strides.
Gemma-4 per-layer embedding slices continue to use the shared Split
path.
- ggml-openvino: initialize ModelExtraInputInfo shape fields
Initialize the newly added shape_source and shape_axis fields in
the ModelExtraInputInfo brace initializer used by
add_extra_inputs().
Without initializers for these fields, GCC emits
-Wmissing-field-initializers, causing CI to fail with
LLAMA_FATAL_WARNINGS.
- ggml-openvino: remove GGML_OPENVINO_REQUANT_EMBD
The option identifies embedding and output weights by their tensor
names, which are defined by core llama.cpp. Remove it from this PR;
preserving the file quantization of these weights will be addressed in
a separate change that identifies them by their roles in the graph.
- ggml-openvino: keep manual GQA attention 4-D
The manual GQA attention path reshaped Q, K, and V to 5-D so that
query heads sharing a KV head could broadcast against it. The GPU
plugin cannot fuse this graph: it materializes the attention scores
and copies the transposed K and V caches for every layer.
Instead, stack the query heads for each KV head along the sequence
axis (Q [B, H, S, D] -> [B, H_kv, factor * S, D]) and tile the mask
accordingly. This keeps the graph 4-D from the MatMuls through the
Softmax, allowing the GPU plugin to fuse the attention into its SDPA
kernel and fold the K and V transposes into the fused operation.
Clamp the mask to -30000, matching the SDPA path, because the fused
kernel can return NaN when an entire block is masked with -inf.
Models where the K or V head size differs from the Q head size now
use the SDPA path, since the manual path assumes one head size.
- ggml-openvino: use a 4-D stateless KV cache on GPU
In stateless GPU mode, bind the entire KV cache as a
[1, ctx, n_heads_kv, head_size] tensor, with one token per row,
instead of binding the first n_kv rows as a
[1, 1, ctx, n_heads_kv * head_size] tensor.
Write cache updates with ScatterUpdate along the token axis, and
read the cache through VariadicSplit and Transpose. This allows
the GPU plugin's StatelessKVFusion to replace cache reads and writes
with in-place StatelessKV nodes, avoiding full-cache copies on every
decode step and repeated memory pool allocations as the bound cache
size grows.
Set GGML_OPENVINO_DISABLE_KV_4D=1 to retain the previous layout.
-
make embedding to have dynamic shape for OV GPU
-
ggml-openvino: make the 4-D stateless KV cache opt-in
-
add support for Q5_0 data type in OV GPU backend
-
resolve CI test error
-
ggml-openvino: fix -fa off and optimize grouped-query attention
With -fa off, llama.cpp stores the V cache transposed, with value
elements as rows and tokens along each row. The KV slicing code assumes
one token per row and binds the wrong data when n_kv < kv_size. Detect
transposed caches with GgmlOvDecoder::is_kvcache_token_rows() and skip
slicing for them.
For grouped-query attention, avoid repeating K and V for each query
head. Stack the query heads for each KV head along the row dimension
and let SOFT_MAX apply the scale, mask, and softmax so the GPU plugin
can fuse the operation with the MatMuls.
Read only the current layer's V-cache rows when layers use different
cache sizes. Reject -fa off in stateful mode with a clear error and
document that it requires stateless mode.
- ggml-openvino: fall back to stateless execution when -fa off
With -fa off, the graph writes the V cache in transposed form, which is
not supported by the stateful KV cache. Detect this pattern in
compute_llm_params() as a SET_ROWS into a KV cache through a [1, n]
reshape.
When stateful execution is enabled, log a warning and fall back to
stateless execution instead of failing graph conversion.
Update the -fa off note in OPENVINO.md to document this behavior.
Co-authored-by: Suhas Somashekar suhas.somashekar@intel.com
- ggml-openvino: preserve flat KV cache layout when -fa is off
With -fa off, the 4-D KV cache layout introduced in #344 breaks MUL_MAT
attention for Phi-3.5 mini and Phi-4 mini, causing oneDNN to fail with
"could not construct a memory descriptor using strides".
Preserve the [1, 1, ctx, n_heads_kv * head_size] layout when the graph
writes a transposed V cache.
- ggml-openvino: extract Q5_0 weights when requantization is off
Q5_0 weights are requantized to Q4_0, but test-backend-ops builds naive
graphs, where requantization is skipped. Q5_0 then reached the normal
weight extraction, which had no Q5_0 case, so every Q5_0 MUL_MAT and
GET_ROWS op test failed with "Unsupported quantized type: q5_0".
Extract Q5_0 as symmetric i8 weights (q - 16) with one f16 scale per 32
weights, the same layout as Q8_0. Models are unchanged: they still
requantize Q5_0 to Q4_0.
- ggml-openvino: Optimize GDN output pre-scale for the NPU static path
The NPU GatedDeltaNet kernel writes its output in FP16 and flushes values below 6.1e-5 to zero.
- Refine the code as review comments
- Add envs desc into doc
- Refine the code with removing duplicated env
- ggml-openvino: reuse graph bindings between decode calls
Reduce host overhead outside infer() from 0.5–1.1 ms to 0.14–0.38 ms
per token on Panther Lake.
- Reuse KV cache tensors and skip redundant port bindings.
- Use persistent USM host tensors for small GPU inputs and outputs.
- Defer graph-wide consumer lookup until after shape checks.
- Return model and compute params by const reference.
- Preserve tensor extras when memory, shape and type are unchanged.
-
ggml-openvino: Add npuw dump envs for profiling
-
Add the comments of new npu dump envs into doc
-
ggml-openvino : ensure all leaves have unique names, remove input_srcs from graph key
-
openvino: slice NPU prefill KV cache by active length
Add GGML_OPENVINO_NPU_KV_SLICE as an opt-in NPU prefill optimization. When enabled, static prefill keeps the graph's attention dimension at the active prompt/KV length instead of expanding it to the full context capacity, and the KV-cache input/output tensors are bound with a shortened context axis over the same backing storage.
This reduces the amount of KV cache and attention-mask data imported by the NPU plugin and lets the prefill graph plan smaller attention buffers for early prompt chunks. For example, with a 4096-token context and a 256-token prefill chunk, the first chunk can expose KV/mask dimensions at 256 instead of 4096, avoiding work over padded cache rows.
The trade-off is graph stability: NPU static shapes now depend on the aligned active attention length for prefill, so a cached prefill model is reused only when the attention sizes match. Decode keeps a fixed full-context graph to avoid recompiling on every generated token, and SWA/ring-cache cases still fall back to full-size binding when the live prefix is not a simple leading slice.
- openvino: use L0 host tensors for NPU buffers
Add GGML_OPENVINO_NPU_L0_HOST_TENSORS as an opt-in NPU buffer allocation mode. When enabled, non-remote OpenVINO backend buffers on NPU are allocated through the NPU default context with create_host_tensor(), and ggml tensor data points directly at that OpenVINO-owned Level Zero host allocation.
The optimization targets the data path rather than the graph math: KV-cache and compute tensors can be passed back to the NPU plugin as OpenVINO tensors backed by importable Level Zero host memory. That avoids an extra ordinary-host-memory-to-plugin-staging copy on each infer boundary, which is especially useful for prefill where large KV/cache buffers dominate input and output traffic.
If Level Zero host allocation fails, the backend logs a warning and falls back to the existing allocation path. The destructor tracks OpenVINO-owned storage so it does not free memory owned by the host tensor. The trade-off is that this is NPU-specific, consumes Level Zero host allocation resources, and improves transfer/import overhead without changing attention compute complexity or tensor shapes.
- openvino: fill NPU prefill masks directly
Add GGML_OPENVINO_NPU_FAST_MASK as an opt-in NPU static prefill input-packing optimization. When enabled, the prefill attention mask is built directly in the OpenVINO input tensor instead of first materializing a padded std::vector and then copying that whole buffer into the tensor.
The new fill_prefill_mask() helper writes each row once: it copies the valid mask prefix, fills padding columns with -inf, handles padded rows, and restores the causal diagonal to zero. This keeps the final mask numerically identical to the existing pad_input() plus set_zero_diagonal() path while removing one intermediate allocation and one chunk_size-by-context_size host copy per prefill chunk.
The optimization targets CPU-side staging overhead, not attention math. It composes with GGML_OPENVINO_NPU_KV_SLICE: KV slice reduces the mask dimensions, while FAST_MASK reduces the work needed to construct whatever mask shape remains. The trade-off is a second mask construction path to maintain, with limited benefit for very small prompts or contexts.
- openvino: make NPU requant layout configurable
Add GGML_OPENVINO_NPU_REQUANT_POLICY as the NPU-side selector for the default weight requantization layout. The existing NPU behavior remains the default: unset or group-128 maps quantized weights to Q4_0_128, while channel-wise selects the experimental Q4_0_C layout. Unknown policy values fail fast during model loading instead of silently compiling a graph with unintended weight packing.
The optimization target is NPU weight bandwidth and compressed-weight code generation. Q4_0_128 keeps weights in a 4-bit symmetric layout with one scale per 128 values, reducing both payload bytes and scale metadata versus smaller groups while preserving a more conservative quantization granularity than channel-wise. Q4_0_C uses one scale over the innermost dimension, which can further shrink metadata and simplify dequant broadcast, but may cost accuracy because the quantization block is much larger.
Include the selected NPU requant policy in the OpenVINO model cache discriminator. The same cgraph compiled with group-128 and channel-wise has different weight constants and dequant shapes, so reusing a cached decoder across policies would be incorrect. This keeps cache reuse stable within a policy while forcing recompilation when the layout policy changes.
Trade-off: policy selection gives a performance/accuracy knob for NPU experiments, but all options still pay the load/compile-time requantization cost. The default remains group-128 for a safer balance of bandwidth, metadata, and accuracy; channel-wise is intentionally documented as experimental.
- openvino: centralize NPU plugin config overrides
Add GGML_OPENVINO_NPU_CONFIG as the recommended catch-all override for OpenVINO NPU plugin and compiler properties. The value is parsed as comma-separated KEY=VALUE pairs and applied after the built-in defaults and named NPU aliases, so advanced users can override any exposed property without adding a new llama.cpp environment variable for every plugin tuning knob.
Keep the common knobs as named aliases for discoverability and compatibility: GGML_OPENVINO_NPU_COMPILER_TYPE maps to NPU_COMPILER_TYPE, GGML_OPENVINO_NPUW_FUNCALL_FOR_ALL maps to NPUW_FUNCALL_FOR_ALL, GGML_OPENVINO_NPUW_UNFOLD_IREQS maps to NPUW_UNFOLD_IREQS, and GGML_OPENVINO_COMPILATION_NUM_THREADS maps to COMPILATION_NUM_THREADS. GGML_OPENVINO_NPU_COMPILE_CONFIG is retained as a legacy alias for NPU_COMPILATION_MODE_PARAMS, while the documentation now recommends expressing it through GGML_OPENVINO_NPU_CONFIG.
This keeps the public tuning surface compact while preserving high-signal aliases for settings with known performance or stability implications. NPU_COMPILER_TYPE can select the driver compiler, which currently generates faster prefill kernels on the tested NPU stack. NPUW_FUNCALL_FOR_ALL and NPUW_UNFOLD_IREQS expose NPUW graph lowering trade-offs around dispatch overhead, memory use, and long-context stability. COMPILATION_NUM_THREADS gives a way to cap compiler worker parallelism to reduce peak host memory during large graph compilation.
Trade-off: a free-form KEY=VALUE string is less type-safe than one env var per option and invalid plugin properties will only be diagnosed by the OpenVINO stack. Applying it last is intentional: it gives one escape hatch for bisecting driver/compiler behavior and for testing new plugin options without rebuilding llama.cpp, while the named aliases continue to document the most important performance knobs.
- openvino: add scatter-elements KV update option
Add GGML_OPENVINO_KV_SCATTER_ELEMENTS as an opt-in lowering for non-stateful single-row KV cache writes. When the flag is enabled and the set_rows indices are the simple decode case, translate_set_rows uses ScatterElementsUpdate with broadcasted rank-4 indices instead of ScatterUpdate.
The motivation is NPU decode performance. The default ScatterUpdate path updates one KV row but the NPU plugin does not perform that update in place; it effectively copies the full destination tensor to write a small slice. As context length grows, that full-tensor movement dominates the per-token KV update cost. ScatterElementsUpdate expresses the same single-row write at element granularity and is measurably cheaper for this shape.
Keep the optimization guarded. Stateful execution keeps its existing concat/update path, and multidimensional indices already use the broader ScatterElementsUpdate lowering. The new flag only changes the common non-stateful decode write, so the default behavior and graph shape remain unchanged unless the user opts in.
Trade-off: ScatterElementsUpdate introduces extra ShapeOf, Reshape, and Broadcast nodes to build indices matching the update tensor. That overhead is worthwhile only when it avoids the NPU plugin's full-destination ScatterUpdate copy, so the path is controlled by an environment variable rather than becoming the unconditional lowering.
- openvino: gather token embeddings from compressed rows
Add NPU token-embedding controls for models whose token_embd.weight would otherwise be widened from Q6_K to FP16. GGML_OPENVINO_TOKEN_EMBD_I8 keeps the embedding table on the normal int8 compressed-weight path, while GGML_OPENVINO_TOKEN_EMBD_I4 requantizes it to the group-128 int4 layout used by the NPU weight path.
The performance target is tied embeddings and large vocabulary tables. When token_embd.weight is also consumed by the lm_head matmul, a normal Gather from the dequantized table gives the dequantization subgraph a second consumer and can force the full table to be materialized. The new gather_compressed_rows helper recognizes the dequantization chain emitted for int4/int8 constants, gathers only the selected packed rows from the leaf constants, and rebuilds the convert/subtract/multiply/reshape chain on those rows.
This reduces memory pressure and avoids dequantizing or materializing a full embedding table when inference only needs the token rows for the current input. The helper is deliberately conservative: it only rewrites static 2D tables above a size threshold, walks only the known compressed-weight node patterns, limits recursion depth, and falls back to the original Gather when the subgraph is not recognized.
Trade-off: int8 and especially int4 token embeddings can change accuracy and may not be faster for small untied embedding tables, because the rewrite adds extra Gather/Reshape nodes and only pays off when it avoids a large full-table dequantization. The default behavior is unchanged; the NPU-specific I8/I4 paths are opt-in through environment variables.
- openvino: stream u4 requantization chunks
Extend the GGML_OPENVINO_REDUCE_COMPILE_MEM streaming requantization path to Q4_0/u4 targets. The previous streaming path intentionally skipped u4 because Q4_0 packs two weights per byte and the zero-point tensor stores two 4-bit values per byte, so the original quantize_q4_0 helper wrote as if every call started at block zero.
Teach quantize_q4_0 to accept a destination block offset and use the absolute block index for weights, scales, and zero-point writes. Streaming requantization emits whole-row chunks in increasing order, and the target block size is required to divide the row width, so no target block straddles a chunk boundary. That keeps packed weight bytes aligned and preserves the even-block assignment before the odd-block zero-point OR for each shared zero-point byte.
With that indexing in place, reduce-compile-memory mode can dequantize a chunk of rows into the scratch buffer and immediately quantize it into the final u4 buffers instead of materializing the entire tensor as temporary F32. This extends the peak-memory reduction already used by Q8/F16 targets to the NPU group-128/channel u4 layouts used for compressed weights.
Trade-off: the u4 path is more sensitive to block alignment than Q8/F16 because of nibble packing, so streaming remains guarded by the same row-divisibility check and only runs when GGML_OPENVINO_REDUCE_COMPILE_MEM is enabled. When the flag is off, the full-materialization path and quantized output layout remain unchanged.
- openvino: fail fast on static recompile after weight release
Extend the GGML_OPENVINO_RELEASE_WEIGHTS safety check to the static graph cache-miss path. Once host weight buffers have been released, any later compile would read invalid/zeroed host pages and bake the wrong constants into a new OpenVINO model.
The dynamic path already rejected this situation. Static execution can still miss the decoder cache when graph shapes or model parameters change, so it needs the same guard before erasing cached infer requests and rebuilding prefill/decode models.
This is a correctness and diagnosability change rather than a throughput optimization. It preserves the memory-saving release-weights mode for stable graph shapes, but turns unsupported recompilation into an explicit abort instead of silently producing a corrupted compiled model.
- openvino: add experimental NPU Q4_0 keep option
Add GGML_OPENVINO_NPU_KEEP_Q4_0 as an opt-in escape hatch for NPU weight handling. When enabled for native Q4_0 tensors, the NPU requant path returns nullopt and keeps the original Q4_0 extraction instead of regrouping the tensor through the selected NPU requant policy.
The motivation is to avoid unnecessary work for weights that are already stored as 4-bit values. The default NPU policy maps quantized weights to layouts such as Q4_0_128, which can reduce scale metadata but requires round-tripping Q4_0 through the requantization path. Keeping Q4_0 native can save that conversion work and preserve the original group-32 packing.
Keep the option experimental and disabled by default. Current NPU driver/plugin stacks can compile the resulting group-32 graph, but inference may hang or fail with ZE_RESULT_ERROR_DEVICE_LOST. The default behavior therefore continues to use the safer NPU requant policy; this flag is mainly useful for debugging driver behavior, measuring the true cost of regrouping, and testing future NPU stacks.
Trade-off: native Q4_0 avoids regrouping overhead and keeps the original quantization blocks, but it gives up the more regular group-128 layout expected by the default NPU compressed-weight path and may be unstable on current hardware/software combinations.
- openvino: unfold NPUW only for decode graph
Compile the static prefill and decode models with separate OpenVINO config maps so the decode graph can opt into NPUW_UNFOLD_IREQS without forcing the same lowering on prefill. The build_static_model helper now receives the config map explicitly, and the decode path gets a copy of the base config with NPUW_UNFOLD_IREQS=YES when the user has not already set it.
The performance target is NPU token generation. Decode executes a long stream of single-token infers, where folded NPUW function-call dispatch overhead becomes visible in every token. Unfolding the NPUW calls into separate infer requests removes that repeated dispatch cost and measured about +20% token-generation throughput on the tested stack, matching the behavior seen in OpenVINO GenAI.
Prefill keeps the original folded config. Prompt processing is a batched workload dominated by larger matmuls, and the folded form is faster there while also using less memory. Splitting the config lets prefill and decode use the lowering that matches their workload shape instead of treating NPUW_UNFOLD_IREQS as a single global compromise.
Trade-off: unfolding can increase memory and request-management overhead, so the automatic default is limited to the NPU decode model. If a user explicitly provides NPUW_UNFOLD_IREQS through the named env alias or GGML_OPENVINO_NPU_CONFIG, that value is respected because the decode override is only applied when the property is absent.
- openvino: tidy NPU tuning helper ordering
Collect the remaining small cleanups from the NPU tuning series. This reorders the NPU KV-slice documentation and environment-variable cache entry next to the other NPU opt-in toggles, adds a short header comment for the centralized NPU helper declarations, and makes the static prefill attention-size assignment in the decoder a little more explicit.
There is no intended behavior change in this commit. The purpose is to make the final source layout match the split commits that introduced the NPU tuning knobs, so future readers can find the related flags and helpers in one place without mixing that mechanical cleanup into the functional performance commits.
- openvino: normalize NPU compile env overrides
Move the default NPU_COMPILER_TYPE=DRIVER setting into the NPU compile_config initializer so the default NPU plugin options are declared in one place.
Route GGML_OPENVINO_NPU_COMPILE_CONFIG and GGML_OPENVINO_NPU_COMPILER_TYPE through the same set_compile_option_from_env helper used by the other named NPU compile option aliases. This keeps the legacy NPU_COMPILE_CONFIG mapping to NPU_COMPILATION_MODE_PARAMS intact while making all optional env-to-compile_config overrides follow the same non-empty-value rule.
GGML_OPENVINO_NPU_CONFIG is still parsed last, so explicit comma-separated KEY=VALUE overrides continue to have the highest precedence over both defaults and named aliases.
- openvino: stream NPU weight requant by default to cut compile peak
Default the streaming requantization path on for NPU so the compile-time F32 dequant transient (~1-2 GB for token_embd) is capped without an env flag. Output is byte-identical to full materialization; GGML_OPENVINO_REDUCE_COMPILE_MEM / MEMORY_OPTIMIZE still override (set to 0 to force off). Kept the g_nonov_weight_cache opt-in to avoid holding token_embd resident (which regressed decode steady memory). pp1024_d0: PrefillPeakPrivWS 5974->4664 MiB (-22%), decode steady and pp/tg throughput unchanged.
- openvino: add self-contained NPU model cache
Add an opt-in embedded-weight cache mode for the static NPU backend. GGML_OPENVINO_SELF_CONTAINED_BLOB=1 exports separate prefill and decode compiled models with ENABLE_WEIGHTLESS=false, uses isolated NPUW weight banks, validates raw GGUF weight manifests, binds imported ports by deterministic frontend order, and serializes the large NPUW compile/export/import operations.
Preserve raw model bytes during cold cache construction by requantizing into separately owned OV tensors instead of rewriting the backend model buffer. This keeps later graph fingerprints and manifests stable when llama-bench creates independent prompt and generation contexts.
Add strict GGML_OPENVINO_SELF_CONTAINED_MMAP=1 warm-import mode. The backend exposes a non-owning buffer_from_host_ptr wrapper over the GGUF mmap, uses raw tensor allocation sizes, avoids allocating/copying the private OpenVINO model-weight buffer, and stages the few mmap-backed runtime constants into small importable NPU host tensors. Invalid or missing cache entries fail clearly instead of trying to requantize into read-only mapped storage.
The normal path and existing weightless MODEL_PTR cache remain unchanged when the new flags are unset. Verified with phi-4-mini: chunk512 pp3072/tg512 cold 1123.10/27.60 t/s and warm mmap 1124.72/27.16 t/s; warm mmap removes roughly 3.9 GiB peak commit in the measured prefill workload. Self-contained blobs consume about 8.34 GiB for the separate pp and tg contexts.
- openvino: add lazy import for self-contained mmap cache
Add GGML_OPENVINO_SELF_CONTAINED_LAZY_IMPORT to make the low-working-set strict mmap path opt in instead of changing the default warm mmap behavior.
When the flag is set with SELF_CONTAINED_BLOB=1 and SELF_CONTAINED_MMAP=1, the static NPU path imports only the current phase blob: prefill imports the prefill blob, decode imports the decode blob later. This avoids the pair-shaped shared compiled-model cache for the lazy path and caches only the current phase infer request.
The regular warm mmap path remains eager and continues to prepare both prefill and decode compiled models. Non-mmap self-contained builds remain serial, and the regular non-self-contained path keeps the existing parallel build behavior.
Also register the new environment variable, document the memory/latency tradeoff, and add profiling-only import timing around model-cache imports.
Validated with ReleaseOV build: cmake --build build/ReleaseOV --target llama-bench llama-simple --config Release. Lazy warm mmap run measured PrefillPeakWS around 5759 MiB versus the eager warm mmap path around 7897 MiB.
- openvino: release GGUF mmap resident pages before blob import
Add GGML_OPENVINO_SELF_CONTAINED_RELEASE_MMAP_PAGES as an explicit opt-in for self-contained compiled-model cache hits.
Before importing a valid self-contained blob, recursively walk the graph to find unique external GGUF mmap-backed weight buffers and drop their resident pages while preserving the mapping address and tensor pointers.
On Linux use page-aligned madvise(MADV_DONTNEED). On Windows, where there is no reliable per-range mapped-file equivalent, call VirtualUnlock for the identified range and trim the current process working set with SetProcessWorkingSetSize(-1, -1). Pages required later can fault back on demand.
Keep the behavior disabled by default and document the physical-memory versus commit-memory tradeoff. Profiling reports the submitted GGUF range and model import duration.
Validated with ReleaseOV llama-bench/llama-simple build and pp3072+tg512 warm mmap lazy import. With page release enabled, PrefillPeakWS dropped from about 5758 MiB to 3053 MiB while throughput remained comparable; PeakCommit stayed about 3331 MiB.
- openvino: add GGML_OPENVINO_NPU_SHARED_BANK to share prefill/decode weights bank
When set, prefill and decode self-contained blobs use one shared NPUW_WEIGHTS_BANK name (ggml-sc-shared--) instead of separate per-phase banks. Since both decoders are built from the same model_weights map, NPUW dedups them into a single in-process bank; on warm import the identical (uid,tensor) pairs are content-verified and reused (patched NPUW weights_bank). Cuts resident compiled-weight footprint ~2GB (pp3072 warm: In-Use 7954->5871 MiB, WS 7891->5765) with prefill throughput unchanged.
- openvino: drop SELF_CONTAINED_LAZY_IMPORT and RELEASE_MMAP_PAGES
The shared weights bank (GGML_OPENVINO_NPU_SHARED_BANK) supersedes both: it deduplicates the prefill/decode compiled-weight copy directly, so lazy per-phase import and GGUF-page release no longer reduce steady memory. Remove the two env vars, the self-contained lazy-import branches (defaulting to the eager path that imports both blobs), the release_external_weight_pages helper, and the now-unused ggml_openvino_buffer_is_external / _release_external_pages buffer APIs.
- openvino: demand-page large NPU compute buffer in strict warm-mmap mode
In strict self-contained warm-mmap mode weights come via buffer_from_host_ptr,
so every alloc_buffer is graph scratch/KV. The large prefill compute buffer is
never touched on the host by the on-device fused OV model, yet the NPU L0 host
tensor (create_host_tensor + memset) makes all ~2.4 GiB resident. For buffers
= 1 GiB in this mode, reserve demand-zero pages (no L0, no memset) so they stay
non-resident; KV/small buffers keep the L0 host tensor for zero-copy import.
pp3072 warm mmap + shared bank: prefill peak WS 5765 -> 3314 MiB (-43%),
In-Use delta ~5940 -> ~3232 MiB; prefill 1256.7 -> 1252.9 t/s and decode
28.27 -> 27.96 t/s (both flat). Commit charge unchanged (demand-paged).
-
perf improvement success
-
openvino: size static NPU sliding-window masks to the window cache
The static graph built the sliding-window KQ mask at the full context length,
while the windowed layers bind their (smaller) window KV cache. With gemma-4 at
-c 4096 -ub 1024 the window cache is 1536 cells, so SDPA shape inference failed
([1,1,1024,4096] mask vs [1,8,1536,256] K/V) and every prompt errored out.
With GGML_OPENVINO_NPU_KV_SLICE the prefill mask was meant to use
attention_size_swa, but the check matched "swa" in the ggml tensor name, and
llama.cpp names both masks "attn_inp_kq_mask"; the windowed layers got the
full-attention sliced length instead ([1,1,1024,256] vs [1,8,1024,256]).
Identify the sliding-window mask by tensor identity (is_swa_mask) and size it to
ctx_per_seq_swa, both in the graph input shape and in the runtime mask padding.
- ggml-openvino: create the model cache dir under a Windows drive root
make_dirs() mkdir'ed every path prefix, including the drive root "D:", which
fails with EACCES (not EEXIST) on Windows. Any absolute cache path therefore
logged "could not create model cache dir ... caching disabled" and silently
ran without GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. Skip the drive root.
- ggml-openvino: split device-specific passes and NPU config out of shared code
Mechanical refactor, no behavior change (step 1 of the NPU/MoE architecture plan):
- openvino/pipelines/: TranslateSession::apply_transformations() now registers
common passes, then the device's passes, then the execution-mode passes
(stateful KV, static shapes), selected by a per-graph GraphTraits. The GPU MoE
fusions and the GPU eltwise-rank workaround move to pipelines/gpu.cpp; CPU and
NPU pipelines are empty placeholders for device lowerings. Pass order is
unchanged. - npu/npu_config.*: the NPU (NPUW) compile config built at backend init, the
NPUW strip for cacheless graphs and the per-graph static prefill/decode
variants (HFA prefill, decode NPUW_UNFOLD_IREQS) move out of
ggml-openvino-extra.cpp and utils.cpp. - npu/npu_quant_policy.*: the NPU requantization rules (requant policy,
token_embd Q6_K, keep-Q4_0) move out of ggml_openvino_get_requant_type().
Verified identical output before/after on OV-CPU, GPU and NPU (gemma-4 E4B).
- ggml-openvino: one KV-slice switch for dynamic and static graphs
KV-cache slicing (bind the KV cache, and size the KQ masks, at the attended
length instead of the full context) existed twice: on by default for dynamic
graphs (GGML_OPENVINO_DISABLE_KV_SLICE to turn it off) and opt-in for static
prefill (GGML_OPENVINO_NPU_KV_SLICE). The static mode is not NPU-specific; it
applies to any static graph, including CPU/GPU with GGML_OPENVINO_FORCE_STATIC.
Describe it as one mechanism with two modes behind GGML_OPENVINO_KV_SLICE
(unset = each mode's default, 0 = off, 1 = on) and two predicates,
ggml_openvino_{dynamic,static}_kv_slice_enabled(), used by the KV binding,
the static graph shapes, the mask padding and the cache keys. The two old
variables remain as deprecated aliases.
- ggml-openvino: initialize all ModelExtraInputInfo fields for the HFA inputs
dev_backend_openvino added shape_source/shape_axis to ModelExtraInputInfo; the HFA past-KV and mask
initializers still listed four fields, which GCC rejects under -Werror=missing-field-initializers.
- ggml-openvino: make new NPU defaults opt-in and fix static model conversion
- Convert the static prefill and decode models one after the other, both
before either is compiled. The decoders share the weight nodes;
converting on two threads crashed intermittently inside libopenvino,
and converting decode after the prefill compile gave wrong output on
Bonsai-8B. - Compile prefill and decode in parallel only with
GGML_OPENVINO_NPU_PARALLEL_COMPILE=1, since it raises the host memory
peak. - Unfold NPUW function calls on the decode graph only with
GGML_OPENVINO_NPU_DECODE_UNFOLD=1. The unfold default gave wrong output
on gemma-4-E4B and slowed decode on several small models. - Stream NPU weight requantization only with
GGML_OPENVINO_REDUCE_COMPILE_MEM or GGML_OPENVINO_MEMORY_OPTIMIZE, as on
the other devices. - Fix a -Werror=missing-field-initializers build error in utils.cpp.
- Declare ggml_openvino_get_npu_requant_type() only in
npu/npu_quant_policy.h. - Docs: document the new switches, deprecate stateful execution, show
stateless-only results in the validated models table, and update the
validation system.
- ggml-openvino: honour the -1 skip id in MUL_MAT_ID
Upstream made an id of -1 in MUL_MAT_ID skip the slot and zero its row of
the result. The translator passed -1 straight to GatherMatmul or Gather,
so 169 of the 743 MUL_MAT_ID op tests on CPU (all skip=1) failed.
A skipped slot now looks up a valid expert, and the result is multiplied
by a 0/1 mask of the kept slots. For the packed MXFP4 path (Gather and
MatMul), clamping the id to expert 0 is enough. GatherMatmul computes a
wrong row when a token names the same expert twice, so the general path
gives each skipped slot an expert that no other slot of the token uses.
TopK indices are never -1, and model graphs take the ids from the
router's TopK. Those graphs skip the handling and stay unchanged, so the
GPU MoE fusion still matches them.
- ggml-openvino: Gather rows before dequantizing a quantized embedding table
- Optimize gather to fetch row before dequant
- add NPU token_embd quantization option
- ggml-openvino: select the NPU token_embd type with GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT
GET_ROWS had two rewrites that gather the rows of a quantized table before
dequantizing them: gather_compressed_rows, which runs on every device, and
gather_before_dequant, enabled on NPU by GGML_OPENVINO_GATHER_BEFORE_DEQUANT.
Both match the int4 and int8 chains built by make_int4_weights() and
make_int8_weights(). Keep gather_compressed_rows and drop the second rewrite
and its switch.
The NPU type of a Q6_K token_embd.weight was set by two booleans,
GGML_OPENVINO_TOKEN_EMBD_I8 and GGML_OPENVINO_TOKEN_EMBD_I4, and by
GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT. Keep only the string variable, with the
values f16 (default), q8_0_c, q4_0_128 and q4_0_c, so the int8 and group-128
int4 settings stay available. An unknown value aborts, as for
GGML_OPENVINO_NPU_REQUANT_POLICY, instead of silently using f16. The variable
replaces the two booleans in the NPU compiled-model cache key.
-
align PAD pads to input rank in stateful mode
-
Gate the changes under is_stateful check to address review comment
-
ggml-openvino: lift Q tensors for non-RoPE layers to rank 4 in stateful mode
-
ggml-openvino: unify KV slicing under GGML_OPENVINO_KV_SLICE, on by default
Replace ggml_openvino_dynamic_kv_slice_enabled() and ggml_openvino_static_kv_slice_enabled() with a single ggml_openvino_kv_slice_enabled(). KV slicing binds the KV cache, and sizes the KQ masks, at the attended length instead of the full context.
GGML_OPENVINO_KV_SLICE now defaults to 1 on CPU, GPU and NPU; set it to 0 to disable. Dynamic graphs (CPU/GPU) bind a view over the cache without recompiling. Static graphs (NPU or GGML_OPENVINO_FORCE_STATIC) compile prefill at the aligned attended length and recompile when it changes; decode keeps its full-context graph.
Remove the deprecated aliases GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_NPU_KV_SLICE from the env var list, the model-cache settings, docs/backend/OPENVINO.md and the backend README.
Behavior change: static KV slicing is now on by default on NPU, so prefill compiles one graph per distinct n_kv (multiples of 256).
- ggml-openvino: keep stateful execution with -fa off
Keep the V cache token-major in stateful mode instead of falling back to
stateless execution. SET_ROWS appends to the state, and PERMUTE reads it
back in the layout attention needs, so MUL_MAT does not transpose it.
Dropping a reshape in grouped attention lets OpenVINO fuse it into SDPA.
For padded V caches, slice after the transpose so the cache is read in
place, without a copy that is recompiled for every KV length.
This removes the stateless fallback for -fa off.
-
ggml-openvino: Add the stateful check to make the gate safer
-
ggml-openvino: support gemma-2 soft-capped attention
Gemma-2 caps its attention scores as softcap * tanh(scores / softcap)
before the softmax. FLASH_ATTN_EXT now applies the cap on the manual
attention path, as SDPA has no soft-capping, and the decoder looks
through the SCALE, TANH, SCALE chain of -fa off when it matches
attention.
One helper finds the K cache of an attention for both the
sliding-window pre-pass and the main pass. The pre-pass copy did not
see through the chain, so the sliding-window layers used the
full-attention sizes once the context exceeded n_swa + n_ubatch.
Soft-capped attention keeps the flat KV cache layout: with the 4-D
layout the GPU plugin fails in its oneDNN path.
- ggml-openvino: enable NPU Level Zero host tensors by default
Back NPU buffers with Level Zero host memory unless GGML_OPENVINO_NPU_L0_HOST_TENSORS=0.
The plugin imports these buffers without a host-to-device copy each infer; on phi-4-mini
Q4_0 (pp3072+tg512) decode goes from 7.2 to 10.6-10.8 t/s. Allocation failures still fall
back to plain host memory.
- ggml-openvino: enable NPU fast prefill mask by default
GGML_OPENVINO_NPU_FAST_MASK now defaults to 1; set it to 0 to fall back to pad_input() + set_zero_diagonal().
The fast path builds each static prefill mask chunk directly in the OpenVINO input tensor (row memcpy of the valid prefix, -inf padding, zero diagonal), avoiding a temporary chunk_size x context_size vector and a full-tensor copy per chunk. The result is bit-identical to the fallback path.
- ggml-openvino: add opt-in exact zero points for asymmetric weights
Asymmetric quantized weights (Q4_K, Q4_1, Q5_1, and non-requantized
Q5_K) represent each group as w = d * q - m. The backend currently
converts these to u4/u8 weights with an f16 scale and an integer zero
point round(m / d), which can shift all weights in a group by up to
half a quantization step.
Add GGML_OPENVINO_EXACT_ZERO_POINT=1 to preserve the zero point in f16
instead, using the same path already used for MoE expert weights.
This is disabled by default because, with current OpenVINO releases, an
f16 zero point moves many GPU fully-connected layers off oneDNN,
resulting in slower prompt processing.
- ggml-openvino: rename npu/ to device/ and openvino/pipelines/ to openvino/pass_registry/
npu/ holds device-specific backend behaviour (NPU compile config and
requant policy), and pipelines/ only registers passes per device; the new
names describe what the folders contain. No functional change.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
- docs: document NPU self-contained model cache and shared weights bank
Add GGML_OPENVINO_SELF_CONTAINED_BLOB, GGML_OPENVINO_SELF_CONTAINED_MMAP and GGML_OPENVINO_NPU_SHARED_BANK to the OpenVINO environment variable table, and a note describing the export/import workflow and the requirement to keep NPU settings identical between runs.
- ggml-openvino: remove experimental GGML_OPENVINO_NPU_KEEP_Q4_0
Keeping native Q4_0 (group-32) on NPU compiles but hangs the device at inference (ZE_RESULT_ERROR_DEVICE_LOST), so the option was never usable. NPU Q4_0 weights now always go through GGML_OPENVINO_NPU_REQUANT_POLICY. Drop the env var, the requant bypass, its self-contained blob fingerprint term and the docs entry.
- ggml-openvino: Update PERMUTE and CONCAT dynamic dim axis selection, Keep fp32 precision for sum-of-squares
All changes of ravi9#358 as one commit. The PR branch had
dev_backend_openvino merged into it, so its commits could not be replayed one by one when
dev_backend_openvino was rebased onto upstream master.
Commits of the PR:
- Update fix to just maintain fp32 precision instead of modifying the graph
- Limit the fp32 precision spreading in the graph by fencing it with Convert. Prevents perf degrade.
- Update OPENVINO.md
- Update flash_attn_ext.cpp
- Update utils.cpp
-
enable flash_attn with sink input
-
ggml-openvino: fix FLASH_ATTN_EXT input detection and sink scale
translate_flash_attn_ext read get_input_names()[3] and [4] on every
FLASH_ATTN_EXT node, also on nodes with only 3 or 4 inputs. The
out-of-bounds read crashed test-thread-safety and
test-speculative-rejection-sampling. The inputs are q, k, v, then the
optional mask and sinks, and supports_op rejects sinks without a mask,
so the input count tells which inputs are present.
With sinks, the scale was replaced with 1.0. llama.cpp passes the
attention scale in op_params[0] and adds the sinks separately, so keep
op_params[0]. The "scale" input branch is removed: ggml's src[4] is
only ever the sinks.
Only SDPA applies the sinks. The manual GQA path and the NPU HFA
prefill path drop them, so attention with sinks now always takes the
SDPA path. SDPA has no logit soft-capping, so supports_op rejects sinks
together with soft-capping.
- ggml-openvino: accept flash attention on layers without RoPE
On layers without RoPE (SmolLM3: every 4th layer), Q reaches
FLASH_ATTN_EXT through PERMUTE -> VIEW -> RESHAPE. The support check
rejected this pattern, causing these layers to fall back to CPU and the
split graph to fail with -fa on/auto.
- ggml-openvino: keep only weight-divisor DIVs on CPU
The GPU DIV guard for per-channel scale vectors (MPT's act_scales) currently checks only the divisor's shape. During single-token Qwen2-MoE decode, the shared-expert gate silu(x) / x has a [1,1,1,1] divisor that matches this shape, causing its DIVs to be kept on the CPU.
As a result, every decode step incurred CPU execution and Qwen1.5-MoE decoded at only ~2 tok/s.
Apply the guard only when the divisor is a model weight, so runtime tensors such as the Qwen2-MoE gate divisor can remain on the GPU.
- ggml-openvino: Qwen MTP and multi-seq optimization with new OV ops
All changes of ravi9#370 as one commit. The PR branch had
dev_backend_openvino merged into it, so its commits could not be replayed one by one when
dev_backend_openvino was rebased onto upstream master.
Commits of the PR:
- openvino : support Qwen MTP decoding
- openvino: Optimize TopK
- openvino: remove single-slot recurrent cache shortcuts
- openvino: fuse generic recurrent caches and remove shape waits
- openvino: include dense Q8 weights in all INT4 requantization
- openvino: use two GDN outputs for snapshots
- docs: explain native OpenVINO recurrent cache contracts
- docs: clarify Qwen3.5 purpose of native OpenVINO cache ops
- llama: skip unallocated hybrid graph inputs
- quantize: preserve Qwen hybrid layer metadata when pruning
- openvino: emit fused recurrent computation and rotate device caches
- openvino: limit KV slicing to attention caches and share compiled blobs
- tools: add natural-text server benchmark and OpenVINO profile summary
- docs: record fused-cache branch split and UD validation
- server-bench: preserve native token counts and save request timings
- openvino: document MTP regression and host USM alias recovery
- openvino: record large-context allocation diagnosis and default-batch validation
- server-bench: count buffered native tokens and reject stream errors
- openvino: document UD split and unified server benchmarks
- openvino: record Vulkan long-context baseline and GPU-fit controls
- openvino: finalize UD benchmark report and MTP comparison
- openvino: use stable in-place recurrent caches and optional separate convolution
- openvino: accept matching empty axes in DIV repeat translation
- openvino: remove the obsolete cache rotation environment setting
- openvino: validate imported KV results in their four-dimensional layout
- docs: record the bounded in-place KV and convolution benchmark rerun
- docs: complete default-batch OV unified-KV 32K benchmarks
- openvino: remove the separate recurrent convolution path
- openvino: handle zero-logits batches and changing sampler sequences
- docs: compare default and large Qwen3.6 benchmark batches
- docs: use the historical Vulkan default-batch reference consistently
- server-bench: use an integer histogram bin count
- benchmarks: summarize current OV unified MMLU timings
- docs: record cache-only unified KV MMLU results
- openvino: preserve the TopK index port in GET_ROWS
- openvino: reuse the MTP sampling model across active slots
- openvino: expose selection flags and offsets in graph dumps
- docs: consolidate Qwen OpenVINO defaults and performance evidence
- docs: retain continuous-request MTP benchmark results
- docs: compare stable MTP with the non-MTP serving baseline
- openvino: isolate GDN snapshots and skip naive cache fusion
- ggml: document Qwen3.6 GenAI benchmarks
- ggml-openvino: keep PR changes inside the backend folder
- ggml-openvino: drop the extra semicolons after OPENVINO_OP
OPENVINO_OP expands to member function definitions that end in a closing
brace, so a semicolon after it is an empty declaration inside the class.
GCC 15.2 reports it with -Wextra-semi, which ggml enables for GCC 8.1 and
newer, so a build with LLAMA_FATAL_WARNINGS=ON (the default in ci/run.sh)
fails in gated_delta_net.hpp. The CI compilers do not report it.
recurrent_cache_update.hpp has the same pattern twice. It is compiled only
when the OpenVINO package provides the recurrent cache-update ops.
- ggml-openvino: make q8_0_c the default NPU token_embd requant type
Default of GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT changed from f16 to q8_0_c.
Applies only to NPU and only to a Q6_K token_embd.weight; other devices and
quantization types are unchanged.
- ggml-openvino: do not treat the recurrent state gather as out_ids
The output_len scan now runs for every model and takes the first GET_ROWS
whose index is a graph input. On Qwen3.5 that is the recurrent state gather
of build_rs, which indexes the inp->s_copy leaf directly, so output_len
became 1. In static (NPU) mode the prompt model then returned one logits
row per chunk, and llama-perplexity read stale rows (PPL about 240000 on
Qwen3.5-2B). The same predicate also gave the s_copy input the out_ids
shape and data in static mode.
is_output_idx now skips GET_ROWS that read a recurrent state cache. The old
predicate stays as is_get_rows_input_idx for the dynamic-dim bookkeeping,
so the s_copy leaf keeps its dynamic length there.
- ggml-openvino: prefer the size-preserving dim when a RESHAPE repeats a stride
The RESHAPE dynamic-dim rule matches the source's dynamic stride against
the result's strides. A size-1 dim repeats the next dim's stride: Gemma 4
E2B has one KV head, so [head_size, n_tokens] -> [head_size, 1, n_tokens]
has the same stride on dims 1 and 2. The first match picked the size-1 head
dim, so the K/V views lost their dynamic token dim and became static
reshapes with the first graph's token count. Every later graph with another
token count failed.
Prefer the stride match that keeps the dynamic size, and fall back to the
first stride match for reshapes that change it.
- ggml-openvino: correct the RESHAPE dynamic-dim comment for size-1 dims
The comment above the RESHAPE stride match in compute_node_dynamic_dims()
still said the match is unambiguous because result strides are strictly
monotone. A size-1 dim has the same stride as the dim after it
([head_dim, T] -> [head_dim, 1, T] with one KV head), so it is not.
Explain the size-1 case in the block comment, drop the duplicate comment
inside the loop, and name the source dim's size once.
- examples: add llama-chat-replay
Replays JSONL prompts like OpenVINO GenAI chat.py (-pf, -i chat history, greedy, -o, -pc metrics). The first prompt is prefilled plus one decode step during load (Warmup time) and the memory is cleared, so compile/import is counted in Load time as in GenAI.
- ggml-openvino: make the NPU lazy host threshold configurable, default 2 GiB
With a large context the KV cache (~1 GiB at ctx 8K) crossed the 1 GiB threshold, lost its Level Zero host tensor in strict warm-mmap mode and made decode copy the KV every token. Default to 2 GiB so only the larger prefill compute buffer stays demand-zero, and allow overriding it with GGML_OPENVINO_LAZY_HOST_THRESHOLD_MB.
- ggml-openvino: fix NPU HFA prefill with ordinal-suffixed KV input names
KV cache inputs are now named with a graph-ordinal suffix (cache_k_l0#l12), but flash_attn_ext looks the HFA past input up as ggml_hfa_past_cache_k_l, so GGML_OPENVINO_NPU_HFA_PREFILL failed with 'ggml_hfa_past_cache_k_l0 not found in tensor map'. Name the past inputs after the bare cache name and resolve the source cache by that name when binding.
- ggml-openvino: keep the NPU prefill model on cache hits after decode
With KV slicing, decode calls overwrite the shared decoder's attention size with the full context, so the next prefill compared that against its sliced size, missed the cache and rebuilt the decoder and both infer requests (plus a cold first chunk). Remember the attention sizes the prefill model was built with and compare against those.
- ggml-openvino: bind NPU HFA past KV zero-copy in the cache layout
The HFA prefill past K/V inputs were [1, heads, past, head_size] copies of the cache, relaid out per chunk on the host and allocated per decoder (256 MiB on phi-4-mini at 3k). Declare them in the ggml cache layout [1, past, heads, head_size], bind the first past rows of the cache directly and transpose in the graph; NPUW HFA reads the tiles from the BSHD past (needs the matching NPUW HFA support). HFA input ABI v4.
- ggml-openvino: size the NPU decode KV window by GGML_OPENVINO_NPU_DECODE_BUCKET
Static decode attended the whole context, so every token read and wrote the full KV cache. With a bucket base set, decode attends the smallest base*2^k rows covering the next position (clamped to ctx): the KV inputs and masks are sliced to the window, the compiled pair's decode graph is built at the window the prefill reaches, and other windows are built or imported from the self-contained cache on first use with the pair's settings and bank. With self-contained blobs, keep the token_embd weight node for the later window models so they reuse the pair's bank entry instead of embedding a second vocab copy. The static build/convert lambdas become functions shared with the window path.
- ggml-openvino: allocate the NPU KV cache on demand with GGML_OPENVINO_NPU_KV_ON_DEMAND
The NPU KV buffer was committed for the whole context up front. With the option set, the KV buffer is swapped for demand-zero memory at its first KV tensor and every KV tensor gets its own storage in a registry, grown in base*2^k rows (base = decode bucket or 1024) as the static graphs reach further. Host accessors, KV inputs and outputs, sliced KV views and the HFA past bind that storage; when it moves, idle requests bound to the old storage are recreated so it is released.
With decode buckets and KV slicing the storage also takes the layout GenAI uses on NPU: K is [heads][rows][head_size] and V is [heads][head_size][rows], each bound whole to prefill and decode. SET_ROWS scatters new K rows on axis 2 and V rows on axis 3, the attention permute only cuts the window, the HFA past is the storage itself, and attention multiplies the probabilities by V^T with transpose_b instead of SDPA. A chunk-sized scatter on the innermost axis is slow on NPU, so the HFA prefill returns each chunk's V^T rows and the host copies them into the cache. Host reads, writes and fills go through a layout-aware copy, storage growth re-packs each head, and the layout is part of the model cache fingerprint.
- ggml-openvino: NPU weightless blob cache with one shared weights file
GGML_OPENVINO_NPU_WEIGHTLESS_BANK=1 (with GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR) exports weightless prefill/decode NPUW blobs plus one -.weights.bin. Weight Constants are laid out once by data pointer and their WeightlessCacheAttribute is retagged with the real offsets. Warm import uses ov::weights_path, and NPUW dedups both models into one bank through the LazyTensor hashes, so the OpenVINO weights_bank duplicate-uid patches are not needed. Decode KV-window models resolve the shared weights through the pair's file, or get their own file when the pair was imported. Also drop the old blob and manifest before publishing a re-export: std::rename does not replace an existing file on Windows.
Based on anikulk/llama.cpp 4ab270a.
Co-authored-by: Anisha Kulkarni anisha.dattatraya.kulkarni@intel.com
- ggml-openvino: prebuild all NPU decode KV windows with GGML_OPENVINO_NPU_DECODE_PREBUILD
Decode windows were built lazily when the sequence first grew into them, which stalled generation for a compile without a cache and made a strict-mmap import fail for windows the export run never reached. GGML_OPENVINO_NPU_DECODE_PREBUILD replaces GGML_OPENVINO_NPU_DECODE_BUCKET: 0 (default) keeps decode over the whole context; 1 sizes decode by 1024*2^k KV windows and the call that builds or imports the prefill/decode pair also builds or imports the decode graphs of every other window up to the context, from the pair's graph and compute params. Requests are still created on first use. Window building is split out of the request lookup so both paths share it, and window ports are checked against the pair's decode ports.
- Revert "examples: add llama-chat-replay"
This reverts commit 4eded2c.
- ggml-openvino: initialize the new compiled-graph and build-args fields
ov_compiled_graph and static_build_args gained members, but three brace
initializations still listed only the old ones. GCC reports
-Wmissing-field-initializers there, and CI builds with warnings as errors.
Set the fields by name, and pass {} for weights_path, which the window
build sets afterwards.
- ggml-openvino: fix gemma-4 when only the output rows reach the KV-shared layers
With one sequence, gemma-4 now drops the rows that are not output before
its first KV-shared layer. Those layers take their positions through
GET_ROWS(inp_pos, out_ids), and the backend mishandled three things:
- The positions are reshaped from [n] to [1, n]. That matched RESHAPE
case 1, which marks the wrong dim dynamic. Use case 3, which keeps
the source dim as dim 1. - ROPE shared one sin/cos table per frequency setting, so the KV-shared
layers reused the table built from the full inp_pos. Add the position
input to the cache key. - compute_llm_params took input_len from the last attention op. That is
now a KV-shared layer with only the output rows, so the static (NPU)
path padded prefill to the wrong length. Take the largest query length.
Stateful execution still fails on the per-layer embedding view and is
not covered here.
- ggml-openvino: keep the inp_pos layout for gemma-4's gathered positions
A batch can have no output rows, for example a prompt chunk that
llama-cli processes before its last tokens. gemma-4 then reshapes an
empty position gather for the ROPEs of its KV-shared layers. The size
checks of the RESHAPE cases cannot place a dim of size 0, so the
positions got the wrong layout and the ROPE multiply failed on CPU and
GPU. A RESHAPE that feeds the ROPE positions now always takes the
inp_pos layout [1, 1, 1, n].
- ggml-openvino: enable stateless GPU execution for MLA models
The per-head MLA weights (attn_k_b / attn_v_b) are 3D quantized tensors. Their MUL_MAT operations were falling back to the CPU, splitting the graph at every layer and causing GPU execution to fail.
Accept these tensors in the GPU support check and address the following issues exposed by enabling GPU execution:
- Quantization: Use exact zero points only for MoE expert weights. U4 weights with F16 zero points fail in batched GPU MatMul.
- Dynamic VIEW dimensions: Select the correct dimension when a size-1 dimension shares its stride (e.g.
k_pe), and match KV cache views by stride. - KV cache V view: Translate the V view of the K cache as a reshape while preserving the dynamic cell axis.
- Flash Attention: Accept the MLA Q (
CONCAT) pattern and V views derived from another view.
Limitations: The GPU plugin fails on batched compressed weights with static shapes. Keep these unsupported in op tests and GGML_OPENVINO_FORCE_STATIC.
- openvino : use writable recurrent cache inputs
Assisted-by: Codex
- ggml-openvino: fix decode crash for Qwen3.5 4B
When the dynamic size is 1, the code now keeps the innermost match. When the size is greater than 1, it still uses the MLA rule, so the MLA k_pe fix still works for multi-token graphs.
- ggml-openvino: keep the GDN state in the OpenVINO layout
Adds an opt-in switch, GGML_OPENVINO_GDN_NATIVE_STATE. When it's on, the GATED_DELTA_NET recurrent state stays in the layout the OpenVINO GatedDeltaNet op uses, [B, H_v, key_dim, value_dim], instead of the ggml layout [B, H_v, value_dim, key_dim]. That removes the two state transposes the translator adds on every call.
- ggml-openvino: support Ling-mini-2.0 group-routed MoE
Ling-mini-2.0 selects experts by group, but the group-selection graph
used ops that were not supported by the backend. This left 370 nodes on
the CPU and reduced decode to ~1.5 tok/s.
-
Translate the batched 4D
GET_ROWSgather used for group scores. -
Support
SET_ROWSfor compute tensors, not just KV caches. -
Preserve the dynamic token dimension in
FILL. -
Skip the
DIVrepeat when dimensions already match. -
Use the tracked token dimension for
RESHAPE/GET_ROWSop cases,
so a graph compiled during one-row warmup also works for ubatches
with more output rows. -
ggml-openvino: disable output staging
The backend stages small outputs (up to 4 MB) in a USM host buffer
and binds larger outputs directly to the ggml buffer. When the number
of outputs changes between calls to the same graph, as in
llama-perplexity, an output port can switch from USM host memory to
plain host memory.
The OpenVINO GPU plugin mishandles this switch for some dynamic
outputs, causing out-of-bounds writes on subsequent calls. This
results in CL_OUT_OF_RESOURCES. For example,
llama-perplexity -c 2048 -ub 512 fails on the fourth ubatch with
Gemma-2, Gemma-4, Phi-3.5, Phi-4-mini, and Qwen3.5.
Disable output staging until the plugin issue is fixed, so each output
is always bound the same way. Input staging remains enabled.
- ggml-openvino: match attention masks by identity
Gemma-4 prunes non-output rows before its KV-shared layers. These
layers select the corresponding mask rows with GET_ROWS(mask, out_ids).
Mask parameters were named by their consumers, but GET_ROWS is not
recognized as a mask consumer. As a result, the sliding-window layers
selected their rows from the full-attention mask instead of the
sliding-window mask. This either caused the mask add to fail
broadcasting beyond the sliding-window cache (d1024 and above with
-ub 512) or allowed attention beyond the sliding window at shorter
contexts.
Track graph-input attention masks by identity and assign them the
same parameter name for every consumer. The SWA pre-pass now keeps
each layer's own attention mask, as the KV-shared layers read the
same cache with mask rows derived from it.
- ggml-openvino: preserve input mask names and window sizes
KV-shared layers appear at the end of the graph, and their derived
masks are not graph inputs. Processing these masks overwrote the
names and sliding-window sizes recorded from the original input
masks with empty values.
Preserve the names and window sizes of graph-input masks so the
attention dimensions continue to be derived from the original
mask shapes.
- ggml-openvino: accept empty tensors in stride checks
In Gemma-4 batches without output rows, the attention tensors of
KV-shared layers are empty. Their strides beyond the empty
dimension are zero, causing has_strides_on_element_grid() to reject
them.
This sent SOFT_MAX, the KQV MUL_MAT, and CONT to the CPU with
-fa off. It also caused the per-layer input crossing the graph
split to be bound with the wrong layout.
Accept empty tensors in the stride check. Their contents are never
read, so the remaining strides do not need to satisfy the element
grid constraints.
- ggml-openvino: classify transposed V caches using v_trans
With -fa off, the V cache is transposed. When the attended length
is the whole cache, as with a prompt that fits in one ubatch, its V
view is contiguous and was mistaken for the K cache layout.
The K layout path does not keep only the rows of the layer's own V
size. Gemma-4 allocates 512 V rows in every layer, the size of its
widest layer, and its sliding-window layers use the first 256. These
layers then attended with all 512 rows, and prompts fitting in a
single ubatch failed with a broadcast error.
Use v_trans to identify the V cache layout instead of relying on
contiguity.
- ggml-openvino: use per-layer Split in stateful execution
Stateful execution was excluded from the per-layer Spl…
Website:
Attestations:
macOS/iOS:
- macOS Apple Silicon (arm64)
- macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
- macOS Intel (x64)
- iOS XCFramework
Linux:
- Ubuntu x64 (CPU)
- Ubuntu arm64 (CPU)
- Ubuntu 24.04 s390x (CPU)
- Ubuntu 26.04 s390x (CPU)
- Ubuntu 26.04 s390x (zDNN)
- Ubuntu x64 (Vulkan)
- Ubuntu arm64 (Vulkan)
- Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries
- Ubuntu x64 (CUDA 13) - CUDA 13.5 libraries
- Ubuntu arm64 (CUDA 13) - CUDA 13.5 libraries
- Ubuntu x64 (ROCm 10.0)
- Ubuntu x64 (OpenVINO)
- Ubuntu x64 (SYCL FP32)
- Ubuntu x64 (SYCL FP16)
- Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide
Android:
Windows:
- Windows x64 (CPU)
- Windows arm64 (CPU)
- Windows arm64 (OpenCL Adreno)
- Windows x64 (CUDA 12) - CUDA 12.4 DLLs
- Windows x64 (CUDA 13) - CUDA 13.5 DLLs
- Windows arm64 (CUDA 13) - CUDA 13.5 DLLs
- Windows x64 (Vulkan)
- Windows arm64 (Vulkan)
- Windows x64 (OpenVINO)
- Windows x64 (SYCL)
- Windows x64 (ROCm 10.0)
openEuler:
- DISABLED
- openEuler x86 (310p)
- openEuler x86 (910b, ACL Graph)
- openEuler aarch64 (310p)
- openEuler aarch64 (910b, ACL Graph)
UI: