Skip to content

LexiEngine Inkling 2

LexiSuite edited this page Nov 22, 2026 · 2 revisions

LexiEngine: Inkling 2 on Vulkan (inkling2_loop)

State of version b753bf3, November 21, 2026. Speed and accuracy against llama.cpp: Benchmarks.

LexiEngine's runtime for Inkling-2-Mini (Apache-2.0, released 2026-11-20): the text backbone on one Vulkan device, served by lexi-engine --backend vulkan. Source:

Part Where
Device runtime and server wrapper crates/server/src/inkling_vulkan.rs (InklingDevice, InklingVulkanRuntime, MemoryPlan)
Kernels crates/vulkan-ops/src/shaders/inkling.comp, inkling_attn.comp, inkling_attn_cm.comp, inkling_expert.comp, inkling_expert_cm.comp; Rust side crates/vulkan-ops/src/inkling_ops.rs
Runner (checks and speed) crates/server/examples/inkling_vulkan.rs
CPU reference crates/model-inkling (tool inkling-reference)
Routing and report crates/server/src/main.rs (ahead of the generic factories), crates/server/src/rcc_served_architectures.rs (Vulkan lane only)
Tokenizer pre-tokenizer inkling2 = the GPT-4o splitting, crates/tokenizer/src/lx36_pretokenizers.rs
llama.cpp patches used for conversion and the baseline at the end of this page

1. The model

24 layers, width 2,304, 24 query heads and 8 key/value heads of 128 values, vocabulary 200,058 (tables padded to 201,024), 128 routed experts of width 1,152 with 8 used per token, context up to 1,048,576 in the file's metadata.

The loop. Layers 6 to 17 (the core) run three times with shared weights. A forward pass applies layers 0-5 once, the core three times (each pass starting again from the stream that entered the core; passes 2 and 3 take a gated message from the pass before through low-rank controllers of rank 64 on the query, key, value and expert inputs), then layers 18-23 once: 24 + 2 x 12 = 48 layer applications, each with its own cache slot. Slots 0-23 are the first pass; slot 24 + 12 (p - 2) + (l - 6) is core layer l in pass p. Residuals are mixed by blocks of 6 layers (attention residuals with learned queries over the block outputs so far).

Attention. Layers 5, 11, 17 and 23 see every position (global); the other 20 see a window of 1,024. Global slots: layers 5 and 23 once, 11 and 17 three times = 8 slots; windowed slots: 40. Three query heads share a key/value head. No rotary positions: a relative bias by distance (16 values per head, distances below 1,024), queries and relative rows clipped to a norm (qk_clip 22.63, relative_clip 8) and scaled on global layers by tau = 1 + 0.1 ln(max(1, (position + 1) / 8,192)); a per-head sink logit (one extra softmax term), and a sigmoid output gate. Keys and values pass a 4-tap causal convolution (static taps, plus taps computed from the input on global layers) before they are cached; the attention and expert outputs pass 4-tap convolutions too. A logit soft cap of 30.

Experts. Router with a bias, 8 of 128, route weights scaled by 8; gate and up fused, activation "situ" with betas 4 and 25.

2. The file it reads

The runtime reads llama.cpp's GGUF layout for inkling2_loop. The file used for every number here is Inkling-2-Mini-NVFP4exp-Q8.gguf (17.6 GB), converted with the patch in section 8:

  • 40 expert tensors in ggml NVFP4 (GGUF type 40: blocks of 64 values, 4-bit E2M1 levels, a UE4M3 scale per 16): the vendor's 4-bit values and block scales repacked unchanged (decoded rows checked bit-identical to the checkpoint), with the vendor's per-expert factor amax / denominator in a per-expert .scale tensor that the kernels apply.
  • 8 expert tensors the vendor stores in MXFP8, and the whole trunk (attention, controllers, router, embedding, head): Q8_0 (blocks of 32 with a binary16 step), re-quantized by the converter from the vendor's MXFP8 / BF16.
  • The draft head (mtp.safetensors) and the vision tower are not in the file.

The expert kernels read NVFP4 and Q8_0 rows; the dense products read Q8_0 through the engine's exact row kernels (runs of up to 8 tokens) and its matrix-core Q8_0 kernel (prompt passes of a multiple of 16).

3. How a token is computed

  • Prompt passes of up to 2,048 tokens (a multiple of 16; a tail of fewer than 16 runs in steps of up to 8). The experts of a pass are grouped by expert on the device and computed as 16x16x16 binary16 matrix products (NVFP4 levels times their block scale are exact in binary16; Q8_0 rows are rounded to binary16; the inputs and hidden rows are binary16). Attention of the global layers in a pass runs on matrix cores too: queries prepared once in binary16 (rotated as the keys are), keys walked in blocks of 64; since 2026-11-21 the windowed layers too (from the block holding the earliest query's window, older keys masked; LEXI_INKLING_WINDOW_CORES=0 keeps the scalar kernel with its online softmax over blocks of 256 keys). Since 2026-11-21 (evening) each subgroup of that kernel keeps its share of the weighted value sums (16 dimensions x 16 queries of each of the three heads) in matrix-core registers across the key blocks: a query's rescale by exp(old largest - new) multiplies the accumulator components of its column (on RDNA3 a lane holds one column of a 16x16 F32 accumulator), and the block's values are added on the matrix cores, so the sums no longer go through shared memory every block (shared memory 52 -> 40 KB; LEXI_INKLING_ATTN_REGACC=0 keeps the earlier kernel). Passes of a multiple of 32 tokens use inkling_attn_cm32.comp: the same with 32 queries a workgroup, so a key block is decoded once for 32 queries and the 24 score products of a block divide evenly over the 8 subgroups (6 accumulators a subgroup, 59 KB shared; LEXI_INKLING_ATTN_Q32=0: 16 queries).
  • Controllers (passes 2 and 3 of the core take a message through four low-rank controllers per layer): a set of controllers in two commands, the low-rank values of each (normalized input, message) pair, then each input row x - tanh(forget) x + tanh(gain) message with gain and forget stored transposed, in passes and single tokens alike (since 2026-11-21; LEXI_INKLING_FUSED_CONTROL=0: the earlier five commands per controller).
  • Single tokens (generation, and runs of up to 8): the experts per (token, route) with F32 inputs, each row read by a subgroup of 32 lanes taking consecutive 16-value units (since 2026-11-21; LEXI_INKLING_EXPERT_LANES=1 gives a row per lane, the earlier kernel), down rows per route with a combine step. The keys a token sees are split into up to 128 chunks (each a multiple of 256 positions), one workgroup per key/value head and chunk (taking the tokens of a short run one after another), and a second dispatch merges the chunks; so at depth the whole card works on one token's keys.
  • One recorded forward per token shape (since 2026-11-21): runs of up to 8 tokens read their position from a device word instead of the recorded commands, so a token's commands are the same at every position, and the queue replays the recorded command buffer of the previous token of that shape instead of recording it again (LEXI_INKLING_STEP_POSITION=0, LEXI_INKLING_REPLAY=0 switch them off; the logits are bitwise equal either way).
  • The head computes logits for the last token (or every token of a 16-token pass for scoring), applies the soft cap on the device; the server's device sampler picks the token there.

The draft head (speculative decoding, since 2026-11-21)

mtp.safetensors holds a 3-layer block drafter (the vendor calls it DSpark) with a recurrent ("Markov") head; its wiring follows the public SGLang implementation. With LEXI_INKLING_DRAFT naming the checkpoint folder (config.json with mtp_config, and mtp.safetensors), the runtime loads it next to the main model (Q8_0 matrices; 1.45 GB with F32 matrices at first, smaller since):

  • Every pass keeps four feature rows per position: the stream (the running sum of the current residual block) just before layer applications 5, 23 and 41 in pass order, and the final normed row. Each draft layer mixes them with its own softmax weights (fuse_weight), normalizes, and writes keys (per-head norm, rotary positions, theta 10^6) and values for those positions into the draft's own context cache (8-bit rows; layers 0 and 2 windowed to 1,024).
  • A speculative step at position p with anchor token a (the last token the main model produced): a block of a's normed embedding and 7 mask rows runs through the 3 layers (pre-norm attention over the context and the block, not causal inside the block; SiLU MLP of 9,216); the 7 draft rows go through the output table once; the recurrent head (state of 256: gated update from [state, embedding of the previous token, the draft row, the mixed features of position p - 1]) adds its output, folded into a 257-row table computed at load (the output table times its output projection and bias), and the largest logit is the proposal. All on the device; 8 token ids are read back.
  • Verification: the anchor and the 7 proposals run as one 8-token pass in which every token is computed exactly as alone (per-token row, expert and split-key attention kernels); the main model's greedy token after each position is chosen on the device; the matching prefix plus one token are kept. The short convolutions' rows are restored for a partial acceptance by replaying the kept tokens' saved input rows over the rows saved before the pass. A run whose attention chunk size would change inside it (every 32,768 positions) takes a plain step instead. So the output is always that of plain greedy decoding (checked by the runner's spec mode).
  • Verification length: 8 tokens (the anchor and all 7 proposals) below 131,072 positions, 4 from there on, because deep in the context every token of the pass walks all keys of the global layers (LEXI_INKLING_SPEC_RUN=N, 2 to 8, fixes it). Measured at 16,384 (8 better) and 261,632 (4 better) only. LEXI_INKLING_SPEC_POLICY=adaptive chooses the length per step by expected tokens per millisecond from measured step times and acceptance; it was slower than the lengths by depth in both measurements (76.1 against 78.7 tok/s at 16,384; 33.6 against 42.1 at 261,632, where a few early rejections made it settle on plain steps), so it is not the default.
  • Served through the greedy speculative step of the scheduler: requests with temperature 0 use it; sampled requests decode plainly.

4. Cache rows

4-bit rows by default and the only format the server uses: llama.cpp's q4_0 layout per head of 128 values, 4 blocks of 32 with a binary16 step and levels (code - 8) x step: 18 words = 72 bytes a head. Before rounding, keys are rotated with a Walsh-Hadamard transform over the head's 128 values and values per half of 64 (as llama.cpp does); queries are rotated alike and attention outputs rotated back. LEXI_INKLING_KV=f32 in the runner keeps rows as computed, for checking the arithmetic only (never a serving or benchmark setting). There is no 8-bit row format.

A windowed slot keeps window + 3,072 positions (one pass of 2,048 writes its rows before they are read); a global slot keeps every position. The short convolutions keep their last 3 input rows per slot.

5. Memory at 262,144 positions

Item Arithmetic Bytes
Weights every tensor of the file on the device 17.62 GB
Global caches 8 slots x keys and values x 262,144 positions x 8 heads x 72 bytes 2,415,919,104
Windowed caches 40 slots x 2 x 4,096 positions x 8 heads x 72 bytes 188,743,680
Caches together 2.60 GB
Prompt scratch planned at 480,000 bytes per pass token (2,048 tokens: 0.98 GB); measured: 0.58 GB of stream/projection rows plus the grouped-expert rows about 1 GB
Kept free pipelines, attention chunk parts, driver 0.54 GB
Draft head (with LEXI_INKLING_DRAFT) Q8_0 matrices, the 257-row fold table, 8-bit context rows for 262,144 positions, scratch 1.45 GB measured (the plan counts 1.38 GB)

MemoryPlan (in inkling_vulkan.rs) puts every weight and cache row on the first device and takes the largest pass of 2,048 / 1,024 / 512 / 256 tokens that fits beside them; a context whose weights and caches do not fit is refused with these figures (the runtime never moves cache rows to system memory). On the RX 7900 XTX (25.73 GB reported free) 262,144 positions load with passes of 2,048, and so do 524,288 positions (5.02 GB of caches; loaded and measured 2026-11-21, below); 1,048,576 need 9.85 GB (refused on a 24 GB card, below). The unit tests caches_at_262144_positions and the_plan_takes_the_largest_pass_that_fits_and_refuses_what_cannot hold this arithmetic, including 524,288 on the XTX's reported 25.73 GB.

Measured load (2026-11-21, lexi-engine --backend vulkan -c 262144, commit 199a3602): "weights 17.62 GB, caches 2.60 GB (262144 positions), scratch 0.58 GB (passes of 2048); loaded in 13.8 s".

524,288 positions (measured 2026-11-21)

Item Bytes
Weights 17.62 GB
Global caches: 8 slots x keys and values x 524,288 positions x 8 heads x 72 bytes 4,831,838,208
Windowed caches (as at 262,144) 188,743,680
Caches together 5.02 GB
Prompt scratch, passes of 2,048 (the plan takes 2,048 on the RX 7900 XTX) 0.58 GB measured
Draft head at this capacity (its global layer's 8-bit context rows double; passes of 1,024 then, scratch 0.30 GB) 2.00 GB measured

1,048,576 positions would need 9.85 GB of 4-bit caches beside 17.62 GB of weights, about 28.1 GB with scratch and margin: more than a 24 GB card holds, so the plan refuses it (it would take a larger card or weights kept off the card).

Measured load at 524,288 (runner, LEXI_INKLING_CONTEXT=524288 LEXI_INKLING_CHUNK=2048): "weights 17.62 GB, caches 5.02 GB (524288 positions), scratch 0.58 GB (passes of 2048); loaded in 13.1 s"; the card showed 23.73 of 25.75 GB in use during the prompt.

6. Serving

lexi-engine --backend vulkan -m Inkling-2-Mini-NVFP4exp-Q8.gguf -c 262144 --host 127.0.0.1 --port PORT
  • The chat completions endpoint /v1/chat/completions, streaming and not, with the file's tokenizer and chat template. The template prints "}}" inside an expression; the engine's template lexer ends a tag only outside string literals since 2026-11-21 (before, the template failed to parse).
  • One sequence at a time; prompts are handed to the runtime one pass at a time (a long prompt can be cancelled between passes). Refused on this runtime: other backends, plan-only, decisions, activation capture, embeddings, adapters, saved profiles, --parallel above 1.
  • Prompt reuse (since 2026-11-21, append-only): a prompt that starts with every token the caches hold keeps them and runs only its new tokens; otherwise a prompt that starts with the previous request's whole prompt (a conversation's next turn, whose rendering of the previous answer differs from the generated tokens) goes back to a checkpoint taken when that request generated its first token; anything else runs from position 0. The checkpoint holds what later positions overwrite: the short convolutions' last 3 input rows per slot (3.8 MB) and the windowed slots' rows of the 1,024 positions before it (47 MB with 4-bit rows), copied to system memory; the global slots keep every position anyway. The reused count is reported as usage.prompt_tokens_details.cached_tokens. Checked with the runner's resume mode (A 500 tokens, then 5,003 tokens that wrap the windowed rings, then A + a new tail and A + those tokens + the tail): both continuations bitwise equal to the same pieces run from a reset, with unrounded and with 4-bit caches; with every kernel in its per-token exact form also equal to a fresh run of the whole sequence (KL 0). With the default kernels a fresh run of the whole sequence differs in the last digits (last-row KL up to 2.5e-3, same top token) because other pass boundaries send some tokens to the binary16 prompt kernels. Two served chat turns at 262,144: turn 2 reported 33 cached prompt tokens (the whole first prompt).
  • The template does not read add_generation_prompt: the prompt ends after the last message and the model opens its own turn. It accepts reasoning_effort (template variable; default 0.9).
  • The model writes its answer in turn tokens: <|message_model|><|content_thinking|> thought <|end_message|> <|message_model|><|content_text|> answer <|end_message|>. The chat endpoint returns the thought as reasoning_content and the answer as content, streaming and not (since 2026-11-21; checked: "What is 17 * 23?" gives the arithmetic in reasoning_content and "391" in content, the same streamed and not).
  • The Panel lists the model when the engine's architecture report lists inkling2_loop (Vulkan lane). A Lexi Engine instance's settings carry INKLING_DRAFT (the draft head's checkpoint folder, passed as LEXI_INKLING_DRAFT; Vulkan only), LORA_ADAPTER / LORA_SCALE (one GGUF adapter at start) and LE_DECISION_TIMEOUT_MS (with LE_DECISIONS=1), checked when the settings are saved (since 2026-11-21).

Typed decisions (since 2026-11-21)

lexi-engine --decisions (and --decision-admin with operator credentials, --decision-artifact-bank DIR with a 0700 folder) serves /v1/decisions, /v1/decisions/batch, /v1/decisions/capabilities and the admin routes (/admin/decisions/calibrate/l1, /admin/decisions/fit/l2, artifacts) on this runtime, through the same Decider the CPU runtime uses.

  • The primitive (InklingVulkanRuntime::decision_capture): run the prompt and return the last token's full logits (after the soft cap) and its stream rows after chosen layer applications. Refused while a generation is active. A prompt that extends the held tokens runs only its tail; each decision prompt's end is checkpointed, so a label's further tokens and the next label continue from it instead of running the prompt again.
  • Shape: 48 blocks of 2,304 values, feature stage stream_after_application_before_output_mix. Block b is layer application b in execution order: 0-5 the layers before the core, 6-17, 18-29 and 30-41 the core's three passes, 42-47 the layers after it. The value is the stream (the running sum of the current residual block) of the last prompt token right after that application, before the output mix and the final norm. Artifacts bind to this numbering.
  • Label scores: the CPU runtime's arithmetic (log-softmax in f64 of each label token after the prompt extended by the label's earlier tokens).
  • Capabilities: next-token log-probabilities, label-span scoring, hidden-state capture, last-token pooling; no early stop (every capture runs the whole forward), no shared-prefix batching.
  • Identity: the fingerprint of the file's metadata with the SHA-256 of its bytes as the stable source identity (gguf-sha256:...; taken from a saved fingerprint when the file's size and time match, else hashed at start, about half a minute for 17.6 GB), and the tokenizer metadata's digest.
  • Limits of the API, not of the runtime: a decision state is at most 64 KiB and a rendered decision prompt at most 256 KiB (about 16,000 and 64,000 tokens). The limits stay: they bound what one request may make the server read, parse and hold. That is why a decision over long context (a question after a 261k-token conversation) needs another entry point: one that asks about a context the runtime already holds (the prompt-reuse checkpoint of the running conversation) instead of resending it as state text. That entry point does not exist yet; the runner's decide-depth mode checks the primitive at that depth.
  • Time limit: --decision-timeout-ms N (default 30,000) bounds every decision and admin operation. One that outlives it is cancelled: the runtime stops at its next model call (for a decision prompt of a few hundred tokens, within about half a second), the operation fails without changing the artifact bank, and only then does the request answer 408 ("... timed out after N ms and was cancelled; nothing was changed"), so the next request finds the server free. L1 calibration on this runtime takes minutes for a few dozen rows (measured below): set the timeout to fit, for example 1,800,000.
  • Artifact bank (--decision-artifact-bank DIR): owned through an exclusive lock on DIR/.active-bank.lock that the kernel holds while the server runs. A second server on the same folder is refused; when the owner stops, however it stops, the kernel releases the lock and the next start takes the folder with no file to clear.

LoRA adapters (since 2026-11-21)

lexi-engine --lora FILE (several allowed, each with its scale as the generic path parses it) applies GGUF LoRA adapters (llama.cpp's convention as lexi-engine-lora reads it: general.type = adapter, adapter.type = lora, adapter.lora.alpha, F32 pairs blk.N.attn_{q,k,v,output}.weight.lora_a [input, rank] and .lora_b [rank, output]) to the attention projections q, k, v and output of each physical layer; the core layers' terms apply on all three passes. Other targets and embedding adapters are refused; the rank must be a multiple of 4. The delta is scale x alpha / rank x B (A x), computed beside the Q8_0 product as two F32 row products and an add, in the prompt and the single-token paths alike; adapters on one projection add. inkling-lora-synth writes a synthetic adapter for checks (fixed seed; --zero for B = 0), and inkling-reference --gguf FILE --lora ADAPTER computes the same terms on the CPU.

7. Environment switches

Switch Where Effect
LEXI_INKLING_PASS=N server prompt pass of N tokens (a multiple of 16, at most 2,048) instead of the memory plan's choice
LEXI_INKLING_DRAFT=DIR server, runner spec load the draft head from the checkpoint folder DIR (speculative decoding for greedy requests)
LEXI_INKLING_EXPERT_LANES=1 runtime per-token experts with a row per lane (the kernel before 2026-11-21)
LEXI_INKLING_SPEC_RUN=N runtime tokens of a verification pass (2 to 8; default 8 below 131,072 positions, 4 beyond)
LEXI_INKLING_WINDOW_CORES=0 runtime prompt attention of windowed layers on the scalar kernel
LEXI_INKLING_ATTN_CHUNKS=0 runtime single tokens walk all their keys in one workgroup per head (the kernel before 2026-11-21)
LEXI_INKLING_ATTN_CORES=0 runtime prompt attention of global layers on the scalar kernel
LEXI_INKLING_EXPERT_CORES=0 runtime prompt experts per (token, route) instead of grouped matrix products (slow; for checks)
LEXI_INKLING_FAST_ROWS=0 runtime dense products of passes on the exact row kernels instead of the matrix-core Q8_0 kernel
LEXI_INKLING_PROFILE=1 runtime waits after every stage and records device time per stage (slower; not a speed figure)
LEXI_INKLING_FUSED_CONTROL=0 runtime the controllers as five commands each (pair, three F32 row products, apply) instead of two commands per set
LEXI_INKLING_ATTN_REGACC=0 runtime matrix-core prompt attention with the sums through shared memory every key block (the kernel before)
LEXI_INKLING_ATTN_Q32=0 runtime prompt attention with 16 queries a workgroup also for passes of a multiple of 32 tokens
LEXI_INKLING_W4A8=1 runtime runs of up to 8 tokens: 4-bit experts' gate and up products with 8-bit inputs (integer dot products); off by default
LEXI_INKLING_SPEC_POLICY=adaptive runtime verification length chosen per step by expected tokens per millisecond instead of by depth
LEXI_INKLING_STEP_POSITION=0, LEXI_INKLING_REPLAY=0 runtime positions in the recorded commands / every token's commands recorded again
LEXI_INKLING_INPUT8=experts|projections|all runtime accuracy ablation: those inputs rounded to 8 bits per block of 32 before the F32 kernels (no speed change)
LEXI_INKLING_CONTEXT=N runner cache positions (default the longest sequence plus 512)
LEXI_INKLING_CHUNK=N runner tokens per prompt pass (default 256; 2,048 for speed runs)
LEXI_INKLING_PROMPT_RUNS=N runner bench prompt repetitions (default 2; the last is the warm one)
LEXI_INKLING_KV=f32 runner unrounded cache rows, for checking the arithmetic only

GGML_VK_ALLOW_SYSMEM_FALLBACK=0 stays set for every run that loads a model.

8. Converting the vendor checkpoint

The checkpoint ships safetensors with 40 expert tensors NVFP4 (E2M1 values, an E4M3 scale per 16, a per-expert amax and denominator), 8 MXFP8, the rest BF16 / F32, and no model code. llama.cpp 29b52d398 (build 12600, the first with Inkling 2) converts it by decoding the experts to F32 and re-quantizing them: at 8 bits that file is 27.8 GB and does not fit a 24 GB card. The patch at the end of this page (llama.cpp-29b52d398-inkling2.diff) adds three things:

  1. conversion/inkling2.py: with INKLING2_NATIVE_NVFP4=1 the NVFP4 experts are repacked into ggml NVFP4 blocks unchanged (gate and up rows de-interleaved into the fused tensor) and the per-expert factor goes into a .scale tensor; the products' inputs stay at full precision (prec_a4 false) unless INKLING2_A4=1.
  2. src/llama-model.cpp: the loop's extra cache slots share the physical layer's expert scale tensors. Without it llama.cpp runs passes 2 and 3 of the core with unscaled experts and produces garbage with this file (upstream does not hit it because its own conversion writes no scale tensors).
  3. src/llama-graph.cpp: LLAMA_NVFP4_ACC_F32 keeps F32 accumulation for NVFP4 products (diagnostic).

Conversion (the trunk type follows --outtype; the run that wrote the measured file used q8_0, its exact command line was not recorded):

git -C llama.cpp checkout 29b52d398 && git -C llama.cpp apply llama.cpp-29b52d398-inkling2.diff
INKLING2_NATIVE_NVFP4=1 python3 llama.cpp/convert_hf_to_gguf.py CHECKPOINT_DIR --outtype q8_0 --outfile Inkling-2-Mini-NVFP4exp-Q8.gguf

The result: 744 tensors, 17.6 GB, architecture inkling2_loop. The same file runs in LexiEngine and in the patched llama.cpp.

9. Reference and checks

The CPU reference (crates/model-inkling) is a port of llama.cpp 29b52d398's inkling2 / inkling2-loop graph. It reads the vendor checkpoint directly (experts decoded exactly) or, with --gguf, the converted file's weights:

cargo build --offline --locked --release -p lexi-engine-model-inkling -p lexi-engine-model-maxwell -p lexi-engine-server --features vulkan --example inkling_vulkan
inkling-reference CHECKPOINT_DIR IDS OUT [--gguf FILE] [--cache-4bit] [--inputs exact|fp4] [--last N] [--score BASELINE]
maxwell-compare BASELINE OTHER IDS        # KL, same top token and perplexity of two logits files for the same ids

Token files hold one sequence per line, comma-separated ids. Checks after a change of the runtime or its kernels:

Check Command Must give
Arithmetic LEXI_INKLING_KV=f32 inkling_vulkan logits GGUF IDS OUT, then maxwell-compare REF OUT IDS against inkling-reference ... --gguf GGUF mean KL 0.00000
Same prompt twice inkling_vulkan prefill GGUF IDS OUT "same prompt twice identical: true"
Second sequence after a reset inkling_vulkan logits GGUF IDS OUT with two lines in IDS, compared row by row with each line run alone equal
4-bit accuracy inkling_vulkan logits and passes on 2 x 512 tokens against the vendor-checkpoint reference, second halves (510 rows) about llama.cpp's figure (0.0037)
Old and new kernels the same command with LEXI_INKLING_ATTN_CHUNKS=0 LEXI_INKLING_ATTN_CORES=0 same generated tokens
GPU kernel tests cargo test -p lexi-engine-vulkan-ops --features vulkan --test maxwell_attn --test maxwell_hc --test qwen4exp_exact_rows pass (shared row kernels)

The reference's --inputs fp4 mode (4-bit inputs to the 4-bit experts, as the vendor requires) is a reading of the checkpoint's quantization config; there is no vendor code to check it against, and it is not verified.

10. Measured (RX 7900 XTX 24 GB, RADV, i7-9700; 4-bit caches unless stated)

Accuracy against the exact CPU reference (vendor checkpoint, F32 weights as stored, F32 caches), text = a 2 x 512-token document, the 510 rows llama-perplexity scores:

Run Commit, date Mean KL Median Same top token
LexiEngine single tokens 2c0f15fe kernels, 2026-11-21 0.00385 0.00011 98.24%
LexiEngine prompt kernels (16-token passes) 2c0f15fe kernels, 2026-11-21 0.00338 0.00009 99.22%
LexiEngine single tokens 2ac529bd, 2026-11-20 0.00425 0.00009 98.24%
LexiEngine prompt kernels 2ac529bd, 2026-11-20 0.00343 0.00010 98.82%
llama.cpp 29b52d398 + patch, -ctk q4_0 -ctv q4_0 2026-11-20 0.00371 0.00007 98.82%

Arithmetic (unrounded caches, against the reference computed from the same file): 48 tokens KL 0.00000 (top logits equal to 7 digits); 1,024 positions one at a time with the split-key attention KL 0.00000, worst 0.000 (the earlier single walk: 0.00002, worst 0.002), 2026-11-21.

Speed: see the numbers recorded for each commit in the table below. Single runs; "prompt" is the warm second prompt in the process unless stated.

Commit, date Prompt 4,096 Prompt 16,384 Generation at 4,096 Generation at 16,384 Prompt 261,632 (first in process) Generation at 261,632
2ac529bd, 2026-11-20 1,531 tok/s 1,324 35.1 31.8 385.6 (678.5 s) 11.0
e2184ab5, 2026-11-21, idle box 1,676.4 1,568.9 39.92 38.96 791.0 (330.8 s) 33.26
394bb21b, 2026-11-21, idle box 2,015.6 1,877.5 43.83 42.78 851.8 (307.2 s) 35.40
a4c86246 (recorded forward), 2026-11-21, idle box 2,004.1* 1,884.2* 50.36 49.12 868.3 (301.3 s) 39.31
this commit (controllers, prompt attention in registers with 32 queries), idle box 2,244.5* 2,147.8* 52.37 50.93 1,145.7 (228.4 s) 40.24
the same with LEXI_INKLING_W4A8=1 2,242.5* 2,146.3* 55.44 53.88 see below see below

* one cold prompt per process (LEXI_INKLING_PROMPT_RUNS=1), 128 generated tokens. At 261,632: load 17.62 + 2.60 + 0.58 GB in 12.9 s. The prompt at 261,632 by attention kernel (switches; the first figure on the round's previous build, which also held the fused mix and norm, neutral for prompts): sums through shared memory 297.9 s (878.2 tok/s), sums in registers with 16 queries 274.7 s (952.5), with 32 queries 228.4 s (1,145.7).

Speculative decoding with the draft head (LEXI_INKLING_DRAFT=src inkling_vulkan spec GGUF IDS COUNT, greedy, 2026-11-21; the speculative tokens are identical to plain greedy decoding in every run). The commit after 394bb21b (the draft attention's key tiles padded against shared-memory bank conflicts) changes only the draft head:

Prompt Commit Plain Speculative Tokens per verification pass Per step: proposing / verifying
16,384 tokens, 256 generated 394bb21b 42.65 tok/s 78.64 tok/s 6.29 (runs of 8) 8.9 / 69.4 ms
16,384 tokens, 256 generated after 394bb21b 42.52 tok/s 79.06 tok/s 6.29 (runs of 8) 7.7 / 70.2 ms
261,632 tokens (262,144 positions), 128 generated 394bb21b 35.36 tok/s 37.40 tok/s 3.28 (runs of 4) 27.5 / 59.5 ms
261,632 tokens (262,144 positions), 128 generated after 394bb21b 35.68 tok/s 40.88 tok/s 3.28 (runs of 4) 19.4 / 60.1 ms
261,632 tokens (262,144 positions), 128 generated this commit 36.55 tok/s 41.84 tok/s 3.20 (runs of 4) 16.7 / 58.9 ms

At 261,632 with runs of 8 (4c9637ed and earlier builds): 30.2 tok/s, slower than plain; runs of 3: 33.2. Served at 262,144 (lexi-engine, the Fibonacci request at temperature 0, prompt included; 394bb21b): 42.3 tok/s without the draft head, 49.5 with it, the same answer.

At 524,288 positions (2026-11-21, code of the controllers / prompt-attention commit; prompt p523264.ids: the engine's source text of the 262,144 prompt followed by 34 Panel source files, 523,264 tokens; r524k10.ids: the same length with ten planted facts at 1%, 10%, ..., 80%, 99%):

Measurement Result
Load (runner, passes of 2,048) weights 17.62 GB, caches 5.02 GB, scratch 0.58 GB, 13.1 s
Prompt 523,264 tokens (bench) 682.6 s = 766.6 tok/s
Generation at depth 523,264, 64 tokens 33.71 tok/s (29.25 ms a token)
Retrieval, ten facts (ask) 10 of 10; that prompt 690.2 s = 758.1 tok/s
Draft head (spec, passes of 1,024, 2.00 GB) speculative 32.07 tok/s against 31.09 plain with the draft loaded (3.37 tokens a step; proposing 26.3 ms, verifying 78.0 ms); identical to plain greedy
Served (lexi-engine -c 524288, greedy chat, 1,143 tokens, prompt included) 43.9 tok/s without the draft head (passes of 2,048), 51.0 with it (the plan takes passes of 1,024); the same answer

At this depth the draft head does not beat plain decoding without it (32.1 against 33.7 tok/s): its own attention over the whole context costs about 26 ms a step. Commands: LEXI_INKLING_CONTEXT=524288 LEXI_INKLING_CHUNK=2048 LEXI_INKLING_PROMPT_RUNS=1 inkling_vulkan bench GGUF p523264.ids 64; LEXI_INKLING_CONTEXT=524288 inkling_vulkan ask GGUF r524k10.ids 8; LEXI_INKLING_DRAFT=src LEXI_INKLING_CONTEXT=524288 LEXI_INKLING_CHUNK=1024 inkling_vulkan spec GGUF p523264.ids 128.

Commands for the e2184ab5 row: LEXI_INKLING_CHUNK=2048 inkling_vulkan bench GGUF pN.ids 64 (two prompts, the second warm), and LEXI_INKLING_CONTEXT=262144 LEXI_INKLING_CHUNK=2048 LEXI_INKLING_PROMPT_RUNS=1 inkling_vulkan bench GGUF p261632.ids 64 (load: weights 17.62 GB, caches 2.60 GB, scratch 0.58 GB, 12.9 s).

Retrieval at depth (runner mode ask, e2184ab5 and again 394bb21b, 2026-11-21): five facts ("the access code for vault NAME is NNNNNN") planted at 2%, 25%, 50%, 75% and 97% of a 261,120-token prompt at 262,144 positions, then asked back one after another as plain text: 5 of 5 answered exactly (the same with an 8,192-token prompt). A second prompt with ten facts at 1%, 10%, 20%, ..., 80% and 99% of 260,864 tokens (the commit after 394bb21b): 10 of 10 answered exactly. Again with the prompt attention in registers and 32 queries a workgroup (this commit): 10 of 10, the 260,864-token prompt in 230.1 s (1,133.6 tok/s).

Typed decisions (2c0f15fe kernels plus the decision commit, 2026-11-21):

  • The primitive against the CPU reference, 48 tokens, every kernel exact (LEXI_INKLING_FAST_ROWS=0, LEXI_INKLING_EXPERT_CORES=0, LEXI_INKLING_ATTN_CORES=0, LEXI_INKLING_WINDOW_CORES=0), unrounded caches: stream rows after applications 0 / 5 / 23 / 41 / 47 differ by at most 3.8e-6 / 3.8e-5 / 1.5e-5 / 3.4e-5 / 1.5e-5 (largest values 9.9 / 176 / 131 / 50 / 36.5), logits by 1.3e-5 (largest 27.6), the same top token. With the default binary16 prompt kernels the rows differ by up to 2.0 (of 50) and the logits by 0.28, the same top token. With 4-bit caches on both sides (the reference rounds its rows without the rotation, so the two round differently): up to 6.2 in a row and 1.7 in the logits, the same top token.
  • Served at 262,144 positions (--decisions --decision-admin), raw level on the labelled holdout rows: yes/no on product facts 28 of 48, four options (largest number) 21 of 32, yes/no on Rust facts 6 of 16; probabilities sum to 1 within 2e-16; about 0.43 s a decision. Fitted on the separate fit rows through the admin API (operator key) and used through /v1/decisions at levels l1 and l2, holdout correct (raw / l1 / l2): product facts 28 / 24 / 24 of 48, four options 21 / 30 / 25 of 32, Rust facts 6 / 15 / 8 of 16. L1 calibration took 80.5 / 117.5 / 29.2 s (48 / 32 / 16 rows) and an L2 fit 20.7 / 15.3 / 7.5 s, so the admin calls need --decision-timeout-ms above its 30 s default (at 30 s, before the change described in section 6, the call answered 408 while the calibration kept running, and further operations answered 429 decision_busy). The API refused, as designed: a 70 KiB state (400, "decision state text is 71680 bytes, limit is 65536"), an L2 fit with 2 rows (400, "L2 requires at least 8 labeled states for this question"), calibration without the operator key (403). The L2 heads on 48 and 16 rows did not beat raw scoring on the yes/no sets: small fit sets.
  • After a 261,120-token prompt (runner mode decide-depth; the prompt in 300.8 s, then about 0.95 s a question with the prompt's checkpoint restored for each label): four yes/no questions about facts planted in the prompt, 3 of 4 right (true claims 0.748 and 0.823 yes; a false claim 0.431 yes, right; another false claim 0.585 yes, wrong).

LoRA (synthetic rank-16 adapter from inkling-lora-synth, scale 1, 2026-11-21): device against the reference with the same adapter, 48 positions one at a time, unrounded caches: KL 0.00000, the same top token throughout; the binary16 prompt kernels (16-token passes) against it: KL 0.00018; the adapter's own effect against the base model KL 0.00095 (same top 97.92%). A zero adapter (B = 0) gives output bitwise equal to the base, single tokens and 16-token passes (4-bit caches). Cost with a 4,096-token prompt (one cold prompt per process): prompt 1,843.9 tok/s against 2,042.2 without (-9.7%), generation 44.29 against 43.94 (no measurable cost). Served with --lora at 262,144 positions: /health lists the adapter with its SHA-256, and the chat answer is unchanged in form (reasoning and content separated).

Host recording against replay (runner mode replay-ablation, 64 decode steps at one position, each step's command buffers either recorded as usual or replayed from a recording): at 4,096 positions 22.42 ms a step recorded, 19.25 ms replayed (22.38 recorded again); at 16,384: 22.92 / 19.84 / 22.93. About 3.1 ms a token (14%) is host-side command recording: the size of the gain available from recording a token's whole forward once. With the recorded forward (position word and replay, the commit after the decision commits): 128 generated tokens at 4,096 positions 50.25 tok/s (43.54 recording every token), at 16,384 49.12 (42.47), the generated ids identical, logits bitwise equal.

Per-token work, 2026-11-21 evening, each change alone against the same baseline in one binary (switches; 4-bit caches; 128 generated tokens after a 4,096 / 16,384-token prompt, one cold prompt per process; accuracy = 4-bit run against the vendor reference, 510 rows, single tokens / 16-token passes):

Change (switch) Generation 4k / 16k Prompt 4k / 16k Accuracy Kept
none (baseline) 50.36 / 49.12 tok/s 2,004 / 1,884 tok/s 0.00419 / 0.00351
controllers in two commands per set (FUSED_CONTROL) 52.31 / 50.89 2,101 / 1,973 0.00428 / 0.00343 yes
mix and norm in one command per token 50.18 / 48.73 2,018 / 1,896 bitwise equal no (slower)
per-token expert loops unrolled 50.10 / 48.63 2,004 / 1,887 bitwise equal no (slower)
one-token history updates folded into cache-row and branch commands 50.38 / 49.10 2,005 / 1,886 bitwise equal no (no gain)
matrix-core prompt attention, sums in registers (ATTN_REGACC) unchanged 2,094 / 1,975 0.00419 / 0.00359 yes
8-bit inputs to the 4-bit experts (W4A8, with the unrolled loop) 52.81 / 51.37 unchanged 0.00372 / 0.00351 option, off
the attention projections of one input in one command (bitwise equal) 1.0 ms a token slower unchanged bitwise equal no (slower)
32 queries a workgroup in prompt attention (ATTN_Q32, on top of the kept ones) unchanged 2,245 / 2,148 against 2,200 / 2,070 16-token passes unaffected yes

The 8-bit input rounding alone (LEXI_INKLING_INPUT8, same kernels): 0.00419 none, 0.00385 experts, 0.00371 projections, 0.00384 all: no measurable cost. W4A8 cannot pass the arithmetic check (KL 0 against the CPU reference with unrounded caches): against the reference rounding the same inputs (inkling-reference --inputs q8) it gives KL 0.00024 at 48 positions and 0.00094 at 1,024, and two device paths that differ only in the rounding formula differ by 0.00052, because an input that differs in its last bit flips a value at a rounding boundary by a whole step and the loop carries it. So it stays off by default. At 261,632 positions REGACC takes the prompt from 878.2 to 959.9 tok/s (297.9 -> 272.6 s; the generated tokens after the prompt differ, the sums being added in another order).

The 2c0f15fe kernels at 16,384 tokens (2026-11-21, development A/B in one session, single prompt per process): prompt 1,573.5 tok/s and generation 37.35 tok/s, against 1,409.8 and 32.69 with the earlier kernels switched back on; the 64 generated tokens identical.

llama.cpp 29b52d398 + patch on the same file and card (llama-bench -ngl 99 -fa on -ctk q4_0 -ctv q4_0 -b 2048 -ub 512, 2026-11-20, a disk copy running beside it): prompt 425 / 397 tok/s at 4,096 / 16,384, generation 25.4 / 25.0. At 262,144 it loads (about 22.3-22.9 GB on the card plus 3.9-4.5 GB reported in system memory) but did not finish a 261,632-token prompt in 55 minutes (two passes, llama-bench) nor a 129,374-token prompt in 45 minutes (one pass, llama-cli, -c 131072); no rate exists for it at depth.

11. Known limits (2026-11-21)

  • Numerical accuracy is verified against the reference only at short context (up to 1,024 positions); at 16,384 the new and old attention kernels agree on 64 generated tokens; deep in the context the retrieval checks find 5 of 5 and 10 of 10 planted facts at about 261,000 tokens and 10 of 10 at 523,264 (still a small sample).
  • Contexts measured: up to 524,288 positions on the 24 GB card (the draft head then forces 1,024-token passes and does not beat plain decoding); 1,048,576 is refused (section 5).
  • LEXI_INKLING_W4A8=1 (8-bit expert inputs, about 1.1 ms a token faster) is off by default: no 8-bit-input path can pass the KL-0 arithmetic check against the reference.
  • The draft head is read from the vendor checkpoint folder (LEXI_INKLING_DRAFT, config.json and mtp.safetensors): it is not in the converted GGUF (the Panel passes it through the instance setting INKLING_DRAFT). Deep in the context its gain is small (41.8 vs 36.6 tok/s at 261,632) and at 523,264 it is slower than plain decoding without it (32.1 vs 33.7): its own attention over the whole context costs about 17 and 26 ms a step there. The switch from runs of 8 to runs of 4 at 131,072 positions is not measured between 16,384 and 261,632.
  • The draft head is used for greedy requests only (sampled requests decode plainly; the scheduler's sampled speculative step is not implemented for this runtime). Its windowed context rows are not part of the prompt-reuse checkpoint, so right after a restored turn its first proposals may be worse (the output is unaffected).
  • The vendor's 4-bit inputs to the 4-bit experts are not implemented; the runtime uses binary16 (prompt passes) or F32 (single tokens) inputs there. Not checkable against vendor code.
  • Vision (the image tower) is not implemented.
  • The 8 MXFP8 expert tensors are Q8_0 in the file (re-quantized by the converter), not on the vendor's grid.
  • Raw completions (/v1/completions) show the model's turn tokens (<|content_thinking|> and so on) as text: the runtime's tokenizer spells them so the chat endpoint can put the thought in reasoning_content and the answer in content.
  • Decisions through the API are bounded by its 64 KiB state and 256 KiB prompt limits (a decision after a 261k-token context is checked through the runner only; section 6).
  • One sequence at a time; prompt reuse only for a prompt that extends what the caches hold or starts with the previous request's whole prompt (section 6); 4-bit cache rows only (no 8-bit format).

The llama.cpp patch (llama.cpp-29b52d398-inkling2.diff)

Changes to llama.cpp (MIT) commit 29b52d398; apply with git apply.

Patches to llama.cpp 29b52d398 ("model : Inkling 2 Mini", build 12600), MIT licensed upstream code, used to
convert Inkling-2-Mini with its experts on the vendor's 4-bit grid and to run that file as the comparison baseline.
Apply with: git -C llama.cpp checkout 29b52d398 && git -C llama.cpp apply llama.cpp-29b52d398-inkling2.diff
See ../runtimes/INKLING-2-VULKAN-20261121.md, section 8.

diff --git a/conversion/inkling2.py b/conversion/inkling2.py
index 63e73a6d5..599c864ed 100644
--- a/conversion/inkling2.py
+++ b/conversion/inkling2.py
@@ -12,6 +12,8 @@ import torch
 if TYPE_CHECKING:
     from torch import Tensor
 
+import os
+
 from .base import LazyTorchTensor, ModelBase, MmprojModel, TextModel, gguf, logger
 
 
@@ -158,6 +160,38 @@ class Inkling2Model(TextModel):
                 raise ValueError(f"Inkling2: {name}_scale: unsupported stored block shape {block}")
             if list(block) != config.get("weight_block_shape"):
                 logger.warning("Inkling2: %s: hf_quant_config.json weight_block_shape %s conflicts with stored weight/scale shapes %s/%s; using %s", name, config.get("weight_block_shape"), tuple(tensor.shape), ss, list(block))
+            if nvfp4 and os.environ.get("INKLING2_NATIVE_NVFP4") == "1":
+                # Local addition: keep the packed experts on their own grid. The 4-bit values and the E4M3 scale per
+                # 16 values are repacked into ggml's NVFP4 blocks unchanged; the per-expert factor amax / denominator
+                # goes into the per-expert scale tensor the MoE step applies.
+                amax_t, den_t = (LazyTorchTensor.to_eager(self.model_tensors[n]()) for n in companions[1:])
+                den = float(den_t.float().item())
+                packed = LazyTorchTensor.to_eager(tensor).view(torch.uint8)
+                scales = LazyTorchTensor.to_eager(scale_tensor).view(torch.uint8)
+                if bool(((scales & 0x80) != 0).any()) or bool(((scales & 0x7F) == 0x7F).any()):
+                    raise ValueError(f"Inkling2: {name}_scale: negative or NaN E4M3 scale, cannot keep as unsigned")
+                blocks = []
+                for e in range(experts):
+                    w, sc = packed[e], scales[e]
+                    if w13:
+                        # stored rows alternate gate and up; the fused tensor holds the gate rows, then the up rows
+                        w, sc = torch.cat([w[0::2], w[1::2]]), torch.cat([sc[0::2], sc[1::2]])
+                    raw, _ = self._nvfp4_pack(w, sc)
+                    blocks.append(raw)
+                merged = np.stack(blocks, axis=0)
+                target = name.replace(".w13_weight", ".gate_up_proj").replace(".w2_weight", ".down_proj")
+                new_name = self.map_tensor_name(target, try_suffixes=()) + ".weight"
+                logger.info(f"Repacked {new_name} natively as NVFP4, shape [{experts}, {rows}, {cols}]")
+                self.gguf_writer.add_tensor(new_name, merged, raw_dtype=gguf.GGMLQuantizationType.NVFP4)
+                self._write_scales_tensor(new_name.replace(".weight", ".scale"), [float(a) / den for a in amax_t.float().tolist()])
+                # Inputs of these products stay at full precision unless INKLING2_A4=1 (4-bit inputs in the backend
+                # gave garbage with this file on Vulkan, 2026-11-20).
+                if os.environ.get("INKLING2_A4") != "1":
+                    self._prec_a4[new_name] = False
+                del blocks, merged
+                for n in [name] + required:
+                    del self.model_tensors[n]
+                continue
             amax = denominator = None
             if nvfp4:
                 amax, denominator = (self.model_tensors[n] for n in companions[1:])
diff --git a/src/llama-graph.cpp b/src/llama-graph.cpp
index 269b920fb..833696ca8 100644
--- a/src/llama-graph.cpp
+++ b/src/llama-graph.cpp
@@ -1897,7 +1897,7 @@ ggml_tensor * llm_graph_context::build_lora_mm_id(
 
     prec_policy.apply(res);
 
-    if (w->type == GGML_TYPE_NVFP4) {
+    if (w->type == GGML_TYPE_NVFP4 && getenv("LLAMA_NVFP4_ACC_F32") == nullptr) {
         ggml_prec_set_acc(res, GGML_PREC_BF16);
     }
 
diff --git a/src/llama-model.cpp b/src/llama-model.cpp
index 4c77f6f68..0971c27a5 100644
--- a/src/llama-model.cpp
+++ b/src/llama-model.cpp
@@ -2006,6 +2006,19 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
                 layer.nextn.shared_head_head_in_s = create_tensor(tn(LLM_TENSOR_NEXTN_SHARED_HEAD_HEAD, "input_scale", i), {1}, TENSOR_NOT_REQUIRED);
             }
         }
+
+        // local fix: cache slots that share a physical layer's expert tensors (looped models) were copied before
+        // the scales above were loaded, and no scale tensor carries their slot index: share the scales as well
+        for (int i = 0; i < n_layer_all; ++i) {
+            for (int j = 0; j < i; ++j) {
+                if (layers[j].ffn_down_exps && layers[j].ffn_down_exps == layers[i].ffn_down_exps) {
+                    if (!layers[i].ffn_up_exps_s)   { layers[i].ffn_up_exps_s   = layers[j].ffn_up_exps_s; }
+                    if (!layers[i].ffn_gate_exps_s) { layers[i].ffn_gate_exps_s = layers[j].ffn_gate_exps_s; }
+                    if (!layers[i].ffn_down_exps_s) { layers[i].ffn_down_exps_s = layers[j].ffn_down_exps_s; }
+                    break;
+                }
+            }
+        }
         // output scales
         if (output) {
             // weight scale

Clone this wiki locally