Siliang Engine v0.1.3

v0.1.3 brings the bounded MoE memory hierarchy into the normal Siliang Engine
CLI/server lifecycle and qualifies it across several model families on the
Windows CUDA reference workstation.

The main goal of this release is not to establish a new absolute performance
ceiling. It is to move the previously experimental memory hierarchy into a
bounded, configurable, release-gated runtime path and verify that it behaves
correctly across different MoE architectures.

What changed

Added typed expert-cache configuration shared by llama-cli and
llama-server; no hidden environment setup is required.
Added managed host L2 policies plus bounded CUDA K residency, per-schema R
exchange slots, and a global pinned P elevator.
Canonicalized Siliang lifetime-frequency admission as slfu;
cumulative-lfu remains accepted as a legacy alias.
Added exclusive K/L2 turnover controls (--admit-k-cold and
--demote-k-hot) for SLFU.
Added the architecture-guarded DeepSeek4 FRONT rolling path. The release
candidate fences a FRONT bank overwrite behind completion of preceding CUDA
consumers; this closes a determinism bug found during release qualification.
Added an opt-in routed-MoE prompt path using bounded route unions. It remains
experimental and is qualified separately from decode.
Kept router scoring and mixture weights on the GPU; the CPU mapper receives
selected expert IDs only for cache control and physical-slot translation.
Added context-owned lifecycle integration, failure-closed capacity checks,
aggregate route telemetry, and release/runtime gates.
L1 K/R/P remains intentionally incompatible with LoRA in this prototype.

v0.1.3 release qualification

The table below is new v0.1.3 server evidence, not a restatement of the
older benchmark claims.

These are natural-generation qualification runs rather than route-replay
benchmarks. No frozen route trace, route oracle, or replayed expert sequence was
used. Each repetition started a fresh llama-server process and the measured
request was issued directly, without a separate generated-token warmup request.

The measured decode used 256 generated tokens with temperature=0, top-k=1,
seed 42, and cache_prompt=false. Routed-expert activity therefore follows the
model's actual autoregressive generation. This does not exclude llama.cpp's
normal internal startup/empty-run initialization.

The exact commands are in
../CONFIGURATION.md.

Model / path	Median decode	Range	Result
Gemma4 26B-A4B, K1440/R16/P16	21.651 tok/s	21.253-21.672	3/3 complete, identical token hash
Qwen3 30B-A3B, K1440/R16/P16	19.261 tok/s	17.360-20.164	3/3 complete, identical token hash
Qwen3.6 35B-A3B, K1440/R16/P16	9.279 tok/s	7.549-9.534	Correct, but not the recommended path
Qwen3.6 35B-A3B, no expert cache	11.011 tok/s	10.210-11.392	3/3 complete, identical token hash; recommended v0.1.3 path
Ornith 1.0 35B, K1920/R16/P16	13.967 tok/s	13.948-14.004	3/3 complete, identical token hash
GPT-OSS 120B, 18 GiB managed L2	3.344 tok/s	3.335-3.356	3/3 complete; all runs observed low host-memory headroom

GPT-OSS is retained as a supported managed-L2 path.

The current 3.344 tok/s median should not be interpreted as a matched
regression against the historical 4.052 tok/s result. The historical
benchmark issued a separate 48-token generated warmup before each 256-token
measured request, while the v0.1.3 qualification above did not. In addition,
all three current repetitions were observed under low host-memory headroom.

The two values therefore characterize different execution conditions; the
current value is retained as release qualification evidence rather than
promoted as a new performance ceiling.

DeepSeek4 current profile

The v0.1.3 DeepSeek4 decode profile is:

& "<llama-server.exe>" -m "<deepseek4-expert-major.gguf>" `
    -ngl 99 -ncmoe 43 -nkvo --no-op-offload `
    -c 4096 -b 512 -ub 512 -t 2 -tb 2 --parallel 1 `
    --expert-cache `
    --expert-cache-l2-mib 8192 `
    --expert-cache-l2-policy lru `
    --expert-cache-l1-k 216 `
    --expert-cache-exchange-r 12 `
    --expert-cache-elevator-p 12 `
    --expert-cache-l1-policy slfu `
    --admit-k-cold on `
    --demote-k-hot on `
    --expert-cache-roll deepseek4 `
    --no-expert-cache-prefill

During the release gate, the original asynchronous FRONT overwrite path produced
different greedy continuations across fresh starts. Isolation showed:

--no-expert-cache: deterministic 3/3;
managed L2 only with FRONT off: deterministic 3/3;
K216/L2/R/P with FRONT off: deterministic 3/3;
FRONT enabled before the fence: divergent output after roughly 19-44 tokens.

The release-candidate FRONT completion fence then produced an identical token
hash in 3/3 fresh 64-token runs at 1.936-1.983 tok/s.

A separate full-depth release gate generated all 2,048 tokens at
1.94436 tok/s (514.31 ms/token). The run reached the 2,048-token route
checkpoint with zero runtime failure.

Host-memory headroom was low during that run, so this is primarily a
depth/stability result, not a clean isolated throughput ceiling.

The historical 2.774 tok/s DeepSeek4 result discussed below is also a real
natural-generation benchmark, but it is not a matched before/after comparison
with this profile. It used an 18 GiB arena and 256-token fully-cold repetitions,
while the current release profile uses 8 GiB L2 plus K216/R12/P12 and FRONT and
was additionally exercised over a 2,048-token decode.

The raw tok/s values therefore represent different operating points and should
not be interpreted as evidence of a v0.1.3 throughput regression.

Historical performance evidence

The earlier README/PERFORMANCE measurements remain part of the project evidence
and are intentionally not rewritten as v0.1.3 claims.

They answer different questions from the release qualification above. The
v0.1.3 qualification uses natural fresh-server generation with no separate
generated-token warmup request and no route replay/oracle. Historical
experiments used their own frozen protocols, including different arena
capacities, fully-cold process starts and, for GPT-OSS, a separate generated
warmup request.

Raw tok/s values across the two sections are therefore not matched
before/after regression measurements.

Notable retained results include:

DeepSeek V4 Flash 0731 expert-major: 2.774 tok/s median versus
2.274 tok/s for the stock layout with the same 18 GiB arena (+22.0%).
This was a matched fully-cold natural-generation benchmark: three fresh
processes per arm, Windows standby-list purge before every process start,
and exactly 256 generated decode tokens per request with no separate
generated-token warmup. All six outputs were byte-identical and reached the
fixed 256-token limit.
DeepSeek V4 Flash stock GGUF: 2.291 tok/s with the 18 GiB arena
versus 1.375 tok/s mmap on the same stock file (+66.6%). This is a
separate matched same-file historical experiment isolating the arena path
rather than the expert-major layout.
GPT-OSS 120B: 4.052 tok/s managed-arena median versus
1.972 tok/s mmap on the same repacked file (+105.5%). This comparison
used three fresh processes per arm and a matched protocol on both sides:
each process first generated a separate 48-token warmup request, followed by
a 256-token measured request. All six measured cells produced 256 nonempty
tokens and the same retained decoded text. The 48 warmup tokens were not
included in the reported decode throughput.
Source-research DS4 natural-depth observations: single 2,000-token starts
at 12/14/16 GiB L2 completed at
2.12587 / 2.20073 / 2.28674 tok/s respectively. These were actual
autoregressive generations, not frozen-route or route-replay experiments,
but they were single starts with different generated work and remain
research observations rather than matched release benchmarks.

See ../PERFORMANCE.md for the complete protocols,
limitations, raw repetitions, output-equivalence evidence and retained artifact
identities.

Model boundaries

Gemma4 26B-A4B: positive v0.1.3 K1440 qualification.
Qwen3 30B-A3B: positive v0.1.3 K1440 qualification.
Qwen3.6 35B-A3B: the K1440 path is correct but slower than the matched
no-cache control; the shipped recommendation is --no-expert-cache.
Ornith 1.0 35B: positive v0.1.3 K1920 qualification.
GPT-OSS 120B: supported through managed L2; the current release receipt
is pressure-qualified and uses a different protocol from the historical
warm-runtime performance benchmark.
DeepSeek4: K216/R12/P12 + 8 GiB L2 + FRONT is the current release profile;
FRONT determinism is explicitly gated and a full 2,048-token decode completed
successfully.
M4: remains inconclusive/no-go; no useful integrated architecture path is
claimed in v0.1.3.

Whole-L2 CUDA registration, heterogeneous batch-copy experiments, route replay
oracles as product inputs, DirectStorage rolling, generic whole-model static
rolling, and the cost-model/pre-bake solver remain outside v0.1.3.

Route replay and frozen-route experiments remain research tools: none of the
release-qualification throughput rows above depend on them.

Server use

Use --parallel 1 whenever L1 K/R/P is enabled. The OpenAI-compatible server
can then be addressed normally at /v1.

DeepSeek non-reasoning integrations can additionally use:

--reasoning off --reasoning-format deepseek

when the client expects plain content while retaining tool parsing.

This release changes the public llama_context_params layout. Applications and
plugins must be rebuilt against the v0.1.3 headers; binary compatibility with a
v0.1.2 library is not claimed.

Navigation Menu

Siliang Engine v0.1.3 (Experimental)

Choose a tag to compare

Sorry, something went wrong.