Skip to content

Delta Inference Engine v0.91

Choose a tag to compare

View all tags
@musicurgy musicurgy released this 26 Oct 21:41
· 245 commits to main since this release
0.91
3f08326

Delta Inference Engine 0.91

One model download instead of two. The engine now reads DeepSeek's Engram tables straight from
nazeshinjite's ds4 Q2 GGUF, which carries
them inline, byte for byte DeepSeek's own. Setup no longer downloads DeepSeek's two Engram shards or makes a slimmed
copy:

  • ~375 GB to download (was ~565), ~385 GB free needed while it runs, ~371 GB kept.
  • Same output: the Engram rows, and the model's logits on a 4K-token prompt, are identical to reading DeepSeek's
    shards. Prompt reading is a few percent faster (one disk read per Engram row group instead of two).
  • An install from 0.9 (the slimmed GGUF + the Engram shards) keeps working as it is.

The three most-downloaded quants of 1017 all run. Besides nazeshinjite's ds4 Q2 (the default) and ISTA-DASLab's
GSQ-RCO, the engine now reads Unsloth's GGUFs as they
are: UD-IQ1_S up to UD-Q8_K_XL (llama.cpp's own layout: split files, the Engram tables in one tensor, IQ4_NL or
DeepSeek's exact FP8). All nine load, and every expert type they use is checked against ggml's reference on real
weights. Measured on an RTX 3090 + 192 GB:

Download Writing speed Agreement with DeepSeek (general / code+maths)
ds4 Q2 (default) 366 GB ~41 tok/s 91.5% / 93.0%
Unsloth UD-IQ1_S 233 GB, nothing else needed 41 tok/s 89.3% / 91.1%
GSQ IQ2_XS 159 GB + the default 38 tok/s 90.0% / 92.9%

The bigger Unsloth quants are not measured yet; once their experts outgrow the RAM they write slower.

New: Choose Model.bat. It lists the models you have and the ones you can add, with size, speed and agreement:
it builds a GSQ quant from ISTA-DASLab's files, or downloads an Unsloth quant (every file SHA-256-checked). The
launcher loads your pick. To use only an Unsloth quant (no 366 GB default): answer N when Start DeepSeek.bat offers
the default model's download, run Choose Model.bat, then start again.

Also:

  • The drafter setup step works with the engines for older CPUs too.
  • A DSpark drafter fix from upstream Xolotl (be8f039): a short final prompt batch seeded the drafter with empty rows
    (output was exact; acceptance was lower).
  • The engine zips now include LICENSE.txt and THIRD-PARTY-NOTICES.txt.

Upgrading from 0.9: pull the new code, then double-click Start DeepSeek.bat. It sees that the 0.9 engine is out
of date and offers to download this release's engine.

Downloads: Start DeepSeek.bat picks the right zip for your CPU by itself.

File CPU
build-ds41-portable.zip AVX2: Intel 2013+ (Haswell on), every AMD Ryzen / Threadripper / EPYC
build-ds41-compat-avx.zip AVX without AVX2 (Sandy / Ivy Bridge)
build-ds41-compat.zip anything x86-64 with SSE4.2 (slower)

SHA-256 checksums are in SHA256SUMS.txt. The zips include NVIDIA CUDA runtime libraries (cudart, cublas,
cublasLt), redistributed under the NVIDIA CUDA Toolkit EULA.

Assets 6
Loading