Skip to content

v1.5.0 — resident short-context serving

Choose a tag to compare

View all tags
@skyguan92 skyguan92 released this 05 Aug 07:52
· 353 commits to main since this release
v1.5.0
d82e694

What changed

  • A q8192 resident service now keeps q1024/q2048/q4096/q8192 AOT prefill buckets loaded and selects the largest fitting bucket per request.
  • Exact request-prefix reuse is now a capacity-bounded LRU: four entries at q8192, two at medium long contexts and one at the largest windows.
  • Health, CLI and request metrics expose resident buckets, cache capacity and executed AOT-prefill tokens.
  • Release qualification now includes a privacy-safe, frozen MMLU-256 scorecard paired against identical GB10 vLLM prompt-token hashes.

Qualified result

  • All 19 batch-1 performance cells passed; minimum frozen-baseline retention was 1.013x prefill and 0.9858x decode.
  • Nine-context full-vocabulary correctness passed with top-1 agreement everywhere and worst KLD 0.002174 < 0.005.
  • The exact q8192 128-token completion matched.
  • MMLU-256 scored 216/256 (84.375%), equal to the frozen GB10 vLLM reference, with zero invalid answers and all 256 prompt-token hashes matched.
  • SSE streaming, OpenAI function tools, disconnect recovery, variable prompts, multi-entry LRU, startup and isolated bundle checks passed.
  • The checksum-identical archive reproduced on a second AMD395: q8192 cold prefill 1722 tok/s, decode512 32.36 tok/s, exact completion and resident HTTP all passed.

Runtime boundary

The archive is self-contained at userspace level: no host Python, PyTorch, vLLM, Triton, Transformers or ROCm userspace is required. The host still needs Linux x86-64, AMDGPU/KFD/render access, Radeon 8060S (gfx1151), the documented 96 GiB GTT setup and the separately obtained pinned model checkpoint. Model weights are not included.

sha256sum -c aima-engine-native-portable-86e806e8bc5d.tar.zst.sha256
tar --zstd -xf aima-engine-native-portable-86e806e8bc5d.tar.zst
cd aima-engine-native-portable-86e806e8bc5d
./bin/aima-engine doctor --model-dir /srv/models/Qwen3.6-35B-A3B --json
./bin/aima-engine serve --model-dir /srv/models/Qwen3.6-35B-A3B --context-tokens 8192

Portable archive SHA-256: 5ca97a234c1132ec0e715f463107faa7bdce6dab6ab053918cf1465b8d2ea62b
Public evidence SHA-256: d47c49b874d8c421fdd270a0eaa698f6938348726576a20b2c5f0931eaac5e36

The immutable release tag points to d82e6943bc50d821011ce79e95afee06f6b12a36; the additive evidence inventory is on main at 483260b0a41933bc85c35f2e364692efc66bd3db.

Assets 6
Loading