Skip to content

Releases: dansupergameprogrammer/SuperSLM

SuperSLM 1.13.1

Choose a tag to compare

@dansupergameprogrammer dansupergameprogrammer released this 19 Oct 15:29

SuperSLM 1.13.1

Released 2026-10-19.

SuperSLM 1.13.1 is a fix release for the GPU backend. A GPU page pool is now capped at the largest buffer
D3D12 guarantees on the card, and a larger request is refused before anything is allocated. On the AMD
Radeon RX 7900 XTX, a 4 GiB pool used to make the driver remove the device. It also creates default-heap
GPU buffers in the state D3D12 actually gives them, which removes a debug-layer warning on every such
buffer. No token, save format, existing status or existing verb changes from 1.13.0, and no model artifact
needs reconversion. One GPU verb is added.

What changed

  • GPU page pools are capped at D3D12's guaranteed resource size. A pool is one committed D3D12
    buffer. D3D12 requires a driver to create a resource only up to max(128 MiB, min(2048 MiB, a quarter
    of the adapter's dedicated video memory)), so sslm_gpu_kv_pool_create now refuses a pool above that
    size with SSLM_GPU_KV_POOL_TOO_LARGE, before allocating anything. In 1.13.0 the only limit was
    2^32 bytes, the reach of the shaders' offsets, which is still the upper bound. The cap is 2 GiB on an
    adapter that reports 8 GiB or more of dedicated video memory: 21,845 pages at Qwen2.5-0.5B and 9,362
    at Qwen2.5-1.5B (cap 4,096, 16-position pages). An RTX 2080 SUPER reports 8,374,976,512 bytes, so its
    cap is 2,093,744,128 bytes: 21,298 and 9,128 pages.

    Why: at 1.13.0's ceiling (43,690 pages at 0.5B, a 4,294,901,760-byte buffer), the RX 7900 XTX's
    driver (32.0.23033.1002) answered the committed-buffer create with DXGI_ERROR_DEVICE_REMOVED, removal
    reason DXGI_ERROR_DRIVER_INTERNAL_ERROR, so the create returned SSLM_DEVICE_LOST. The RTX 2080
    SUPER created the same buffer. A size between D3D12's guarantee and 2^32 bytes is up to the driver.

  • New verb: sslm_gpu_kv_pool_max_page_count(ctx, model, &pages) reports the largest page count
    sslm_gpu_kv_pool_create admits for a model on the context's adapter. It validates its arguments as
    the create does (SSLM_SEQUENCE_KV_BUFFER_MISMATCH with *pages 0 on a null out-pointer or a context
    or model mismatch) and allocates and submits nothing.

  • Default-heap buffers are created in COMMON. D3D12 creates an ordinary default-heap buffer in
    COMMON whatever initial state is requested, and the debug layer reports the request as
    CREATERESOURCE_STATE_IGNORED. The allocator now asks for COMMON, the state the buffer really has.
    Upload and readback buffers keep their required initial states, and acceleration structures keep
    theirs. The engine's logical state tracking is unchanged: a COMMON buffer is promoted to its first
    use's state, which D3D12 allows.

Compatibility and required action

  • The ABI change is additive: one new GPU verb in gpu_1p0.h. No existing declaration, struct layout or
    status ordinal changes. The CPU C ABI is unchanged.
  • A host that asked for a pool above its adapter's cap now gets SSLM_GPU_KV_POOL_TOO_LARGE where it got
    a pool (or, on the RX 7900 XTX at 4 GiB, a lost device). Use sslm_gpu_kv_pool_max_page_count to size
    pools. A pool at or below the cap behaves as in 1.13.0.
  • The shaders are unchanged from 1.13.0.

Verification

  • Pool-cap arithmetic: gpu_kv_pool_geometry_tests, 2,083 checks, 0 failures (Windows, MSVC). The same
    tests built against a copy of the cap function that returns 1.13.0's 2^32 fail every cap check.
  • RTX 2080 SUPER, D3D12 debug layer on: paged-KV GPU cells 11.4 (the pool size guard, 33 checks: a pool at
    the 2,093,744,128-byte cap admitted at both model sizes, one page more refused), 2.6 (13 checks, including
    sslm_gpu_kv_pool_max_page_count against the test's own arithmetic and its refusals), 4.8 (102
    checks, a full-size pool driven in its upper half), 2.11 and 5.7: all green.
  • RX 7900 XTX (driver 32.0.23033.1002), cell 11.4 with the debug layer on: 33 checks, 0 failures; pools of
    21,845 pages (0.5B) and 9,362 pages (1.5B), 2,048 MiB each, admitted. At 1.13.0 the same cell read
    SSLM_DEVICE_LOST at the 2^32 ceiling.
  • SuperSLM-Unreal's validation battery on the RX 7900 XTX, built on this source: 243 editor test cells and
    its packaged 50-NPC GPU scenario without a renderer, all passing.
  • Default-heap COMMON creation: the buffer-state probes (gpu_buffer_initial_state_tests and its
    state-tracking build) pass on the Windows runtime and on an Unreal Agility SDK runtime; with the
    previous allocator, the Agility controls reproduce the creation warnings.
  • Windows, MSVC Release, at this release's tree, RTX 2080 SUPER: all 23 ctest entries pass, including
    superslm_tests and the three Windows GPU schema cells (t2933_schema_query_native_state,
    t2933_schema_lifecycle_native_state, t2933_schema_real_model). Those three build their own executables,
    and since 1.13.0 their builders omitted src/gpu/gpu_inflight_registry.cpp, so on a GPU machine they
    failed to link; 1.13.1 adds it. CI does not run them (they need a hardware D3D12 device).
  • Linux: the release tag's CI run.

Files changed

  • src/gpu/gpu_kv_pool_geometry.h -- MaxPoolBytesForAdapter, and the pool-size refusal and largest
    admissible page count under a given cap.
  • src/gpu/d3d12_harness.h -- records the selected adapter's dedicated video memory; default-heap buffers
    are created in COMMON.
  • src/gpu/gpu_1p0.cpp, include/superslm/gpu_1p0.h -- the cap at pool create, and
    sslm_gpu_kv_pool_max_page_count.
  • tests/gpu_kv_pool_geometry_tests.cpp, tests/paged-kv-gpu/ (pool_cap_helpers.h new; cells 2.6, 4.8
    and 11.4 and their helpers), tests/gpu_buffer_initial_state_tests.cpp (new), CMakeLists.txt -- the
    tests above.
  • tests/t2130-g5-red-suite/run_t2922_query_runtime.ps1, run_t2922_lifecycle.ps1, run_t2922_real_model.ps1
    -- link src/gpu/gpu_inflight_registry.cpp.
  • docs/api.md, README.md, CHANGELOG.md, tests/install-consumer/CMakeLists.txt, this note --
    version 1.13.1.

SuperSLM 1.13.0

Choose a tag to compare

@dansupergameprogrammer dansupergameprogrammer released this 16 Oct 00:01

SuperSLM 1.13.0

Released 2026-10-15.

SuperSLM 1.13.0 brings 1.12.0's paged K/V memory to the GPU backend. A GPU page pool holds the K/V of
budgeted sequences and prefixes in pages. A budgeted sequence reserves only the pages its budget
needs, sequences that adopt a frozen prompt prefix map the prefix's full pages instead of copying
them, and a budgeted sequence's save, the new SLM6 blob, holds its live rows, so its size follows
its live length. It also folds in the fixes planned for a 1.12.1: a no-budget CPU save now writes a
large block with streaming stores, aimed at 1.12.0's save slowdown (the bytes are unchanged; on the
release machine a full-length save reads 1.27 and 1.14 times 1.11.0's time where 1.12.0's read 1.88
to 1.93 times, so it is still slower than 1.11.0's, see Speed), and two GPU defects that shipped in
earlier releases are fixed (see Fixes).

GPU sequences made by sslm_gpu_seq_create keep their own whole-cap buffer and keep saving SLM5,
which 1.12.0 still reads. They now address K/V through a page table with one page, which by
construction is the old flat layout. Their outputs are meant to be byte-identical to 1.12.0's; the
release machine's check compares them with a record taken on this branch's build before the
page-table change, not with a 1.12.0 build (see Verification). On the NVIDIA GeForce RTX 2080
SUPER, superslm_tests with its GPU cells, the GPU prefill-read suite and the GPU allocation-status
suite passed on this branch before its last library fixes, and the two race cells, the GPU
batched-prefill suite and the GPU red-suite cells listed under Verification passed after them with
the D3D12 debug layer on, while the
cross-vendor battery reads red for reasons that predate this release and some release-machine runs
are still open; nothing in this release is verified on the AMD Radeon RX 7900 XTX (see
Verification). Paged GPU decode reads within its bar of 5% slower than the one-page view, not
faster (see Speed). The memory savings come only from the new pooled verbs. Two GPU
behaviours change for every host (see Compatibility): two decode calls raced on one sequence
handle from two threads (sslm_decode_step_gpu, a batch slot, the bridge decode or the finish, in
any pairing) leave one running and return SSLM_BUSY to the other where it was undefined, and a
call that would reset the process's GPU command list while earlier work is still unobserved
returns SSLM_BUSY where it was undefined. The shipped shaders changed, so deploy the whole 1.13.0 shader
set. The CPU C ABI is unchanged (51 verbs, no new status), and no model artifact needs reconversion.

What changed

  • GPU page pools. sslm_gpu_kv_pool_create(ctx, model, page_count, &pool) creates one device
    buffer of page_count pages, zero-filled before it returns, with a host mirror of the same size.
    A GPU page is B positions, where B is the artifact's kv_block_size when it divides the
    context cap and the cap otherwise: 16 for converted artifacts at cap 4,096. That is not the CPU's
    page (64 positions at Qwen2.5-0.5B, cap 4,096), so the two backends' page counts differ for the
    same cohort. A page of Qwen2.5-0.5B is 96 KiB (24 layers × K and V × 2 K/V heads × 16 positions ×
    64 bytes) and of Qwen2.5-1.5B 224 KiB, from the page rule. sslm_gpu_kv_pool_destroy destroys an
    empty pool and refuses with SSLM_MODEL_HAS_LIVE_SEQUENCES while anything drawn from it lives.
    sslm_gpu_kv_pool_stats reports the pool's page count, free and shared pages and its B.
  • Budgeted GPU sequences. sslm_gpu_seq_create_budgeted(ctx, model, pool, budget, &seq) takes a
    budget, 1 ≤ budget ≤ cap, and reserves R(budget) = min(ceil(budget / B) + 1, ceil(cap / B))
    pages at create, the CPU's rule, or returns SSLM_GPU_KV_POOL_EXHAUSTED with nothing drawn. A
    pooled sequence's cap is the model's context cap; unlike sslm_gpu_seq_create's context_cap,
    the host does not choose it. Decode and prefill then take pages only from that reservation. At the
    limit, min(origin + budget, cap), below the cap, a decode returns SSLM_GPU_KV_BUDGET_EXCEEDED
    with no row written, and a prefill that reaches the limit commits the tokens below it and then
    returns SSLM_GPU_KV_BUDGET_EXCEEDED, with the ready flag set when any landed, so the next bridge
    decode finishes the last committed token (the prefill bridges' header comments state it); at the
    cap the existing cap behaviour applies. Every other GPU verb (decode, the batch,
    prefill, the finish, reset, release, schema binding) takes a pooled sequence as it takes a
    dedicated one, and its reset and release touch only its own pages.
  • Shared GPU prefixes. sslm_gpu_prefix_begin, sslm_gpu_prefix_set_schema,
    sslm_gpu_prefix_prefill, sslm_gpu_prefix_freeze and sslm_gpu_prefix_release build budgeted
    prefixes in a pool, base-only. sslm_gpu_prefix_begin_from builds a prefix on a frozen parent,
    sharing its full pages and copying its partial last page, so prefixes nest (a world, then a persona
    on the world). sslm_gpu_seq_adopt_prefix adopts a frozen prefix onto a pooled sequence the same
    way, with the CPU adopt's field and schema rules; every check runs before any state change. One
    failure comes after them: if uploading the prefix's partial last page fails, the adopt returns
    the fault's status and leaves the sequence reset, its earlier history gone. A shared page is
    freed when the prefix and every holder that maps it have let go of it.
  • Save and restore, SLM6. A pooled sequence's sslm_gpu_seq_save writes SLM6: the SLM5
    header and tail, then its mode, budget and origin, its residual, and its rows from position 0 to
    its live length (an adopted prefix's span included), so its size follows its live length.
    sslm_gpu_seq_restore_into_pool restores an SLM6 blob into a fresh pooled sequence with its
    budget and origin, and takes an optional live prefix handle: when the handle matches, the restored
    sequence maps the prefix's pages again, and otherwise it restores privately and still succeeds.
    Dedicated sequences keep writing SLM5, and sslm_gpu_seq_restore keeps reading SLM5 and SLM4.
    Each restore verb refuses the other's blobs with SSLM_GPU_HOLDER_MODE_MISMATCH, changing
    nothing. In the software-GPU runs, a GPU SLM6 blob's K/V section equals the CPU SSB6 blob's of
    the same state byte for byte; the two remain separate formats, and neither backend restores the
    other's blobs.
  • Statuses. Six SslmGpuStatus values are appended, ordinals 24 to 29:
    SSLM_GPU_KV_BUDGET_EXCEEDED, SSLM_GPU_KV_POOL_EXHAUSTED, SSLM_GPU_KV_POOL_TOO_LARGE (a pool
    above 2^32 bytes, the reach of the shaders' 32-bit offsets: 43,690 pages at Qwen2.5-0.5B and 18,724
    at Qwen2.5-1.5B, from the page rule), SSLM_GPU_HOLDER_MODE_MISMATCH, SSLM_GPU_PREFIX_REJECTED
    and SSLM_GPU_PREFIX_SCHEMA_MISMATCH. No existing ordinal moves.
  • Work in flight, stated. gpu_1p0.h now states two promises. Safety: the library never resets
    a device's command allocator while work submitted on it has not been observed complete, whatever
    path reached the reset. A clean status: a guarded verb that would do so returns SSLM_BUSY and
    changes nothing, or SSLM_DEVICE_LOST once a submission is stranded, never a Busy a retrying host
    would spin on. The header lists, for every verb, which device it resets, submits or waits on, and
    the order of each verb's refusals. Both hold for a host that serializes every call that submits on
    a device, as before; the list of those calls is now complete (see Compatibility).

gpu_1p0.h adds 12 verbs and 2 handle types (SslmGpuKvPoolHandle, SslmGpuPrefixHandle), plus
two test-only verbs (sslm_gpu_test_only_peek_pool_mirror_bytes,
sslm_gpu_test_only_wait_in_flight) that a host does not call. docs/api.md (Paged KV memory on
the GPU
) is the summary, and the header is the contract.

Fixes

  • A barrier race in the shipped shaders. Four steps run more than once in one dispatch: the
    residual candidate of the attention and MLP residual sites, the bias reconcile and the funnel
    maximum. Thread 0 reset a shared flag or maximum slot for the second run without a barrier after
    the first run's reads, so another thread could read the reset value and reach a barrier the others
    did not. The fix adds the barrier. It was found by running the GPU path on a software Vulkan device
    (lavapipe), where the race fired on every run: 16 GPU-against-CPU mismatches in superslm_tests
    went away with it. In earlier releases' certification runs on the two certified GPUs, GPU outputs
    were byte-identical to the CPU's, so the race had not shown there. No output is meant to change.
    On the RTX 2080 SUPER, superslm_tests with its GPU cells passes with the fix (see
    Verification); the AMD card has not run it.
  • No-budget CPU save speed. 1.12.0's no-budget sslm_seq_save copied the block one page run at
    a time, each copy too small for the C runtime to use non-temporal stores, so every destination
    line was first read into the cache, and the save took 1.14 to 2.00 times as long as 1.11.0's. A
    block of 16 MiB or more (24 MiB at Qwen2.5-0.5B, cap 4,096; 448 MiB at Qwen2.5-1.5B, cap 32,768)
    is now written with streaming stores on x64; a smaller block, and every non-x64 target, keeps
    1.12.0's copy. The bytes are unchanged: blobs equal 1.11.0's and 1.12.0's at 9 lengths, and the
    streaming path is checked against the pinned 1.11.0 blobs at every destination offset from 0 to
    63 (cell 9.3s). On the release machine the save is faster than 1.12.0's but a full-length save
    still reads slower than 1.11.0's (see Speed).
  • Two decodes racing on one GPU sequence. Two threads decoding the same sequence handle at once
    is not a supported use, but...
Read more

SuperSLM 1.12.0

Choose a tag to compare

@dansupergameprogrammer dansupergameprogrammer released this 12 Oct 19:25

SuperSLM 1.12.0

Released 2026-10-12.

SuperSLM 1.12.0 stores the CPU backend's K/V memory in pages. A sequence can declare a budget, the
positions it will write itself, and reserve only the pages that budget needs instead of a whole
context-length block. Sequences that adopt a shared prompt prefix can share the prefix's pages
instead of copying them, and a budget sequence's save blob holds its live rows, the adopted prefix's
span included, so its size follows its live length instead of the context length.

Output does not change. Tokens and K/V values are bit-identical to 1.11.0 on every path, as read on
Windows x64 with MSVC; the paged-KV suites were run there only (see Verification). A host that
calls only the 1.11.0 verbs keeps 1.11.0's results, admission counts and save blobs, apart from the
changes listed under Compatibility. Its speed is not unchanged: saving a no-budget sequence takes
1.14 to 2.00 times as long as in 1.11.0 on the release code, at every length measured (24 of 24
readings across four captures; readings, not resolved changes; see Speed), and a later release is
planned to fix that.
The memory savings come only from the new budget-mode verbs. The GPU API is unchanged (gpu_1p0.h
changed in comment text only), and no model artifact needs reconversion (the loader and the
converter tools are unchanged since 1.11.0).

What changed

  • Pages. A KV pool is now a pool of pages of B positions (sslm_kv_page_positions). B
    follows from the model's head dimension, context cap and K/V element width, and the artifact's
    stored block size does not select it: 64 for Qwen2.5-0.5B at cap 4,096 and for Qwen2.5-1.5B at
    cap 32,768, the two sizes measured below. docs/api.md states the rule. A sequence or prefix
    reaches its K/V through a page table. The old block pool (sslm_kv_pool_create) is the same
    number of bytes at a cap B divides, cut into block_count · ceil(cap / B) pages.
  • Budget-mode sequences and prefixes. sslm_seq_create_budgeted and
    sslm_prefix_begin_budgeted take a budget, 1 ≤ budget ≤ cap, and reserve
    R(budget) = min(ceil(budget / B) + 1, ceil(cap / B)) pages up front (sslm_kv_pages_for_budget).
    Prefill and decode then take pages only from that reservation, so they never fail for lack of pool
    pages below the holder's limit, min(origin + budget, cap). A write past the limit returns the new
    status SSLM_KV_BUDGET_EXCEEDED (ordinal 29), with no row written. At the cap the existing
    SSLM_CONTEXT_CAP_EXCEEDED fires instead.
  • Shared prefixes. A budget sequence that adopts a frozen prefix maps the prefix's full pages
    and copies at most one page. sslm_prefix_begin_from builds a budget-mode prefix on top of a frozen
    parent the same way, so prefixes can nest (a world, then a persona on the world). A shared page is
    freed when the prefix and every holder that maps it have let go of it. Sequences made by
    sslm_seq_create still copy, as before. Sharing, residency by budget and a cheap high cap hold at
    a cap of more than one page; at a cap that fits in one page a holder is a single page, so adopt
    copies it and a budget reserves all of it.
  • Page pools and diagnostics. sslm_kv_page_pool_create sizes a pool in pages
    (sslm_kv_page_size, sslm_kv_page_pool_overhead_size). sslm_kv_pool_stats reports a pool's
    free and shared pages, and sslm_seq_kv_stats a sequence's mode, origin, budget, limit and pages.
  • Contiguous pool option (CPU). sslm_kv_pool_contiguous_size and
    sslm_kv_pool_create_contiguous build a pool whose page is the whole context cap, one page per
    block: 1.11.0's contiguous layout, through the same page module. Every verb works on it. A holder
    in it takes one page, so adopt and sslm_prefix_begin_from copy and share no page, and saves are
    the same in every pool kind, so a blob restores across pool kinds. It gives up what a page pool
    gives a budget holder, which is sharing a prefix's pages and reserving by budget, and it is
    outside the claims for reset and adopt cost, shared prefixes and residency by budget. It is the
    reference that decode speed is measured against (see Speed).
  • Save and restore. A budget sequence saves a new format, SSB6: the SSB5 header plus its mode,
    budget and origin, then its live rows, from position 0 to its live length, the span of an adopted
    prefix included, so its size follows its live length. A sequence that adopted a 1,200-token prefix
    and generated 512 tokens saves 1,712 rows. Cell 7.11 asserts the exact size at live lengths of 0,
    1, one page and 1,000. A no-budget sequence keeps saving SSB5, byte-equal to 1.11.0's blob, so 1.9.0 to 1.11.0 still read
    every save a 1.11.0-style host makes. Restore keeps a budget sequence's budget, origin and limit.
    sslm_seq_restore_shared takes a live prefix handle and re-shares its pages when the prefix's
    bytes match the blob's exactly; without a matching handle it restores privately and still
    succeeds.
  • Softmax fast path. On the AVX2 and AVX-512 tiers, SoftmaxRowQ15's fast path admits rows up
    to width 32,767 (it was 2^14). Every probability and the returned bool stay bit-identical to the
    1.9.0 body, pinned by the softmax golden stream (docs/attention-rowsites/s4/golden.txt, 882,516
    values, regenerated from a build of the 1.9.0 tag) and by cell 6.9. No speed figure is claimed for
    it.

docs/api.md (Paged KV memory) is the full contract: every verb, the SSB6 layout, the restore
rules, the thread rules and the limits.

Memory, graded by admission

The memory figures are graded by what a pool admits, never by the library's own page counts. The
example cohort is an assumption, not a measured workload: a 1,000-token world prefix, 10 persona
prefixes of 200 tokens built on it with sslm_prefix_begin_from, the world handle then released, and
50 budget sequences of 512 tokens each (5 per persona), each run to its budget. At Qwen2.5-0.5B with
cap 4,096 a page is 384 KiB (64 positions), and a 1.11.0 block is 64 pages (24 MiB).

  • A page pool of 505 pages admits the whole cohort, and a pool of 504 pages refuses exactly
    one create, the 50th sequence, with SSLM_KV_POOL_EXHAUSTED. 505 pages is 189.4 MiB. 1.11.0's
    layout holds the same cohort in 60 whole blocks, 1,440 MiB (arithmetic from the 24 MiB block, 60
    times 24; no 1.11.0 run measured it). The admission and the refusal are confirmed on the
    release machine (cell 10.4, real Qwen2.5-0.5B artifact).
  • A pool of 1.11.0-style blocks (sslm_kv_pool_create, with the same bytes and admission at this
    cap as in 1.11.0) in the same 189.4 MiB
    admits 7 sequences and refuses the 8th, because an eighth block needs 192 MiB. Cell 10.4 runs
    this contrast leg on the 1.12.0 build with the 1.11.0 verb; it is not a run of the 1.11.0 binary.
  • After every sequence is saved mid-budget and at its limit, the world and personas rebuilt in a
    fresh pool and the world released, the 50 sequences restored with their persona handles fit in
    505 pages, and 504 refuses exactly the 50th, for both saves; confirmed on the release machine
    (cell 10.6, real Qwen2.5-0.5B artifact).
  • Restored without handles, the same blobs need 1,405 pages (1,404 refuses the 50th), until
    each sequence resets and re-adopts its persona; after that the pool admits 100 more budget-512
    sequences, for both saves; confirmed on the release machine (cell 10.8, real Qwen2.5-0.5B artifact).

These numbers hold in budget mode only, and for this artifact's page size. Another model or cap has
another page size and its own figures. They are confirmed on the real artifact on the release machine.

Speed

This release makes speed claims for Windows x64, CPU only, at the ordinary background load of the
release machine: an AMD Ryzen 9 3950X (16 cores, 32 logical processors) with AVX2 and without
AVX-512, running Windows 10.0.19045, where the dispatched kernel tier is AVX2. A process watch ran
beside every timing row and listed no process other than the row's own, and the shared-prefix run
read 2% and 3% CPU load before and after. Decode speed is claimed for single-conversation
generation only: one sequence, decoding one token at a time. Nothing is claimed under load, on the
GPU, or for batched decode.

  • Decode, one conversation. Attention reads K/V one page run at a time. Single-conversation
    decode is no more than 5% slower than the contiguous cache at both targets, to the 0.5B cap of
    4,096 and to width 18,682 at 1.5B, with 7.9b reported (below). The contiguous
    cache is the contiguous pool option of the same build. The comparison is not against 1.11.0, and
    no measurement ties the option's speed to an older release. Cell 7.9 restores one saved sequence into each
    of two arms on one artifact: the paged arm, in a page pool at the model's page size with its pages
    in scattered order, and the contiguous arm, in a contiguous-option pool. It decodes the same
    tokens in both, alternating one token per arm in an order drawn at random for each pair, and
    times each token alone. A process of 104 pairs gives a median slowdown, paged time over
    contiguous time minus one. A row is five processes on one boot, and it passes when the largest of
    its five process medians is at most 5.0%. The scattered order is no more step-regular than a
    random one; it is not claimed to be a worst case. Cell 7.9 times whole-token sslm_decode_step
    calls with no host parallel-for hook, on the AVX2 tier. Decode with the hook, decode sliced
    across calls, and decode on the AVX-512 or SSE2 tier are not measured.

    All six rows pass. Each row's figures, in percent slower than the contiguous cache:

    Model, cap Restored at Timed widths Largest process median SD of the five medians Mean of the five, ± 95% half-width
    Qwen2.5-0.5B, 4,096 1,200 1,204 to 1,307 +1.001 0.067 0.941 ± 0.083
    Qwe...
Read more

SuperSLM 1.11.0

Choose a tag to compare

@dansupergameprogrammer dansupergameprogrammer released this 30 Sep 21:57

SuperSLM 1.11.0

Released 2026-09-30.

SuperSLM 1.11.0 lets a host's parallel-for hook split decode, and a one-token prefill, across
threads. It is opt-in: nothing changes unless the host sets the new bit. Every output is
bit-identical with and without it, at any task count: the same tokens, the same save blobs and the
same axis digest. There is no change to any ABI struct, status ordinal or save format, and no model
artifact needs reconversion.

What changed

1.10.0's parallel-for hook (sslm_parallel_for) was read only by the logits step. In 1.11.0 a host
can set bit 1 of sslm_parallel_for.reserved, the new SSLM_PARALLEL_FOR_MATVEC, to have the CPU
backend also split its one-row projections through the hook's run. These are each decode layer's
q, k + v, o, gate + up and down projections, and the same projections in a prefill call that admits
exactly one token.

  • When a group splits. Output rows are split when the split gives a group two or more tasks.
    That needs max_tasks of at least 2, more than 64 rows, and at least twice 256 KiB of weight bytes.
    The 256 KiB constant caps the task count by the group's weight bytes, so the last task can be
    smaller. docs/api.md gives the exact rule.
  • Why output is unchanged. Every row is still one exact integer sum computed on one thread.
  • Allocation. The split adds no heap allocation.
  • Setters. Both setters accept bit 1 and still refuse every other bit, so a host that sets
    bit 1 against 1.10.0 is refused rather than silently ignored. The GPU setter accepts the bit and
    ignores it.
  • A hook that fails inside a decode layer. If run breaks exactly-once inside a decode layer,
    the call fails with SSLM_INVALID_ARGUMENT. The sequence rests at the start of that layer, with
    that layer's saturation counts put back, and the next call resumes there.
  • Behaviour change for every host, with or without the bit. A prefill or prefix-prefill call
    that fails before admitting a token now leaves the sequence's five saturation counts as they were
    before the call. In 1.10.0 it kept counts that no K/V entry backed.
  • C++ headers. RunLayerLoop (both overloads) and RunLayerLoopChunkBatched gain threaded
    overloads that take a trailing GemmThreading, for the engine's own ABI layer. They are not in
    the SUPERSLM_API export slot, so a shared engine library exports the same symbols as 1.10.0.
    The existing symbols stay and run serially.

Measured speed

Measured on a Zen 2 desktop (Ryzen 9 3950X, Windows, MSVC Release) with the Qwen2.5-0.5B model
(24 layers) and a std::thread pool as the hook. Each figure is the median of 15 interleaved
off/on pairs.

max_tasks decode tok/s, context 16 decode tok/s, context 300
2 25.5 → 33.8 (1.32x) 24.4 → 32.0 (1.31x)
4 26.0 → 39.5 (1.52x) 25.0 → 37.1 (1.48x)
8 24.9 → 35.2 (1.41x) 25.0 → 35.4 (1.42x)
  • One-token prompt at 4 tasks: 33.07 → 20.31 ms per token (1.63x).
  • Task-size constant: a sweep of 64 KiB, 256 KiB and 1 MiB at 4 tasks kept 256 KiB.
  • Under memory contention (a 4-thread, 4 GB streaming-read competitor running alongside):
    decode at context 300 goes from 12.0 to 21.8 tok/s (1.82x). The competitor's own bandwidth falls
    more while threaded decode runs: to 18.1 GB/s, from 29.7 GB/s alone, against 23.6 GB/s beside
    serial decode.

Threaded decode is therefore faster, but it takes more memory bandwidth from other memory-heavy
work on the same machine. A game should measure it in its own frame before turning it on. The
SuperSLM-Unreal plugin ships with it off until it has been measured in game.

Compatibility and required action

None for a host that does not set the bit, with one exception: the saturation-count behaviour
change above, which applies to every host. The C ABI verbs, struct layouts, status ordinals and
save formats are unchanged. A host that wants threaded decode must set bit 1 and pass a
sslm_parallel_for whose run calls each task exactly once. A host can test for the feature at
compile time by checking whether SSLM_PARALLEL_FOR_MATVEC is defined.

Verification

  • Bit-identity:
    • The axis digest's GLOBAL hash is f740f833…, unchanged from 1.10.0, on all ten Linux legs:
      auto and forced scalar/SSE2/AVX2/AVX-512, each on GCC 13.3 and Clang 18.1.
    • On the real Qwen2.5-0.5B, 64 greedy tokens and the final save blob are byte-equal to a run with
      no hook, at max_tasks 1, 2, 3, 4 and 8. That run makes 97 run calls per decode token at 2
      tasks and above.
    • Against a v1.9.0 library, a threaded save blob equals v1.9.0's blob, and each library re-saves
      the other's blob byte for byte (Linux and Windows).
  • Mutants: 23 of 23 killed.
  • Found by the release machine's build.bat and fixed before release: a default
    BUILD_TESTING=OFF build failed, because a new test target was built unconditionally
    (e46c0f1). The three new overloads were also in the SUPERSLM_API export slot, which the
    export-slot check (X2) caught; they were taken out of it. A floating-point scan test had been
    moved to 18 objects, but the archive it reads is a fixed, pre-built one that still has 17; it
    was put back.
  • Code review: SHIP on a Zen 2 desktop (Windows, MSVC), after a confirmation re-check of four
    fixes. Windows suites: 133,931 checks, 0 failures; the forced SSE2 and AVX2 suites are green.
  • GitHub CI: green on the merged head d31c65f, 78 of 78 checks.
  • Release machine (Zen 2 desktop, Windows, MSVC 19.33, RTX 2080 SUPER; AVX2, no AVX-512):
    • At d200cf9, CMake Release with the GPU on, built fresh, with this machine's decode-threading
      fixtures: superslm_tests 133,931 checks, 0 failures, including the decode-threading cells
      (2,152 checks). The commits after it change build files, export attributes, tests and notes,
      not engine arithmetic.
    • At 445e206, build.bat runs to completion. Every gate passes: the CPU-only default build,
      the export-slot check (63 of 63), the floating-point scan suite (231 passed), and its own
      superslm_tests step (129,093 checks, 0 failures).
    • The release gate's tests pass, 4 of 4, after its two compiled diagnostics were rebuilt from
      445e206 and re-pinned.
    • The decode figures above were measured on this machine.

Files changed

  • src/forward/parallel_split.cpp, src/forward/parallel_split.h: the one-row split runner.
  • src/forward/forward_sites.cpp, include/superslm/forward_sites.h: the split call sites and the
    threaded overloads.
  • include/superslm/parallel_for.h, src/sslm_abi.cpp, include/superslm/sslm_abi_functions.inc,
    src/gpu/gpu_1p0.cpp, include/superslm/gpu_1p0.h: bit 1 in both setters.
  • include/superslm/checked_chain_funnel.h, include/superslm/gpu_layer_loop_guards.def: comments
    and line citations.
  • tools/gen_decode_threading_fixture.py and the decode-threading tests: the fixtures and cells.
  • tools/build_adapter_dump.bat and 26 other MSVC build lists: they now include
    parallel_split.cpp.
  • CMakeLists.txt, tests/install-consumer/CMakeLists.txt, README.md, CHANGELOG.md, docs/api.md
    and this note: version 1.11.0.
  • tests/data/t2704_release_manifest.json, tools/test_t2704_release_gate.py: the release gate's
    compiled diagnostics, re-pinned to 445e206.

SuperSLM 1.10.0

Choose a tag to compare

@dansupergameprogrammer dansupergameprogrammer released this 30 Sep 16:16

SuperSLM 1.10.0

Released 2026-09-30.

SuperSLM 1.10.0 is a CPU speed release. Every output is bit-identical to 1.9.0: the same tokens,
the same save blobs, and the same axis digest on every tier and compiler. There is no change to the
C ABI, to any status or SslmGpuStatus ordinal, or to the save format, and no model artifact needs
reconversion. The GPU path is unchanged.

What changed

All of the speed-ups below apply only on the AVX2 and AVX-512 tiers. The scalar and SSE2 tiers run
the 1.9.0 code. Each fast path has a guard that admits only inputs where it is provably exact;
everything else takes the 1.9.0 code unchanged.

  • Tiled prefill GEMM. A prefill GEMM of 8 or more tokens runs a register-tiled kernel over
    weights packed per call, instead of one dot product per output cell. At 128 tokens, one layer of
    batched prefill runs about 1.5–1.6x faster (engine level, on a cloud host). On a Zen 2 desktop
    (AVX2), one measured operator (the 1.5B model's gate/up GEMM at 32 rows) runs 2.59x faster.
    Decode and prefills under 8 tokens are unchanged.

  • Hardware SHA-256 for the load-time integrity hash. When the CPU has the x86 SHA extensions,
    the hash runs on them. On a Zen 2 desktop, hashing the 510 MB Qwen2.5-0.5B model takes 0.31 s
    instead of 3.25 s. The digests are unchanged, and the portable hash remains the fallback.

  • Attention and per-row sites, one row at a time:

    • The sigmoid, residual-landing and norm sites use 256-entry per-row tables on rows of 512 or
      more.
    • Probability-times-value runs on 16-bit multiply-add.
    • The requantization funnel runs in 64-bit integer lanes (RequantRowWide).
    • The softmax row has a guarded fast path.
    • On QK-norm models (the Qwen3 path), the Q31 attention score is computed for all of a query
      head's keys in one call (QkQ31ScoreRow). On a Zen 2 desktop that costs 0.59 ms per token
      instead of 10.26 ms at 128 tokens, and 3.63 ms instead of 94.07 ms at 1,024 tokens.

    At Qwen2.5-0.5B depth (engine level, cloud host, AVX2), these save about 4.3 / 7.2 / 11.0 ms per
    prompt token at 128 / 512 / 1,024 tokens, and about 7.8 ms per decode token at context 300.
    CHANGELOG.md gives the per-step figures.

These are engine figures, not end-to-end figures for any consumer. How much faster a game gets
depends on how it drives prefill: a consumer that prefills one token per call gets the attention
and per-row savings but not the tiled GEMM's.

MSVC and clang-cl

On MSVC and clang-cl builds, the AVX-512 tier keeps the 1.9.0 code for the tiled GEMM and for the
attention and per-row sites until those paths have executed on such a build. Two switches turn
them on: SUPERSLM_TILED_AVX512_MSVC=1 and SUPERSLM_SITES_AVX512_MSVC=1. The AVX2 tier runs the
new code on every compiler.

Internal declarations

Five internal C++ declarations join the installed headers:

  • SitesKernel, SelectSitesKernel and DispatchSitesKernel in matmul.h;
  • RequantRowWide in intmath.h;
  • QkQ31ScoreRow in forward_sites.h.

They follow the pattern those headers already use (GemmPath, RequantTokenCodeWide,
QkQ31ScoreForTier). None of them is exported, and none is part of the C API.

Compatibility and required action

None. Every C ABI verb, struct, status ordinal and save format is unchanged. A blob saved by 1.10.0
is the same SSB5 blob 1.9.0 writes, and the two releases read each other's blobs. The FP-free
scan now accepts the three SHA-256 instructions (sha256rnds2, sha256msg1 and sha256msg2),
which are integer-only. The SHA-1 instructions stay rejected.

Verification

  • Bit-identity:
    • The axis digest's GLOBAL hash is identical on all ten Linux legs: auto, plus forced scalar,
      SSE2, AVX2 and AVX-512, each on GCC 13.3 and Clang 18.1.
    • Each slice's golden pins are generated from the v1.9.0 tag.
    • Save blobs and decoded tokens match 1.9.0 in each slice's blob protocol.
  • GitHub CI: green on the merged head 1cfe368, 78 of 78 checks. The forced AVX-512 suite ran
    on a runner that has AVX-512: 121,375 checks, 0 failures, with the AVX-512 kernels confirmed
    active.
  • Code reviews on a Zen 2 desktop (Windows, MSVC) of the tiled GEMM, the SHA-256 path and the
    attention and per-row sites: no bugs found. The review's two findings on the attention sites
    are fixed in this release.
  • Release machine (Zen 2 desktop, Windows, MSVC 19.33, RTX 2080 SUPER; AVX2, no AVX-512), at
    e4d263e:
    • CMake Release with the GPU on, built fresh: superslm_tests 131,772 checks, 0 failures,
      including the tiled GEMM cells (247 checks) and the attention and per-row site cells (91,714
      checks).
    • build.bat runs to completion: every gate passes, including its own superslm_tests step
      (126,934 checks, 0 failures) and the ABI header inventory.
    • The release gate's tests (tools/test_t2704_release_gate.py) pass, 4 of 4, after its two
      compiled diagnostics were rebuilt from e4d263e and re-pinned. CMakeLists.txt changed in
      this release, and the gate counts that as a changed build input. The commits after e4d263e
      change only that pin and this note.

Files changed

  • src/matmul.cpp, include/superslm/matmul.h: the tiled GEMM, prob·V and the sites kernel
    selection.
  • src/sha256.cpp, include/superslm/sha256.h: the hardware SHA-256 path.
  • src/intmath.cpp, src/forward/forward_sites.cpp, src/forward/checked_chain_funnel.cpp,
    include/superslm/intmath.h, include/superslm/forward_sites.h: the attention and per-row sites.
  • include/superslm/gpu_layer_loop_guards.def: line citations moved with forward_sites.cpp
    (comments only). src/bad_alloc_wrap.h: the tiled path's packing-buffer allocation.
  • tests/ci/check_fp_free_scan.py: the three SHA-256 instructions.
  • tools/ci/branch_coverage_floors.json: the src/matmul.cpp and src/intmath.cpp floors are
    pinned from a CI run on a runner without AVX-512, where the AVX-512 bodies cannot execute.
  • CMakeLists.txt, tests/install-consumer/CMakeLists.txt, README.md, CHANGELOG.md and this
    note: version 1.10.0.
  • tests/data/t2704_release_manifest.json, tools/test_t2704_release_gate.py: the release gate's
    compiled diagnostics, re-pinned to e4d263e.

SuperSLM 1.9.0

Choose a tag to compare

@dansupergameprogrammer dansupergameprogrammer released this 25 Sep 21:17

SuperSLM 1.9.0

Released 2026-09-25.

SuperSLM 1.9.0 changes the CPU save format so a saved sequence keeps its four per-site saturation
counts, refuses at map time an artifact whose damped-greedy scale no decode can use, and adds C-ABI
tests for schema-constrained and damped-greedy decoding. No token, status ordinal, ABI surface,
SslmGpuStatus ordinal or measured figure changes, and no model-artifact reconversion is required.

What changed

SequenceLayerState carries a saturation total, kv_saturation_count, and the four per-site counts
it sums: kv_landing_saturation_count, k_channel_landing_saturation_count,
rope_q_saturation_count and rope_k_saturation_count. 1.8.1 made the four travel with the total
through prefill and prefix adoption, and stated one limit it left: sslm_seq_save recorded only the
total, so a sequence restored with sslm_seq_restore had its total and per-site counts of 0.

  • Save. sslm_seq_save writes a new format, SSB5: the SSB4 layout with the four per-site
    counts appended to its fixed header (little-endian 64-bit each, in the order above, at byte offsets
    124, 132, 140 and 148). Every field SSB4 has keeps its offset. The blob grows by 32 bytes, and
    sslm_seq_state_size grows by the same 32 bytes, so it stays an upper bound on what
    sslm_seq_save writes.
  • Restore. sslm_seq_restore reads the four counts from an SSB5 blob, so the restored
    sequence's per-site counts are the ones it was saved with and sum to its total.

The counters are written beside the arithmetic, never read by it, so no token changes.

Blobs saved by earlier releases

sslm_seq_restore still accepts SSB4 (1.2.1 to 1.8.1), SSB3 and SSB2 blobs, under the same
rules as before. They record only the saturation total, so a sequence restored from one has that
total and per-site counts of 0; its per-site counts do not sum to its total until it is reset. The
blob never recorded which site produced each saturation, so any non-zero split would be invented,
and restoring the total as 0 would discard a value the blob does record. Restore does not require an
SSB5 blob's four counts to sum to its total: a sequence restored from an older blob and saved again
carries a per-site sum below its total, and it restores as saved.

The GPU path

Checked at source and by execution; unchanged. GPU prefill and decode both pack the four counts into
the device's sequence state and read them back with the total, and the GPU save format (SLM5, and
the legacy SLM4 it still reads) already carries all four. The GPU has no prefix adoption.

Damped greedy: an unusable scale is refused at map

sslm_model_map already refused, with SSLM_ARTIFACT_REJECTED, an artifact whose DGC1 damped-greedy
scale falls outside the i-exp domain. Its check derived the scale's i-exp constants and checked the
softmax row width, but not the i-exp peak that sslm_decode_step_v2 requires. A scale such as
(m = 0, e = 0) derives constants of 0, 0 and 0. Before 1.9.0 it passed that check: the artifact
mapped, sslm_decode_params_init offered damped greedy, and every damped-greedy decode then returned
SSLM_INVALID_ARGUMENT. Map time and decode now apply the same peak check, so such an artifact is
refused by sslm_model_map with SSLM_ARTIFACT_REJECTED. The scale tools/convert_model.py writes
passes both, and a converted model maps and decodes as before; this path is reachable only from a
hand-built or corrupted artifact.

New C-ABI tests

tests/test_slm19x_schema_damped_greedy.cpp adds C-ABI cells to superslm_tests for
include/superslm/schema_masks.h and the three src/damped_greedy_*.cpp files. They cover
schema-bound prefill and decode, damped-greedy availability, parameter domains, anti-LM and top-k
behaviour, and workspace and save/restore equivalence.

Compatibility and required action

A blob saved by 1.9.0 is SSB5 and is not readable by 1.8.1 or earlier, which reject its magic with
SSLM_INVALID_ARGUMENT. A host that keeps saved sequences across a downgrade must keep them in the
older format. A consumer that reads or checks the save blob's magic must accept SSB5 where it
accepted SSB4; every field SSB4 has keeps its offset, so a reader of an existing SSB4 field
needs only the magic check widened. An artifact whose DGC1 scale fails the i-exp peak check,
which 1.8.1 mapped, is now refused at map with SSLM_ARTIFACT_REJECTED; no converter output has
that shape. Every C ABI verb, struct and status ordinal is unchanged. The per-site counts are
internal SequenceLayerState fields, not part of the public ABI.

Verification

Each binary was built fresh from a git archive of 66e5828 into an empty directory. Commits
after it change comments and documentation, add test cells, and re-pin the release gate's
compiled diagnostics; no engine code changes.

  • Red first, as the test authors read it. At 1.8.1 the save/restore census cells fail on both
    models, 6 checks on Qwen2.5-1.5B-Instruct and 12 on Qwen3-Embedding-0.6B: the restored sequence
    carries its total with every per-site count at 0. At 1.8.1 the map-time DGC1 cell fails 1
    check: a (0, 0) scale maps.
  • Windows, MSVC, CMake Release with the GPU on: superslm_tests 35,631 checks, 0 failures; at
    4f2183b, which adds one decode-side damped-greedy cell, 35,655 checks, 0 failures. The
    damped-greedy suite's Phase D cells at 4f2183b on build.bat's fixture: 18/0, 251/0 (the
    adapter-composition cell skips, as above), 29/0 and 1,805/0.
  • build.bat: every gate up to its damped-greedy Phase D acceptance run passes. That run stops as
    it did before this release, on 251 checks, 0 failures and 1 skip: its adapter-composition cell
    needs a runtime adapter whose base model carries damped-greedy constants, and no such pair exists
    on the release machine. Run to completion with that one gate recorded instead of stopping,
    build.bat passes every later gate, including the T-2138 ABI suite, the T-2899 cell build and
    its own superslm_tests step (35,625 checks, 0 failures).
  • Save/restore and GPU census suite (tests/te441-sat-restore-red-suite), run on
    Qwen2.5-1.5B-Instruct and Qwen3-Embedding-0.6B on an RTX 2080 SUPER. Every cell passes: CPU
    save/restore, the pinned 1.8.1 CPU blob restoring with per-site counts of 0, GPU prefill,
    decode and save/restore, and the pinned 1.8.1 GPU blobs restoring their per-site counts. A GPU
    blob with its per-site counts zeroed fails the cells, as it must.
  • Linux, GCC 13.3 Release: superslm_tests 25,274 checks, 0 failures; ASan and UBSan 25,274
    checks, 0 failures.
  • pytest tests/ci on a fresh full-history clone: 916 passed. tools/test_t2704_release_gate.py:
    4 passed, with the gate's compiled diagnostics rebuilt from 4e8d3de.

Files changed

  • src/sslm_abi.cpp -- SSB5 written by sslm_seq_save and read by sslm_seq_restore;
    sslm_seq_state_size's fixed header grows from 128 to 160 bytes.
  • include/superslm/sslm_abi_functions_g5_comparable.inc -- the save/restore contract names SSB5
    and what an older blob restores as.
  • include/superslm/sslm_abi.h, docs/api.md -- the save format descriptions; docs/api.md also
    states the map-time DGC1 refusal.
  • src/damped_greedy_phaseD.cpp, include/superslm/sslm_phaseD.h, src/sslm_abi.cpp --
    DampedGreedyPeakInDomain, the one i-exp peak check the map-time DGC1 check and
    sslm_decode_step_v2 both apply.
  • tests/test_slm19x_schema_damped_greedy.cpp, tests/test_main.cpp, CMakeLists.txt,
    build.bat -- the new C-ABI cells, wired into superslm_tests.
  • tests/te441-sat-restore-red-suite/ -- the save/restore and GPU saturation-census cells.
  • tests/test_slm18x_saturation_census.cpp and the t2138, t2199 and t2899 suites -- cells for
    SSB5 save/restore and the SSB5 layout.
  • tests/data/t2704_release_manifest.json, tools/test_t2704_release_gate.py -- the release gate's
    compiled diagnostics re-pinned to binaries rebuilt from the 1.9.0 sources.
  • tools/t2693_prefix_clone.cpp, docs/fused_k_numeric_spec.md -- a comment naming the retired
    save format.
  • CMakeLists.txt, tests/install-consumer/CMakeLists.txt, README.md, CHANGELOG.md, this note
    -- version 1.9.0.

SuperSLM 1.8.1

Choose a tag to compare

@dansupergameprogrammer dansupergameprogrammer released this 25 Sep 16:55
e0d56fd

SuperSLM 1.8.1

Released 2026-09-25.

SuperSLM 1.8.1 is a single diagnostic fix on the CPU path: the four per-site saturation counts
travel with the saturation total they sum. No token, status, ABI surface, SslmGpuStatus ordinal,
save format or measured figure changes from 1.8.0, and no model-artifact reconversion is required.

What changed

SequenceLayerState carries a saturation total, kv_saturation_count, and the four per-site counts
it sums: kv_landing_saturation_count, k_channel_landing_saturation_count,
rope_q_saturation_count and rope_k_saturation_count. RunLayerLoop, the path decode uses,
already kept the four in step with the total. Two CPU ABI paths did not:

  • Prefix adoption. sslm_seq_adopt_prefix copied the prefix's total but left the adopting
    sequence's four per-site counts at whatever the sequence held before. They now come from the
    prefix, like the total.
  • Prefill. sslm_prefill and sslm_prefix_prefill run through PrefillWholeTokens and
    RunLayerLoopChunkBatched, which grew the total without being handed the four per-site
    out-parameters, so a prefilled sequence's per-site counts no longer summed to its total. They are
    now passed, and the counts are filled.

Neither path changes a token: the counters are written beside the arithmetic, never read by it.

Compatibility and required action

None. Every C ABI verb, struct and status ordinal is unchanged from 1.8.0. The per-site counts are
internal SequenceLayerState fields, not part of the public ABI.

One limit is unchanged and stated here so it is not mistaken for part of this fix: sslm_seq_save
records only the saturation total, so a sequence restored with sslm_seq_restore has its total and
per-site counts of 0, so after a restore the per-site counts no longer sum to the total. Carrying them would change the save format, which this release does not do.

Verification

  • Red first: two cells, one for adoption and one for prefill (both sslm_prefill and
    sslm_prefix_prefill), failed 6 checks against 1.8.0's engine for the reasons above. The prefill
    prompt saturates 5 times on the committed fixture, so the prefill cell cannot pass on zeros.
  • Linux, GCC 13 Release: superslm_tests 0 failures; all 15 ctest entries pass; ASan and UBSan 0
    failures; Clang 18 forced SSE2, AVX2 and AVX-512 tiers 0 failures each.
  • Windows, MSVC: CMake Release superslm_tests 34,759 checks and build.bat's superslm_tests
    step 34,753 checks, 0 failures each; the four Windows CI ctest cells pass. With the engine fix
    reverted, 6 failures, all in the new cells.
  • GPU, TE-425 suite on the dev-box device: every one of 119 jobs gave the same verdict as the
    1.8.0 exit check. src/gpu/ is unchanged by this release.

Files changed

  • src/sslm_abi.cpp -- adoption copies the four per-site counts; prefill passes them to
    RunLayerLoopChunkBatched. Adds a test-only SslmPrefixLiveStateForTest accessor beside the
    existing SslmSeqLiveStateForTest, compiled only in the test-injection build.
  • tests/test_slm18x_saturation_census.cpp -- new: the two cells above, in their own unit because
    the C ABI header's constants collide with test_main.cpp's Windows-only using enum SslmGpuStatus.
  • tests/test_main.cpp, CMakeLists.txt, build.bat -- wire the new unit into superslm_tests
    and the three forced-tier test builds; build.bat's superslm_tests step links sslm_abi.cpp and
    the four damped_greedy sources, as CMake already did.
  • CMakeLists.txt, tests/install-consumer/CMakeLists.txt, README.md, CHANGELOG.md, this
    note -- version 1.8.1.

SuperSLM 1.8.0

Choose a tag to compare

@dansupergameprogrammer dansupergameprogrammer released this 25 Sep 11:51

SuperSLM 1.8.0

Released 2026-09-25.

SuperSLM 1.8.0 makes the GPU path report memory exhaustion as memory exhaustion. An allocation
failure during a GPU call, on a device that is not removed, returns SSLM_GPU_ALLOCATION_FAILED,
leaves the process's GPU command list closed, and leaves the context, the device and every other
handle usable. No token, save format, SslmGpuStatus ordinal or ABI struct changes. Several
statuses change for the same fault; read "Compatibility and required action" before upgrading a
host that maps GPU statuses.

The status contract a caller relies on

Stated once in include/superslm/gpu_1p0.h, beside SSLM_GPU_ALLOCATION_FAILED:

  • When. One of these fails inside a GPU call, on any heap, while the device is not reported
    removed at the moment the call classifies the failure:
    • a host allocation (std::bad_alloc, std::length_error);
    • a D3D12 call returning E_OUTOFMEMORY -- resource creation, Map, Reset, pipeline creation,
      and a Close or Signal the engine confirmed recovered;
    • the first-time setup of the process's submission device, at the first call that submits GPU
      work.
  • What stays valid. The context and the device, with the submission command list closed; every
    other sequence, model and adapter handle. An entry point with an output handle leaves it null. In
    sslm_decode_step_batch_gpu the status is that sequence's own out_statuses[i], and the other
    sequences are recorded and complete as usual.
  • What the caller may do next. Retry the call; it succeeds once memory is available. Retrying
    smaller is also valid: a smaller context_cap, or a model mapped without
    SSLM_GPU_RESIDENCY_HEAD_ON_DEVICE. A sequence the failed call was advancing must be reset
    (sslm_gpu_seq_reset) or restored before its next generation call: a prompt or schema prefill
    may have committed earlier sub-chunks; its prefill snapshot is empty, so
    sslm_gpu_seq_read_prefill_final_hidden returns SSLM_PREFILL_HIDDEN_UNAVAILABLE; a decode
    step's finish may have written K/V the host mirror does not hold.
  • Named residuals, both reported as SSLM_DEVICE_LOST, the conservative direction:
    • an allocation failure racing an unrelated device removal;
    • a stranded submission: memory exhausted during command recording where the command list
      cannot be confirmed Closed after Close, a retry and a confirm step, or a fence Signal or
      wait that cannot be confirmed. That is the terminal case gpu_1p0.h already documents: every
      later call on that list's context fails, and for the process's submission list, which every
      decode and prefill uses, every later submitting call in the process.
  • SSLM_DEVICE_LOST now means one of: the device is reported removed; a GPU operation failed
    for a reason other than memory (a missing shader, a D3D12 call failing with another HRESULT); a
    stranded submission; the submission device's first-time setup failed for a reason other than
    memory; or, as before, a saturated context cap in a prefill. It never means that memory ran out
    on a live device with the submission clean.
  • A failed first-time setup of the process's submission device returns one status per cause,
    the same from every entry point: SSLM_DEVICE_LOST, on every call for the life of the process,
    when setup failed for a reason other than memory (no adapter, device creation refused, a
    non-memory D3D12 failure); SSLM_GPU_ALLOCATION_FAILED when it ran out of memory, and the next
    call sets up again.

What changed

1.7.1 got the status wrong in three ways when memory ran out during a GPU call (TE-419, TE-421,
TE-423):

  • At many sites -- every D3D12 upload and readback heap allocation during an encode, and the host
    allocations of a submission's finish -- it reported the GPU as lost, although the engine had
    recovered.
  • At every host allocation inside a command-recording window it reported the allocation failure
    correctly but left the process's shared command list recording, so the next GPU call in the
    process, on whichever handle ran next, failed and reported the GPU as lost.
  • If memory ran out while the process set up its GPU submission device, the failure was cached for
    the life of the process, with a different permanent status depending on which call touched the
    device first (SSLM_DEVICE_LOST, SSLM_SEQUENCE_REJECTED or SSLM_SEQUENCE_KV_BUFFER_MISMATCH).

The fix, in src/gpu/:

  • Classified at the throw. SSLM_GPU_HR throws harness::GpuAllocationError for
    E_OUTOFMEMORY from any D3D12 call, and a plain std::runtime_error for every other failure.
  • One classifier at every site that turns an exception into a status. Rules, in order: a
    stranded submission is SSLM_DEVICE_LOST; a removed device is SSLM_DEVICE_LOST; an
    allocation failure is SSLM_GPU_ALLOCATION_FAILED; anything else keeps the site's
    non-allocation status.
  • The status is no longer discarded. The prompt and schema prefill twins, per-token decode,
    sslm_gpu_ready, the decode-step bridge and batch decode return the classified status.
  • Every command-recording window closes its list on every exception type, including a
    catch-all that closes the list and rethrows, and each submission tail contains every exception
    type, waiting its submitted work out before the buffers that work uses are released. Every
    Close on the 1.0 path goes through
    CloseListConfirmed: Close, a retry, then a confirm (Reset then Close), so a Close that
    fails with E_OUTOFMEMORY but did close the list is recoverable, and one that did not is
    reported as the stranded terminal case.
  • Every other try on the GPU path contains every exception type too. A decode's finish did
    not: a non-standard exception type from one of its host allocations left the sequence
    Submitted with its in-flight token freed, so reset, release and model unmap refused it as busy
    and the next sslm_gpu_ready on it read freed memory (also true of 1.7.1). Sequence restore, the
    batched prefill's chunk submission and every handle-creating site now end in the same clause.
    tools/ci/check_gpu_catch_totality.py fails CI on an outermost try under src/gpu/ whose
    handlers do not end in catch (...).
  • The internal forward status splits. SslmForwardStatus::GpuAllocationFailed now means only
    an allocation failure on a live device with the submission clean. A new, appended
    SslmForwardStatus::GpuOperationFailed carries every non-allocation GPU failure on a live device,
    so a missing .cso during recording, or finishing a call with no in-flight token, is never
    reported as out-of-memory.
  • harness::GetDevice() never throws, and a first-time setup that ran out of memory is retried
    by the next submission, at most one attempt per submission (a batch submits each sequence in
    turn, so a later sequence in the same batch may set the device up again).

No shader, dispatch, barrier, recording order, readback or host arithmetic on data changed: every
changed statement in src/gpu/ is inside a catch clause or a helper only catch clauses call, at a
throw site, in a status mapping or
status-carrying signature, in the GetDevice() holder and Device::Init()'s setup-failure record,
in the close and fence helpers, or the one flag set after the command list's Reset.

Compatibility and required action

Statuses change for the same fault.

  • SSLM_DEVICE_LOST becomes SSLM_GPU_ALLOCATION_FAILED for memory exhaustion on a live device,
    host or GPU, from sslm_gpu_context_create, sslm_gpu_model_map (with or without
    SSLM_GPU_RESIDENCY_HEAD_ON_DEVICE), sslm_gpu_adapter_map, sslm_gpu_seq_create,
    sslm_gpu_seq_restore, both prefill twins, sslm_decode_step_gpu, sslm_gpu_ready's
    out_status, sslm_decode_step_batch_gpu's out_statuses, the decode-step bridge and
    SslmGpuSeqFinishTokenForG5Bridge. A host that treated SSLM_DEVICE_LOST as the only GPU
    memory signal must handle SSLM_GPU_ALLOCATION_FAILED on these entry points as a recoverable,
    per-call failure.
  • A batch no longer marks the rest of its sequences SSLM_DEVICE_LOST when one sequence runs out
    of memory; only a device loss does that.
  • A failed first-time device setup returns SSLM_DEVICE_LOST from every entry point, on every
    call; one that ran out of memory returns SSLM_GPU_ALLOCATION_FAILED and is retried by the next
    call. In 1.7.1 decode-first read SSLM_SEQUENCE_REJECTED and restore-first
    SSLM_SEQUENCE_KV_BUFFER_MISMATCH, permanently.
  • SslmGpuSeqDecodeStepForG5Bridge returns the status its underlying decode calls reported --
    SSLM_GPU_ALLOCATION_FAILED, SSLM_SEQUENCE_REJECTED for a guard refusal,
    SSLM_DISPATCH_BUDGET_TOO_SMALL for a budget below one layer, SSLM_DEVICE_LOST for a device
    failure -- where 1.7.1 reported every such failure as SSLM_DEVICE_LOST.

No new SslmGpuStatus value, and no ordinal moves. A host whose build fails on an unmapped
SslmGpuStatus (the SuperSLM-Unreal plugin's G5 guard) needs no new arm; it needs to map the
changed meanings above deliberately.

SslmForwardStatus gains one appended enumerator, GpuOperationFailed (installed header
include/superslm/checked_chain_funnel.h). No existing ordinal moves. It never crosses the C ABI
(MapForwardStatus maps it to SSLM_ARTIFACT_REJECTED; the CPU path never produces it), but a
consumer that switches exhaustively over SslmForwardStatus must add an arm.

superslm_gpu::RestoreGpuSequenceState (installed header include/superslm/gpu_port.h) gains
a trailing, defaulted out-parameter, out_setup_status. Source-compatible; a caller compiled
against 1.7.1's header must be recompiled.

For consumers that include the internal harness header (src/gpu/d3d12_harness.h;
SuperSLM-Unreal does):

  • harness::GetDevice() still never throws, and is now declared noexcept.
  • Device::available i...
Read more

SuperSLM 1.7.1

Choose a tag to compare

@dansupergameprogrammer dansupergameprogrammer released this 24 Sep 08:14

SuperSLM 1.7.1

Released 2026-09-24.

SuperSLM 1.7.1 is a single fix: the engine library writes nothing to stdout. No token, ABI
surface, or measured figure changes from 1.7.0.

What changed

The engine library writes nothing to stdout. 1.7.0 wrote # adapter: <name> to stdout
whenever SSLM_GPU_ADAPTER_INDEX was set, so a certification run could confirm which GPU actually
ran (T-2116, tools/t2116_crossvendor/run_crossvendor.ps1). The write came from one call site,
Device::Init() in src/gpu/d3d12_harness.h, reached once per GPU context, plus once per
process the first time that process dispatches (prefill, decode, or sequence restore) under that
setting (TE-407 M3: context creation and model mapping alone never touch
harness::GetDevice()'s process-wide singleton; the calls that do are superslm_gpu.cpp's
dispatch, submit, restore, and bind functions) -- reached from both SslmGpuContext::device and,
on that first dispatch, the singleton. A host that writes its own data to stdout had that line
spliced into it -- semb query --device gpu's own
output was corrupted this way (TE-393, D-SLM7752). The line now goes to stderr, the channel
every other diagnostic in the same file already uses (the SSLM_GPU_ENABLE_DEBUG_LAYER messages,
the D3D12 validation callback, the HRESULT-failure macro). The information a certification run
needs is unchanged, but run_crossvendor.ps1 DID need a script fix, not none -- see
"Compatibility and required action" below.

Sweep. Every stdout write reachable from the shipped superslm and superslm_gpu libraries
in src/ and include/ was enumerated before any fix (printf, wprintf, puts, std::cout,
fwrite/fprintf to stdout, and equivalents). The one call site above was the only one found;
every other library call already wrote to stderr or wrote nothing. Tools and tests still print
to stdout freely -- this release constrains only the shipped libraries. That the libraries write
nothing to stdout is verified for this release by that sweep, repeated on the release candidate
over a wider set of forms by the code review (TE-409).

CI scan. tests/ci/check_no_library_stdout_write.py, run by the no-library-stdout-check job
in .github/workflows/tests.yml, refuses any src/ or include/ source that names a banned
stdout-writing identifier outside a comment or a string/char literal: stdout, cout, the
printf, puts and putchar families (MSVC's _printf_l/_printf_p family included), write,
WriteFile, WriteConsole, GetStdHandle, fdopen, and the rest of the list. The token-paste
operator ## is banned too. The complete list is defined in one place, the script's
_BANNED_IDENTIFIERS, _BANNED_IDENTIFIER_FAMILY and _BANNED_OPERATORS, and its module docstring
describes it. The scan is a tripwire for those spellings, not a proof: code that reaches stdout
without naming a listed identifier can pass it (TE-402 S3, TE-409 S1).

Compatibility and required action

GpuContextConfig, GpuResidencyConfig, every C ABI verb, and every SslmGpuStatus ordinal are
unchanged from 1.7.0. GPU sequence saves are unchanged (SLM5), and no model-artifact
reconversion is required. No arithmetic on the compute path changed: the fix touches only a
diagnostic print statement, gated the same way it always was (only when SSLM_GPU_ADAPTER_INDEX
is set), on Device::Init()'s existing early-return structure.

Required action: a host that reads the # adapter: label from stdout must read it from stderr
instead.
A host that already merges the two streams needs a further check: Windows PowerShell
5.1 wraps each native stderr line reaching a 2>&1 capture in an ErrorRecord, and rendering that
record (Out-String) repeats the line's own text -- a naive merge on 5.1 sees the label twice,
once mangled, which breaks an exact-match reader (TE-402 C1). tools/t2116_crossvendor/ run_crossvendor.ps1 needed exactly this fix (both of its capture points now stringify the
pipeline, ForEach-Object { "$_" }, before parsing it). Verified by execution: the committed
per-cell capture-point expression, run directly against one rebuilt battery binary
(t2113_b1_context_smoke.exe), under both powershell.exe 5.1 and pwsh 7 -- see the build log
for the transcript. run_crossvendor.ps1 itself was not separately run under 5.1 in this round;
the enumeration capture point (the same fix, applied for consistency) was not exercised in
isolation. pwsh 7 alone was never affected. See tools/t2116_crossvendor/README.txt's "SHELL
COMPATIBILITY" section.

Files changed

  • src/gpu/d3d12_harness.h -- the one stdout write (Device::Init()) moved to stderr; its
    header comment corrected to match (TE-402 M1).
  • tools/t2116_crossvendor/run_crossvendor.ps1 -- the 5.1 ErrorRecord-wrapping fix, both capture
    points (TE-402 C1).
  • tools/t2116_crossvendor/README.txt -- documents the shell hazard and its fix.
  • tests/ci/check_no_library_stdout_write.py -- new: the CI scan for banned stdout-writing
    identifiers (TE-402 S3, TE-409 S1).
  • .github/workflows/tests.yml -- wires the scan in as a CI job.
  • tests/test_main.cpp, tests/te399-stdout-red-suite/build.bat,
    tests/te399-stdout-red-suite/build_test_main.bat,
    tests/te399-stdout-red-suite/cell_gpu_pipeline_stdout_empty.cpp,
    tests/te399-stdout-red-suite/run_cell.ps1,
    tests/ci/test_te399_stdout_write_scan_commission.py -- TE-399's (Curie's) red suite for this
    property, including its own fix-round additions (stderr-arrival pinning, the CI scan's
    must-reject/must-accept/synthetic-mutant commissioning constructions).
  • CMakeLists.txt, README.md, CHANGELOG.md, tests/install-consumer/CMakeLists.txt -- release
    bookkeeping for 1.7.1.

SuperSLM 1.7.0

Choose a tag to compare

@dansupergameprogrammer dansupergameprogrammer released this 23 Sep 18:27

SuperSLM 1.7.0

Released 2026-09-23.

SuperSLM 1.7.0 can take the token finish's logits off the calling thread in two independent ways:
a caller-supplied host parallel-for hook on both backends, and an opt-in device-resident head on
the GPU. Neither changes a token. The release also lets a host choose the directory GPU shaders
load from, keeps a GPU context usable after an out-of-memory failure during an upload, and makes
sslm_inspect and tok_verify build again.

What changed

Shader directory. GpuContextConfig gains shader_dir, an absolute UTF-8 directory holding
the compiled .cso set, read only during sslm_gpu_context_create. NULL, which every
zero-initialized config already holds, keeps loading from a shaders directory beside the host
executable. The directory is process-wide: the first create that names one, or the first shader
load through the default location, fixes it. A value that cannot name a shader set is refused with
SSLM_GPU_SHADER_DIR_INVALID, and a directory that differs from the one the process already uses
with SSLM_GPU_SHADER_DIR_CONFLICT, both before any device is created.

The token finish. The finish is the final norm, the logits over the whole vocabulary, then the
argmax, schema mask, damped-greedy selection and dead-end rule. The logits are the expensive part.
Every logit is an exact integer sum, and the narrowing, mask, argmax and dead-end rule run
unchanged on the host, on the identical row, so both options below produce the same tokens as
1.6.0.

  • A host parallel-for hook, both backends. superslm/parallel_for.h defines
    sslm_parallel_for, a caller-supplied run that the finish hands its logits row blocks to.
    Install it with sslm_workspace_set_parallel_for (CPU) or
    sslm_gpu_context_set_host_parallel_for (GPU). SuperSLM never creates a thread. A run that
    omits, repeats or invents a task index fails the call without producing a token and leaves the
    sequence ready to retry. docs/parallel_for_reference.hpp is a reference run over
    std::thread for hosts with no job system.
  • A device-resident head, GPU backend, opt-in per model. Mapping a model with
    SSLM_GPU_RESIDENCY_HEAD_ON_DEVICE uploads its output head table to the device, and the finish
    computes the exact int64 logits row there and narrows it on the host. The new VRAM per mapped
    model is the head table plus two rows, and is measured below. With the flag clear a model maps
    exactly as before and uses no new VRAM.

With no hook installed the finish runs serially on the calling thread, as in 1.6.0, and its tokens
are identical to 1.6.0. Its cost is not identical: see No-hook cost.

One host copy of a tied head. A tied model's head is its embedding table, which the GPU model
handle already holds, so the handle no longer takes a second copy of it. An untied model mapped
without the device head still keeps its lm_head copy.

Out-of-memory recovery. A GPU allocation failure during a model map, an adapter map, a
sequence create or a sequence restore no longer leaves the context's command list recording, so
the next call on the context succeeds. The statuses those calls return are unchanged. Those four
calls, and sslm_gpu_context_create, now set their output handle to null on every refusal.

Tools and build. tools/build_inspect.bat links again, and CMake builds sslm_inspect and
tok_verify as targets linking the superslm library. build.bat completes again: its ABI
verb-count gate now expects 38, and it compiles four test suites that previously had no build
recipe.

Compatibility and required action

GpuContextConfig grows from 4 to 16 bytes on x64 and is passed by value. The change is
source-compatible and binary-incompatible: code compiled against an earlier gpu_1p0.h must be
recompiled against this one before it links with 1.7.0.

GpuResidencyConfig's ignored int reserved becomes uint32_t flags, with the same size, offset
and alignment, so a zero-initializing caller is unaffected. Undefined flag bits are refused with
SSLM_GPU_RESIDENCY_FLAGS_INVALID.

Five statuses are appended to SslmGpuStatus (19 to 23), so no existing ordinal moves. The CPU C
ABI adds one verb, sslm_workspace_set_parallel_for, and no status.

The shipped shader set gains logits_site.cso. Deploy the whole 1.7.0 set: a model mapped with
the device head refuses with SSLM_GPU_SHADER_BINARY_STALE on a stale binary and
SSLM_DEVICE_LOST on a missing one.

GPU sequence saves are unchanged (SLM5), and no model-artifact reconversion is required.

A host gets the speedups below only by installing a hook or mapping with the device head. Without
either, see No-hook cost.

Measured performance

The cell. Measured 2026-09-23 on the reference development machine: AMD Ryzen 9 3950X (16
cores, 32 threads) with 64 GB of DDR4 at 2133 MT/s (four 16 GB modules), Windows 10 Pro, and an
NVIDIA GeForce RTX 2080 SUPER with driver 32.0.15.6094. CPU load was 3% at the start of the run and
0% at the end. 1.6.0 and 1.7.0 are both CMake Release builds with the GPU backend (MSVC 14.33),
from the v1.6.0 tag and the 1.7.0 product source. The models are Qwen2.5-0.5B-Instruct (24
layers) and Qwen2.5-1.5B-Instruct (28 layers) as .sslm artifacts, with no adapter. Every run
decodes greedily from the same 11-token prompt, at the full layer budget per step.

Tokens were identical to 1.6.0 in every cell below, at every thread setting and with the
device head on or off.

These are measurements of these models, this prompt and this machine, not guarantees. No other
CPU or GPU was measured for these figures.

Device-resident head (GPU)

Flag set, no hook. tok/s is decode steps per second after the first token, each step being the
token's embed, layers, wait and finish. Host finish is the median time of the finish call.

Model and run 1.6.0 tok/s 1.7.0 device head tok/s Host finish median, 1.6.0 → 1.7.0
Qwen2.5-0.5B, 100 tokens 42.67 73.21 (1.72×) 11.118 → 1.379 ms
Qwen2.5-1.5B, 100 tokens 27.85 55.49 (1.99×) 18.047 → 1.661 ms
Qwen2.5-0.5B, schema potion_shop_order, 29 tokens to the schema's dead end 43.94 75.54 (1.72×) 11.163 → 1.456 ms

VRAM cost. Measured as the process's VRAM use across sslm_gpu_model_map:

Model Flag clear (same as 1.6.0) Flag set Device head cost
Qwen2.5-0.5B 366,936,064 B 504,369,152 B 137,433,088 B (131.07 MiB)
Qwen2.5-1.5B 1,352,531,968 B 1,587,159,040 B 234,627,072 B (223.76 MiB)

Nothing further is allocated per sequence or per token.

Host parallel-for hook

64 tokens, device head off. T is the number of threads the finish's logits run on: the calling
thread plus T − 1 workers of the reference run in docs/parallel_for_reference.hpp, installed
with max_tasks = T. T = 0 is no hook installed. On both backends tok/s is decode steps per second after
the first token. On the GPU, finish is the host finish median; on the CPU it is the median of 30
(0.5B) or 10 (1.5B) finish-only calls.

Backend and model Measure 1.6.0 T = 0 T = 2 T = 4 T = 8 T = 16
GPU, 0.5B tok/s 42.61 42.29 55.23 59.64 59.12 57.95
finish ms 11.056 11.356 6.598 5.438 5.500 5.688
GPU, 1.5B tok/s 27.85 28.12 36.52 39.37 37.90 38.06
finish ms 18.045 18.450 10.510 8.900 9.138 9.269
CPU, 0.5B tok/s 21.78 21.36 23.54 24.16 24.06 23.89
finish ms 10.072 10.237 5.969 5.006 4.998 5.224
CPU, 1.5B tok/s 7.68 7.56 7.98 8.09 8.09 8.04
finish ms 16.789 17.134 9.637 8.477 8.881 8.793

Thread guidance. T = 4 gave the best or equal-best tok/s in every row on this 16-core CPU; 8
and 16 were flat or slightly worse. The finish is a small share of a CPU token, which is mostly
layer time, so on the CPU halving the finish moves tok/s by about 11% at 0.5B and 5% at 1.5B.
Measure T on the target machine.

Grade. Each cell in these two tables is one process per build, with medians taken over the
steps inside that process. At T = 4, and with the device head, the finish times' whole min–max
range lies below 1.6.0's in every row. The CPU 1.5B tok/s gain is smaller than the process-to-process
spread of CPU 1.5B step times in the whole-token runs below, so it is not a resolved figure.

No-hook cost

With no hook installed and the device head off, 1.7.0's tokens are identical to 1.6.0, but it is
measurably slower in some cells. Each figure below is ten processes per build, run alternately
(1.6.0, 1.7.0, 1.6.0, …), with the 95% Welch interval of the difference in means. A difference is
resolved when its interval excludes zero; otherwise the resolution achieved is given. The cause has
not been identified yet.

The finish-alone figures come from the same run as the tables above. The whole-token figures come
from a second run later the same day, on the same machine, builds, models and prompt, with CPU load
2% at its start and 46% at its end.

The finish alone (Qwen2.5-0.5B; CPU median of 30 finish-only calls per process, GPU host
finish median over 32 tokens per process):

Measure 1.6.0 1.7.0 Difference, 95% interval Reading
CPU finish 10.0596 ms 10.2698 ms +0.2102 ms (+2.09%), [0.1748, 0.2456] resolved
GPU host finish 11.0903 ms 11.4455 ms +0.3552 ms (+3.20%), [0.3182, 0.3922] resolved

Whole tokens (CPU: median step time; GPU: median token time):

Backend, model, tokens Measure 1.6.0 1.7.0 Difference, 95% interval Reading
CPU, 0.5B, 32 step 45.850 ms 46.786 ms +0.936 ms (+2.04%), [0.871, 1.001] resolved
tok/s 21.764 21.334 −0.430 (−1.98%), [−0.4...
Read more