Straydog v0.1.11

Straydog 0.1.11. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes; what 0.1.10 changed is in its
notes.

This release is about reading long prompts when the model does not fit in VRAM: Qwen3.8 on one GPU with most of its experts in host
memory or on the drive. Decode rates on the setups measured are where 0.1.10 left them.

Faster prompts when the model is partially resident (Qwen3.8, one GPU)

• Layer-ordered prefill. A long prompt can now be read one layer at a time: each layer's experts are brought to the GPU once and
  every row of the prompt goes through them, where the batch-by-batch order brought most of them again for each batch. Straydog picks
  the order per request from the prompt's length, how wide the layer batches can be, and how much memory the experts have at that
  moment (not only how much RAM the machine has). STRAYDOG_LAYER_PREFILL=0 turns it off.
  • Windows, RTX 5070 Ti, UD-IQ2_XS with the MTP drafter at --vram 5.5, 62 GiB of RAM, a new 32,646-token conversation in the
    batch-by-batch order and in the order Straydog now picks: 161.8 s and 57.0 s while other programs hold most of the RAM (4 GiB
    available), 73.8 s and 57.3 s when they do not (each the second request of its process); a new 8,210-token conversation 25.4 s and 17.6 s. (Measured before the CUDA
    kernels below.)
  • Linux, R9700, the same model in a 32 GiB memory limit: a 14,937-token prompt 34.9 s before, 25.7 s now; a 3,232-token
    follow-up 24.6 s before, 13.3 s now.
• CUDA: the expert kernels of the prompt decode every weight format the bundles use in their own code, where all but Q2_0 went
  through one general path; IQ4_NL reads its levels from registers, and IQ2_S/IQ2_XXS gate/up weights decode from small tables in
  shared memory with packed half products. The 32K prompt above, in layer order and the first request of a new process, took
  108.5 s with the general path, 53.6 s with the per-format kernels and 46.7 s with the IQ2 tables.
• Layer order keeps its hidden rows in VRAM where they fit, moving them between layers by device copies, where every layer
  sent every row to the host and back (980 GB for a 250K prompt). A layer whose experts are locked in memory is read from its
  mapping instead of being copied again. Windows, RTX 5070 Ti, UD-IQ2_XS with the MTP drafter at --vram 11, 64 GB of RAM, a new
  250K-token prompt: 307 s before, 259 s now; the follow-up after it 6.1 s before, 5.7 s now.
  • Two switches for machines with memory to spare, not on by default yet: STRAYDOG_LAYER_PREFILL_ROWS=65536 (fewer passes over
    the layers, each reading the experts again) and STRAYDOG_LOCK_DURING_PREFILL=1 (the experts are locked in memory while the
    prompt runs, not after it). With both, the same prompt took 192 s and its follow-up 4.2 s; the lowest free memory during the
    prompt was 7.3 GiB where it was 12.1 GiB.
• Reading the experts from the drive. Layer banks are read with verified direct reads into a buffer that is reused, the bank-page
  hints pause while a request runs, and their budget comes from the cgroup's headroom without counting file cache as used.

Qwen3.8's K/V in half the memory

• The attention K/V is held as 8-bit blocks (Q8_0) where it was FP16: 3.45 GiB at 262,144 tokens where it was 6.5 GiB, and
  half what decode reads of a K/V in host memory. Against the FP16 K/V, a 128-token reply after 8K, 65K and 250K tokens: KL mean
  6.3e-4 to 1.2e-3, the same top token on 366 of 367 rows (RDNA4); on CUDA 8.2e-4 and 1.2e-3 at 8K and 65K. Decode rates stay where they were: one
  R9700 with its K/V in host memory 38.9 and 35.9 tok/s after 65K and 250K tokens, against FP16's 39.2 and 36.2; fully resident on
  two R9700s (UD-IQ3_XXS), the fixed scenarios' long replies take the same time a decode pass as with FP16 (within 2%).
  STRAYDOG_KV=f16 keeps FP16.

Long conversations

• A conversation's long segments are tokenized once. Every request renders the whole conversation again, and every request
  tokenized it whole: 0.30 to 0.34 s for a 250K-token conversation's follow-up, now 0.003 to 0.006 s.
• DeepSeek-V4.1 can be served with up to its declared 1,048,576-token context (--ctx-size 1048576). The default stays
  262,144: on two R9700s it decodes as fast as 32K, where 1M decodes about 7% slower (not yet explained). With 1M the host keeps
  room for two checkpoints of a full-length conversation.
• straydog setup --list says each bundle's longest context (context_max), which the app's context slider uses as its limit.

Two GPUs on Windows (#2)

• A GPU that cannot register host memory read-only (CUDA's cudaDevAttrHostRegisterReadOnlySupported 0, as two RTX 3090s on
  Windows 10) now keeps a verified page-locked copy of each locked layer's experts instead of locking none. The start says so in one
  line.
• The two GPUs' waits ask the driver to run what is queued (WDDM can hold queued work until it is asked), and the split output head
  does the same for each GPU's part.
• A request's line splits decode's "other" time into its parts (draft, head, first layer and PLE, accept).
• Sending the request lines of a slow run to the issue is the most useful way to show what is still slow.

Shorter waits and fewer failures

• DeepSeek-V4.1: a helper conversation no longer evicts the main one's latest checkpoint. Going back to the main conversation
  after a short side request took 7.18 s and now 0.63 s (two R9700s).
• Windows with little free memory: a row read the working set refuses (error 1453) is read again in smaller pieces and then
  through the page cache, still verified, and a request that still cannot read fails without ending the server; an expert slot pool
  that cannot be allocated at start is made smaller instead of failing the start.
• The first request after ready no longer waits for cuBLAS to load (the prefill GEMM is primed while the model is placed).
• A warm-up behind ready lets requests in between its layers.

Setup

• Qwen3.8's state per token of context is the engine's own count (28,288 bytes with the MTP layer, 6.91 GiB at 262,144 tokens),
  where setup used a table made when the K/V was F32 (13.4 GiB). Which bundles fit in RAM and which are recommended change on
  machines near the line: a 16 GB card with 62 GiB of RAM now has IQ3_S's experts in RAM, and a Linux 16 GB card with 32 GB of RAM
  is recommended UD-IQ1_M instead of UD-IQ1_S.

Diagnostics and tools

• Every switch the engine reads now reaches the model's worker process (the worker gets an allowlist of the server's environment;
  STRAYDOG_KV, the layer-order switches and the development switches were missing), and a test keeps the list complete.
• The start line and the dashboard say the build's version and commit.
• A partially resident request's line says how much the drive gave back for page faults, the working set's limit and the K/V's
  capacity, as well as the prefill's batches and the seconds that were not reading the prompt.
• --check-model says whether the configured model loads as it is.
• A crash line names what called the failing code.
• A request's timings say which prefill order ran and why (layer_prefill_policy), how wide its batches were, and what growing the
  K/V took.