Straydog v0.1.11

Straydog 0.1.11. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes; what 0.1.10 changed is in its
notes.

This release is about reading long prompts when the model does not fit in VRAM: Qwen3.8 on one GPU with most of its experts in host
memory or on the drive. Decode rates on the setups measured are where 0.1.10 left them.

Faster prompts when the model is partially resident (Qwen3.8, one GPU)

• Layer-ordered prefill. A long prompt can now be read one layer at a time: each layer's experts are brought to the GPU once and
  every row of the prompt goes through them, where the batch-by-batch order brought most of them again for each batch. Straydog picks
  the order per request from the prompt's length, how wide the layer batches can be, and how much memory the experts have at that
  moment (not only how much RAM the machine has). STRAYDOG_LAYER_PREFILL=0 turns it off.
  • Windows, RTX 5070 Ti, UD-IQ2_XS with the MTP drafter at --vram 5.5, 62 GiB of RAM, a new 32,646-token conversation in the
    batch-by-batch order and in the order Straydog now picks: 161.8 s and 57.0 s while other programs hold most of the RAM (4 GiB
    available), 73.8 s and 57.3 s when they do not (each the second request of its process); a new 8,210-token conversation 25.4 s and 17.6 s. (Measured before the CUDA
    kernels below.)
  • Linux, R9700, the same model in a 32 GiB memory limit: a 14,937-token prompt 34.9 s before, 25.7 s now; a 3,232-token
    follow-up 24.6 s before, 13.3 s now.
• CUDA: the expert kernels of the prompt decode every weight format the bundles use in their own code, where all but Q2_0 went
  through one general path; IQ4_NL reads its levels from registers, and IQ2_S/IQ2_XXS gate/up weights decode from small tables in
  shared memory with packed half products. The 32K prompt above, in layer order and the first request of a new process, took
  108.5 s with the general path, 53.6 s with the per-format kernels and 46.7 s with the IQ2 tables.
• Layer order keeps its hidden rows in VRAM where they fit, moving them between layers by device copies, where every layer
  sent every row to the host and back (980 GB for a 250K prompt). A layer whose experts are locked in memory is read from its
  mapping instead of being copied again. Windows, RTX 5070 Ti, UD-IQ2_XS with the MTP drafter at --vram 11, 64 GB of RAM, a new
  250K-token prompt: 307 s before, 259 s now; the follow-up after it 6.1 s before, 5.7 s now.
  • Two switches for machines with memory to spare, not on by default yet: STRAYDOG_LAYER_PREFILL_ROWS=65536 (fewer passes over
    the layers, each reading the experts again) and STRAYDOG_LOCK_DURING_PREFILL=1 (the experts are locked in memory while the
    prompt runs, not after it). With both, the same prompt took 192 s and its follow-up 4.2 s; the lowest free memory during the
    prompt was 7.3 GiB where it was 12.1 GiB.
• Reading the experts from the drive. Layer banks are read with verified direct reads into a buffer that is reused, the bank-page
  hints pause while a request runs, and their budget comes from the cgroup's headroom without counting file cache as used.

Qwen3.8's K/V in half the memory

• The attention K/V is held as 8-bit blocks (Q8_0) where it was FP16: 3.45 GiB at 262,144 tokens where it was 6.5 GiB, and
  half what decode reads of a K/V in host memory. Against the FP16 K/V, a 128-token reply after 8K, 65K and 250K tokens: KL mean
  6.3e-4 to 1.2e-3, the same top token on 366 of 367 rows (RDNA4); on CUDA 8.2e-4 and 1.2e-3 at 8K and 65K. Decode rates stay where they were: one
  R9700 with its K/V in host memory 38.9 and 35.9 tok/s after 65K and 250K tokens, against FP16's 39.2 and 36.2; fully resident on
  two R9700s (UD-IQ3_XXS), the fixed scenarios' long replies take the same time a decode pass as with FP16 (within 2%).
  STRAYDOG_KV=f16 keeps FP16.

Long conversations

• A conversation's long segments are tokenized once. Every request renders the whole conversation again, and every request
  tokenized it whole: 0.30 to 0.34 s for a 250K-token conversation's follow-up, now 0.003 to 0.006 s.
• DeepSeek-V4.1 can be served with up to its declared 1,048,576-token context (--ctx-size 1048576). The default stays
  262,144: on two R9700s it decodes as fast as 32K, where 1M decodes about 7% slower (not yet explained). With 1M the host keeps
  room for two checkpoints of a full-length conversation.
• straydog setup --list says each bundle's longest context (context_max), which the app's context slider uses as its limit.

Two GPUs on Windows (#2)

• A GPU that cannot register host memory read-only (CUDA's cudaDevAttrHostRegisterReadOnlySupported 0, as two RTX 3090s on
  Windows 10) now keeps a verified page-locked copy of each locked layer's experts instead of locking none. The start says so in one
  line.
• The two GPUs' waits ask the driver to run what is queued (WDDM can hold queued work until it is asked), and the split output head
  does the same for each GPU's part.
• A request's line splits decode's "other" time into its parts (draft, head, first layer and PLE, accept).
• Sending the request lines of a slow run to the issue is the most useful way to show what is still slow.

Shorter waits and fewer failures

• DeepSeek-V4.1: a helper conversation no longer evicts the main one's latest checkpoint. Going back to the main conversation
  after a short side request took 7.18 s and now 0.63 s (two R9700s).
• Windows with little free memory: a row read the working set refuses (error 1453) is read again in smaller pieces and then
  through the page cache, still verified, and a request that still cannot read fails without ending the server; an expert slot pool
  that cannot be allocated at start is made smaller instead of failing the start.
• The first request after ready no longer waits for cuBLAS to load (the prefill GEMM is primed while the model is placed).
• A warm-up behind ready lets requests in between its layers.

Setup

• Qwen3.8's state per token of context is the engine's own count (28,288 bytes with the MTP layer, 6.91 GiB at 262,144 tokens),
  where setup used a table made when the K/V was F32 (13.4 GiB). Which bundles fit in RAM and which are recommended change on
  machines near the line: a 16 GB card with 62 GiB of RAM now has IQ3_S's experts in RAM, and a Linux 16 GB card with 32 GB of RAM
  is recommended UD-IQ1_M instead of UD-IQ1_S.

Diagnostics and tools

• Every switch the engine reads now reaches the model's worker process (the worker gets an allowlist of the server's environment;
  STRAYDOG_KV, the layer-order switches and the development switches were missing), and a test keeps the list complete.
• The start line and the dashboard say the build's version and commit.
• A partially resident request's line says how much the drive gave back for page faults, the working set's limit and the K/V's
  capacity, as well as the prefill's batches and the seconds that were not reading the prompt.
• --check-model says whether the configured model loads as it is.
• A crash line names what called the failing code.
• A request's timings say which prefill order ran and why (layer_prefill_policy), how wide its batches were, and what growing the
  K/V took.

Straydog v0.1.10

Straydog 0.1.10. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes; what 0.1.9 fixed is in its
notes.

This release is about waits and failures rather than steady speed: decode and prefill rates on the setups we measure are where 0.1.9
left them.

Shorter waits

• A prompt stopped while it was being read keeps what it read (Qwen3.8). A request whose client left or was stopped during a long
  prefill used to throw away every batch it had read, and sending the same prompt again read it all from the start. The cancel is now
  seen between batches, and the next request that shares the prompt goes on from the last whole batch. A 32,483-token prompt cut at
  4,096 tokens, sent again: 22.5 s before, 17.1 s now (one R9700, partially resident). With two GPUs and the MTP drafter the request
  that goes on leaves the same tokens, logits and state as one that read the prompt whole.
• DeepSeek-V4.1 starts sooner when memory is in pieces (Linux, AMD). After another large model's run had filled the page cache,
  page-locking the host tier while it was still being allocated made Linux move locked pages, and every move stopped the GPUs' queues.
  The host tier is now allocated in full before most of it is locked. Two R9700s, UD-IQ2_XXS: ready in 16 s instead of 29 to 38 s, and
  the queue stops in the first 150 s after ready cut from 15 to 27 s to 7 to 12 s.
• Windows, partially resident: the read-ahead hints no longer hold up decode. The hints that ask Windows to read the CPU's experts
  ahead are now issued by a thread of their own. RTX 5070 Ti, Qwen3.8 UD-IQ1_M at 9.5 GiB: 42.7 tok/s before and 48 to 52 tok/s now
  while the hints are on; unchanged while they are off.
• Prefill. An expert cache that starts empty is filled after the first batch has shown what the prompt routes to; the cache is
  left alone when the batch is already as wide as borrowing could make it; the first dense weights are counted before the expert
  slots are sized; on two GPUs, prompts of 64K tokens or more borrow wider workspaces and the caches come back on both GPUs at once.

Fixes

• A prefill that failed and whose caches then could not be restored reports the first failure, not only the second.
• Windows: a refused allocation or thread when the read-ahead thread starts is reported as an error instead of ending the process.

Diagnostics

• Windows: a crash says what and where before the process ends, on stderr: Straydog crashed: access violation (0xC0000005) reading 0x10 at module.dll+0x1A2B3C. A program that runs Straydog and shows its last line now shows that instead of an exit code alone.
• CUDA: the start says, for each GPU, whether it supports read-only host registration (the capability behind
  #2).
• The dashboard shows the drive's reads, and labels the host expert budget and the model's weights separately in its memory figures.
• The read-ahead hints switching on and off is logged only with --verbose; each request's line still counts them.

Straydog v0.1.9

Straydog 0.1.9. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes; what 0.1.8 brought is in its
notes.

This release fixes a start that failed on GPUs that cannot page-lock memory read-only.

Fixes

• CUDA: the model loads on GPUs that cannot page-lock memory read-only. Straydog page-locks the buffers it loads the model through,
  and the expert banks of a partially resident layer, asking for read-only registration (cudaHostRegisterReadOnly). A device without
  cudaDevAttrHostRegisterReadOnlySupported refuses that with error 801, and the start failed: "Straydog GPU: operation not supported
  (kernels.cpp:2135, code 801)" (two RTX 3090s on Windows 10, #2). The loader's buffers,
  which are only copied from, are now page-locked as plain writable memory (no read-only flag, no device mapping), whatever the device
  supports. A bank's read-only registration that a device refuses is asked again without the flag, and a bank that still cannot be
  locked is read without the lock, as before when locking was refused. Devices that take the flag register the banks as before.

Straydog v0.1.8

Straydog 0.1.8. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes.

This release reads DeepSeek-V4.1 prompts much faster when VRAM and host memory only just hold the model, uses DeepSeek's DSpark drafter
with every bundle, and reads long conversations with their K/V in host memory without falling off a cliff past the VRAM stage.

Changes

DeepSeek-V4.1 when the model only just fits

On two Radeon AI PRO R9700s, Unsloth UD-IQ2_XXS keeps 5,003 experts in VRAM and the rest in a 102 GiB host tier: the two hold the model
with almost nothing to spare. Prompts there waited on the bus and on the drive far more than they had to.

• A fresh 7.8K-token prompt reads in 5.6 s (7.6-8.8 s before), five agent turns of 1.6-5.8K new tokens in 15.1 s (17.3-17.9 s), and a
  fresh 93K-token prompt in 50.8 s (87.6 s with only the first of these changes in).
• No more write-back chains. While a prompt streamed a layer's experts through VRAM, a full host tier gave a displaced resident the
  host copy of an expert being streamed, whose release then had to be written back too, and so on to the end of the pass: 17,696 of a
  7.8K prompt's 19,804 write-backs, and 3.3 s of waiting. A write-back now never takes a streamed expert's copy.
• What is let go of is what the prompt wants last. With no room left, the store lets go of experts of the layers the pass has done
  (first those it did not route to, then the latest), never of the layers still to come: a 93K prompt reads 74 GB from the drive, 283 GB
  before.
• Linux no longer stops the GPUs. The host tier is page-locked for the GPUs, and each time Linux moved one of its pages (compaction,
  khugepaged, swap) the driver stopped every GPU queue of the process, for 0.03-2.5 s: some 20 s over a 93K prompt and the turns before
  it. The host tier now takes huge pages (compaction passes them by), and other allocations no longer ask for huge pages (asking had Linux
  compact memory whenever it was short): 0.35 s. Loading takes 12 s, 17 s before.
• These change no bit of what is computed. UD-IQ1_M (with more room to spare) is as fast as before or faster: five agent turns 17.4 s,
  code 78 tokens/s.

DSpark with every DeepSeek-V4.1 bundle

• The DSpark drafter was used only with the bundles listed beside it; any DeepSeek-V4.1 bundle now uses it when its dspark/ folder is
  there. UD-IQ2_XXS: code 45.2 to 55.7 tokens/s, thinking 44.5 to 60.8.
• Its three stages keep 48, 28 and 16 of their most used experts in VRAM (87, 50 and 29 before): the drafts are kept about as often,
  and the target gets 138 more expert slots (UD-IQ2_XXS code 51.7 to 55.7 tokens/s). DS41_DSPARK_RESIDENT still sets them.

Long conversations with the K/V in host memory

• Past the VRAM stage (some 45K tokens at a 6 GiB budget), each token of a prompt batch read its 2,052 selected positions straight from
  host memory: an RTX 4060 read a token of a 48K conversation in 52-68 ms against 4.3 ms below the stage. A batch now copies the union
  of its selections into the stage once and reads them there, the same bits, on CUDA and on RDNA4's batched attention alike.

Long prompts on two GPUs (Qwen3.8)

• A prompt batch's rows go through page-locked buffers of their own, so the GPUs no longer wait for each other between batches: a 95K
  prompt on two R9700s (UD-IQ3_XXS, --vram 30) 21.5 to 21.0 s, a 52K one on an RTX 5070 Ti at 5.5 GiB 61.9 to 52.5 s, the same bits.
  (Served, a 95K prompt took 22.5 s with 0.1.7's huge pages for every allocation, and takes 22.7 s now.)
• On an RTX 5070 Ti at 5.5 GiB (UD-IQ1_S, K/V in host memory), a 42K-token conversation's first prompt reads in 31.3 s and each
  640-token turn after it, up to 48K, in 1.2-1.4 s (39.6 s and 1.7 s with only the K/V stage's fix above).

Drafts by what a pass costs (partially resident models)

• How many drafts pay depends on the machine: where the GPU's part of a layer is the longer, a second and third row cost little; where
  the CPU's experts are the limit, each row doubles their routes. A partially resident model now times its passes by their rows and
  takes the width that is fastest by measure, and says so: Straydog drafts: 0 18.2 ms/token, 1 16.9, 2 17.4 (accepted 0.93, 0.71); keeping 1.

Linux with NVIDIA GPUs (new; tested under WSL2)

• A Linux executable for NVIDIA GPUs (CUDA 13), straydog-v0.1.8-linux-x86_64-cuda.tar.gz. It needs the NVIDIA driver, the CUDA 13
  runtime (libcublas.so.13 and libcublasLt.so.13 in /usr/local/cuda/lib64, /usr/local/cuda-13/lib64 or wherever ldconfig
  lists them), bubblewrap and glibc 2.39 or later (Ubuntu 24.04 and later, as the ROCm build), and it is built for RTX 20 to 50 series
  GPUs (as the Windows build; data-center GPUs need a build of their own). It has been run under WSL2 (Ubuntu 24.04, RTX 5070 Ti): the GPU tests as on Windows, the same logits bit
  for bit as the Windows build on a 32K-token prompt, Qwen3.8 and DeepSeek-V4.1 served isolated. Nothing in it is WSL-only: the
  isolated worker is given the NVIDIA device nodes the host has (/dev/nvidia*, and /dev/dxg under WSL2), so it is meant for
  native Linux as well, where it has not been run yet.

Windows: the conversation kept across a restart (off by default)

• With --keep-conversation (or keep_conversation = true), a server stopped on purpose seals the state of the conversation it held
  last for the Windows user (AES-256-GCM, its key wrapped with DPAPI) and the next start takes that conversation's next request from
  there. A 39K-token conversation went on with its first token in 2.2 s, against 48.9 s read again (RTX 5070 Ti, UD-IQ1_S).

Fixes

• DeepSeek-V4.1: lending the cards' spare VRAM to the expert store could use up the budget, and the next request that allocated a buffer
  failed for good. Lending leaves 128 MiB; a budget error says what was asked and what was left.
• A frame's body crosses the worker's pipe in 4 MiB reads (a kept conversation's 1 GB took 7.5 s in 4 KiB ones).

Straydog v0.1.7

Straydog 0.1.7. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes.

This release reads long prompts faster on two GPUs, holds Qwen3.8's K/V in half the memory, and says at the start what a little
more VRAM would have bought.

Changes

Faster long prompts on two GPUs (Qwen3.8)

• A 95K-token prompt on two Radeon AI PRO R9700s (UD-IQ3_XXS, MTP drafter, prefill batch 2048) reads in 23.3 s, 25.5 s before; a
  32K one in 8.2 s, 8.9 s before. At the batch the placement takes itself (4096 at 262144 tokens, --vram 30), a 95K prompt reads
  in 22.4 s.
  • A DeltaNet layer's scan spreads a head's columns over four blocks and stages 16 tokens at a time: 1.9 times as fast, the same
    bits.
  • The two GPUs' exchanges move 16 bytes a thread. A scatter's residual update also normalizes the rows for the HyperConnection
    after it, whose projections then read FP16 values only. The MTP layer reads a prompt batch at once rather than 512 rows at a
    time.
  • These change no bit of the prompt's outputs; the MTP layer's change moves only how many drafts are accepted.

Qwen3.8's K/V in FP16

• The K/V holds FP16 values: half the bytes of F32 in VRAM, in host memory, in the VRAM stage and its mirrors (which hold twice the
  positions) and in a conversation's saved positions. With its K/V in host memory at 262144 tokens, a two-GPU placement's state
  falls from 14.4 to 7.4 GB.
• A long prompt's attention rounded keys and values to FP16 already; decode and verification now read them so. Against F32 on
  UD-IQ3_XXS, each row's logits on 255-token replies: KL mean 3.6e-4, 4.6e-4 and 6.6e-4 at 8K, 32K and 95K tokens, the same top token
  on 762 of 765 rows (the others near ties). A 4-row verification pass at 95K takes 22.85 ms, 24.09 ms before.

The ready line says what more VRAM would buy

• Qwen3.8's placement takes the first of: full residency with the K/V in VRAM, then with the K/V in host memory at prefill batches of
  4096 down to 1024, with the MTP drafter and then without, then partial residency. A few hundred MB decide which, and the speed
  with it: on two R9700s an 8K prompt takes 2.5 s fully resident at --vram 30, 6.0 s partially resident at --vram 26. When the
  first choice did not fit, the ready line now says what it would have needed, for example prefill batch 4096 (K/V in VRAM, prefill batch 4096, MTP drafter would need 2.6 GiB more VRAM), or with partial residency (fully resident: K/V in host memory, prefill batch 1024 would need 2.3 GiB more VRAM).

Machines short of VRAM and RAM

• straydog setup adds Qwen3.8's MTP drafter by default from a 5 GiB budget (it was 5.5), and --mtp adds it whatever the
  budget. On an RTX 5070 Ti at 5.5 GiB with UD-IQ1_S, 41.9-49.1 tokens/s without the drafter, 53.9-60.0 with it.
• Memory freed after the start is used. A partially resident model planned its page-locked layers, and under --vram auto its
  VRAM budget, once at the start. After each request it now locks the layers the memory available has room for, and under auto
  raises its budget and places its expert slots again. An RTX 5070 Ti started beside a program holding 1 GiB of VRAM went from 672
  to 1021 slots once it was gone (44.9 tokens/s, then 52.2 and 48.0).
• CUDA: a decode step's head waits once. The mix, the output projection and a check of the logits run in one queue with one wait:
  the step's head 1.11 ms to 0.89 (RTX 5070 Ti, UD-IQ1_S).
• CUDA: a partially resident layer routes in one kernel. The router, the shared expert's gate, the top-10 and the export to the
  CPU were four launches; the GPU's wait for the layers falls from 6.93 to 6.51 ms a token.
• The request line splits the GPU's wait by the GPU's own clock: GPU wait 8.8 (the GPU's layer work 8.2, its experts 7.6 beside the CPU).

Fixes

• CUDA: a partially resident model no longer fails after its expert slots are placed again. Raising the budget or ending a long
  prompt allocated the slots anew while captured graphs still held their old address; the next request ended with an illegal memory
  access. The graphs are dropped and captured again.
• The VRAM budget's changes and the warning that no more layers could be locked reach the log without --verbose.

Straydog v0.1.6

Straydog 0.1.6. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes.

This release is for machines whose RAM holds only part of a model's experts, such as a 64 GB Windows PC with an RTX 4060.

Changes

Fixes

• An engine that stopped for memory says why. A worker that stopped between requests was logged with a bare exit code, and
  the lines it wrote last, which name the cause, were kept out of the log. The error line now carries its last six lines (an
  allocation that failed, a GPU error) and its exit code, on Windows in hex with the memory codes named (STATUS_NO_MEMORY,
  STATUS_COMMITMENT_LIMIT).
• A prompt's CPU share no longer stops the engine when pinned memory is refused. It takes pageable memory then, and where even
  that is refused, the batch streams its experts as before. A batch gives the CPU at most 4096 routes.
• The engine no longer stops when the conversation outgrows the memory left. After a request, Straydog copied the conversation's
  state to host memory in one ordinary allocation of hundreds of MB to a few GB; where the system's commit charge was used up (a 64
  GB Windows machine with most of a model's banks page-locked), the allocation failed and ended the process (0xC0000017). That copy
  now asks first, gives back page-locked layers to make room, and else goes on without it. A refused growth of the K/V likewise
  takes back locked layers, newest first, and fails only the request when none is left; the locking behind ready leaves room for
  the K/V of the first 64K tokens. On an RTX 5070 Ti machine with its commit charge reduced, a conversation that failed at 30K
  tokens went on to 63K.

Faster prompts on machines short of RAM

• Layers that are not page-locked give the CPU only the experts it computes for less than their gather. There the CPU's
  workers gather each streamed expert into pinned memory first, and the CPU's own routes waited behind those gathers. On a Radeon
  AI PRO R9700 at 5.5 GiB in a 40 GiB memory scope (27 of 48 layers locked): prompts of 2.1K, 4.2K and 10.6K tokens 7-15% faster.
• Windows: a prompt's PLE rows are read at the drive's speed. NTFS ran unbuffered reads of a mapped file one at a time; the
  table's file is now opened apart. On an RTX 5070 Ti at 5.5 GiB the wait for the rows fell from 0.9-1.9 s to 10-26 ms, a 1.9K-token
  prompt from 4.5-5.0 s to 2.9 s.

Straydog v0.1.5

Straydog 0.1.5. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes.

Changes

Qwen3.8 on machines short of VRAM (partially resident)

• Ready sooner. A partially resident model now says it is ready before it page-locks its host banks and warms its GPU code; both
  go on behind, and the GPU's share of host experts turns on layer by layer as the banks are locked. On a Radeon AI PRO R9700 with
  UD-IQ1_S at 5.5 GiB, ready came after 3.0 s instead of 17.2 s; on an RTX 5070 Ti, 3.5-3.8 s instead of 12.9 s. The locking
  runs at a low priority and pauses between layers while a request generates (for at most 20 s at a time).
• A cold start reads its banks ahead. Right after ready, Straydog asks the system to read the banks it is about to lock, 256 MiB
  at a time at a low priority, ahead of the locking, so a first request does not wait on the drive page by page. On the R9700 with
  the model's pages out of the page cache, a request 12 s after ready: first token 1.67 -> 0.65 s, 33.3 -> 48.6 tok/s (49 warm),
  and the locking was over 31 s after ready instead of 39. On an RTX 5070 Ti, 33-39 -> 43-47 tok/s. On Windows the reading and the
  locking keep their memory priority: at the background priority Windows put the pages they read first in line to be dropped.
• The GPU's share of host experts is chosen by timing. A partially resident layer can have the GPU read a few of the
  experts kept in page-locked host memory instead of leaving them to the CPU. How many was fixed, which could slow a machine with
  a narrow bus (a PCIe 4.0 x8 card). Now Straydog times the decode passes at each level (0, 2 or 4 a layer with ROCm; 0, 1
  or 2 with CUDA) and moves to another level only when it is at least 2% faster by more than twice the measurements' noise;
  the log's Straydog assist: line gives the times. On a Radeon AI PRO R9700 it keeps
  4, as before (45.1 tok/s against 44.3).
• A warm start. Straydog counts which experts each layer routes to and keeps the counts beside its other state, one file
  per bundle (written after requests, at most every 120 s, and at exit). At the next start it fills the GPU's expert slots
  from the counts and asks the system to read the most-used experts' pages ahead. On an RTX 5070 Ti with UD-IQ1_M, a first
  request sent 12 s after the start (as a person types) began 30% sooner (4.00 -> 2.81 s) and decoded 10% faster
  (34.8 -> 38.5 tok/s); one sent within a second of the start gained little (first token 4.06 s). Later requests were no
  slower.
• The CPU's experts are computed on 8-bit activations (AVX-512 with VBMI: Ryzen 7000 and later, Intel's 11th generation and
  others; or AVX2: Intel's 12th to 14th generation, Zen 3): activations rounded to 8-bit integers a block, exact integer products,
  one F32 scale a block, as llama.cpp's CPU path and Straydog's GPU GEMVs do. For IQ2_XXS, IQ2_S and Q2_0, the formats of
  Unsloth's UD-IQ1_S bundle. Where the CPU has VNNI (AVX512-VNNI: Intel's 10th and 11th generation, Zen 4 and 5; AVX-VNNI: Intel's
  12th generation on, Zen 5), its dot instructions take the products, with the same results (the log names avx512-vnni-int8). On a Ryzen 5 7500F, six threads, experts cold: 48.0 -> 35.2 us a route on its IQ2_XXS layers,
  46.0 -> 36.9 us on its IQ2_S layers; served at 5.5 GiB of VRAM, 48.4 -> 50.4 tok/s with the CPU's time a token 9.7 -> 8.4 ms.
  Held to AVX2, served: 42.1 -> 48.4 tok/s. On a Ryzen 7 9700X a route takes 32.6 us instead of 48.8. VNNI on top: on the 7500F
  the CPU's time a token 8.7 / 8.2 -> 8.2 / 7.8 ms; on a Ryzen 7 9700X with an RTX 5070 Ti, 11.9 -> 9.6 ms, 44.7-47.3 -> 50.2-51.9 tok/s. The replies change by KL
  0.0022 against F32 (two F32 runs differ by 0.0009). Other formats keep F32, and STRAYDOG_CPU_INT8=0 keeps it everywhere.
• Routers in half the bytes. Unsloth's bundles store Qwen3.8's routers in F32, though every value is a BF16 value; they now go
  up as BF16, the same values, so a decode step reads 126 MB less, and the VRAM they free holds more experts (UD-IQ1_S at
  5.5 GiB: 953 -> 1040 expert slots). This helps cards whose memory, not their CPU, is the limit.
• A prompt's rarely used experts on the CPU. A prompt batch streamed every expert the GPU did not hold over the bus, whatever
  its rows, while the CPU waited. Now the experts with the fewest rows go to the CPU, which computes their routes while the bus
  streams the rest; how many is set by how the last batches' CPU and GPU finished, so that both end about together. The request
  line counts them (prompt streamed N experts and the CPU M routes). On the R9700 at 5.5 GiB, prompts of 2,101, 2,397 and 3,394
  tokens: 2.1 -> 1.7 s, 2.1 -> 1.7 s and 2.4 -> 2.1 s; on an RTX 5070 Ti, 5-7% faster. Which experts go where depends on timing, so
  such a prompt is not computed alike from run to run; STRAYDOG_PROMPT_CPU=0 streams them all.
• Streamed experts in fewer copies. The experts a prompt batch streams were copied a matrix at a time, three copies of about
  0.4 MB an expert; a run of consecutive experts is now one copy a matrix kind, and the bus moved 22 GB/s instead of 19 on the R9700:
  a 10,609-token prompt 6.1 -> 5.8 s.
• A verification's CPU work in one pass. With MTP drafts, the rows a pass verifies used to reach the CPU one at a time; now
  they go together, and an expert two rows share is read from memory once. Each row's result is the same as before, bit for
  bit. On an R9700 at 5.5 GiB with the drafter: 55.9 -> 57.2 tok/s.
• The request line splits the CPU's time into gate and up, the products and down, and the log names the CPU kernels at load
  (Straydog CPU experts: avx512-int8 kernels).
• The request line says what "other" is made of. A partially resident model's line follows the time a token spends outside
  the GPU wait and the CPU with its parts (head, first layer, PLE rows wait, PLE layer, the rest of the layers), timed by the
  host's clock without extra GPU synchronization. A request that verified drafts has no breakdown, and parts under 0.05 ms
  are left out.

Windows

• Reads in parallel. On Windows, reads from one file handle without FILE_FLAG_OVERLAPPED run one at a time, whatever the number
  of threads issuing them; the PLE rows, the expert reads, loading and the verity trees all shared one such handle. They now use an
  overlapped handle: twelve 4 KiB reads took 246 us instead of 1354, and a token's wait for its PLE rows fell from 1.1 to 0.8 ms.
• The worker is not slowed down as a background process. Windows 11 may lower the clocks of a process it judges to be in the
  background, which a worker without a window is. The worker now asks the system not to (ProcessPowerThrottling). On an RTX
  5070 Ti machine at 5.5 GiB: +2 to 8% tok/s.
• Qwen3.8's dense projections decode on 8-bit activations with CUDA too (dp4a: the kernels, the rounding and the order of
  every sum are those of the ROCm build's). On an RTX 5070 Ti the dense front of a layer takes 40 us instead of 56 (the card
  streams those bytes in 38), and served at 5.5 GiB the GPU's wait a token 8.5 -> 6.7 ms, 40-44 -> 44-49 tok/s. KL 1.1e-3
  against F32 (356 decode steps). STRAYDOG_Q8=0 keeps F32.
• The next start answers its first request at once. After the first reply, Straydog keeps the state of the request's system
  block (the system prompt and its tools) in its data folder, sealed for the user: AES-256-GCM under a key that Windows' DPAPI
  keeps, written to a temporary file and renamed, at most once a start and two files a bundle. The next start reads it back after
  it is ready (under a second), so a first request with the same system block does not read it again: on an RTX 5070 Ti with
  Qwen3.8 UD-IQ1_S at 5.5 GiB and a 4,048-token system block, the first token came after 1.63 s instead of 12.35 s.
  prefix_cache = false in the settings file, or --no-prefix-cache, turns it off. Linux keeps no such state.

Fixes

• Speed first, the engram table switch and STRAYDOG_CPU_INT8 reach a Linux worker. The sandbox (bubblewrap) cleared every
  variable not on its own list, and these were on the server's list alone: on Linux, --expert-floor and STRAYDOG_ENGRAM did
  nothing. One list now serves both systems' sandboxes and the server.

DeepSeek-V4.1

• Long prompts wait less for Engram rows. Before each pass of a long prompt Straydog reads the Engram rows of that pass and
  the next; the first Engram layer waited for both passes' rows. The current pass's rows are now read first: a fresh
  47.8K-token prompt's first token 24.7 -> 22.6 s on two Radeon AI PRO R9700s. Outputs are unchanged.

Dependencies

• rustls-webpki 0.103.16 and webpki-roots 1.0.11, past the certificate advisories RUSTSEC-2026-0382 and
  RUSTSEC-2026-0381. Straydog uses them when it downloads a bundle, and checks every file it downloads against its pinned
  SHA-256 as before.

Downloads

• straydog-v0.1.5-linux-x86_64-hip.tar.gz: the Linux executable with LICENSE, README.md and
  THIRD_PARTY_LICENSES.txt.
• straydog-linux-x86_64-hip: the Linux executable alone.
• straydog-v0.1.5-windows-x86_64-cuda.zip: the Windows executable with NVIDIA's cuBLAS DLLs (cublas64_13.dll,
  cublasLt64_13.dll, CUDA 13.3) and their license. It runs as is.
• straydog-windows-x86_64-cuda.exe: the Windows executable alone. It needs the CUDA 13 runtime's cuBLAS on PATH.
• THIRD_PARTY_LICENSES.txt: the notices and the licenses of every crate linked into either executable.
• SHA256SUMS: the checksums of these files.

The Linux build needs:

• x86_64 and glibc 2.39 or later (Ubuntu 24.04 or later);
• the ROCm 10 runtime (libamdhip64, hipBLAS, hipBLASLt);
• bubblewrap;
• an RDNA4 GPU (gfx1201: Radeon AI PRO R970...

Straydog v0.1.4

Straydog 0.1.4. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes.

Changes

Bundles

• A bundle is its model, provider and quantization. --model-name, --provider and --quant choose it, and setup --list
  shows all three. Earlier names are still read.
• Unsloth's Qwen3.8-Flash-Next bundles UD-IQ1_S, UD-IQ1_M, UD-IQ2_XS and UD-IQ3_XXS are pinned, and setup lists and
  recommends Unsloth's bundles first.
• Engram tables. A Qwen3.8 bundle reads its own engram table unless --engram names another (BF16, Q8_0, Q5_1); the
  ready line says which table it read.
• Setup counts the RAM other programs leave free, not the machine's total.

DeepSeek-V4.1

• A new conversation does not wait for the last one's checkpoint copies. After a long conversation, a new one began only
  once the earlier checkpoints were copied to host memory (up to 2.6 s before its first token); now it starts at once.
• Long prompts compute about 5% less: the index queries are prepared once per row, a layer's engram_kv expansion is kept
  for its attention batches, and linears of one input round it to FP16 once. Outputs are unchanged, bit for bit.

Speed first

• --expert-floor WEIGHT (the settings file's expert_floor) leaves out the routes a partially resident model would run on
  the CPU when their router weight is below WEIGHT. Off by default. At 0.04 the replies stay close (Qwen3.8 UD-IQ2_XS: KL 0.002
  against every route); it saves the reads of experts a machine short of RAM keeps on its drive.

Windows

• Partially resident models read cold experts ahead while the system is reading the drive for page faults: on a 16 GB
  RTX 5070 Ti with 62 GB of RAM, half of it held by other programs, warm-up decode 27.7 -> 36.0 tok/s and later requests
  +6 to 10%.
• Expert reads that end inside a file's last block no longer fail (os error 87).
• The load says how many layers' expert banks were page-locked, and why no more.

Other

• A worker that panics at start is reported with its panic.
• The request log line names the id a client sent in X-Request-Id.

Downloads

• straydog-v0.1.4-linux-x86_64-hip.tar.gz: the Linux executable with LICENSE, README.md and
  THIRD_PARTY_LICENSES.txt.
• straydog-linux-x86_64-hip: the Linux executable alone.
• straydog-v0.1.4-windows-x86_64-cuda.zip: the Windows executable with NVIDIA's cuBLAS DLLs (cublas64_13.dll,
  cublasLt64_13.dll, CUDA 13.3) and their license. It runs as is.
• straydog-windows-x86_64-cuda.exe: the Windows executable alone. It needs the CUDA 13 runtime's cuBLAS on PATH.
• THIRD_PARTY_LICENSES.txt: the notices and the licenses of every crate linked into either executable.
• SHA256SUMS: the checksums of these files.

The Linux build needs:

• x86_64 and glibc 2.39 or later (Ubuntu 24.04 or later);
• the ROCm 10 runtime (libamdhip64, hipBLAS, hipBLASLt);
• bubblewrap;
• an RDNA4 GPU (gfx1201: Radeon AI PRO R9700, RX 9070 series).

For other AMD GPUs, build from source with STRAYDOG_GPU_ARCHITECTURES.

The Windows build runs on NVIDIA GPUs only; AMD GPUs on Windows are not supported. It needs:

• x86_64 Windows;
• an NVIDIA driver that supports CUDA 13;
• an RTX 20, 30, 40 or 50 series GPU (75-real;86-real;89-real;120).

tar xzf straydog-v0.1.4-linux-x86_64-hip.tar.gz && cd straydog-v0.1.4-linux-x86_64-hip
./straydog setup    # choose a model and where it goes; downloads and authenticates it
./straydog          # serves http://127.0.0.1:8080

Known limitations

• Windows runs on NVIDIA GPUs only. AMD GPUs are supported on Linux (ROCm), not on Windows.
• DeepSeek-V4.1-Flash has run only on Linux with ROCm. Qwen3.8-Flash-Next runs on Windows with CUDA as well, down to 5-6 GB
  of VRAM and 32 GB of RAM.
• The released Linux executable carries GPU code for gfx1201 only.

License

MIT. THIRD_PARTY_LICENSES.txt holds code and data from other projects, and the licenses of the Rust crates linked into
the Linux and Windows executables; the source has the same notices in THIRD_PARTY_NOTICES.md. The Windows zip's cuBLAS
DLLs are NVIDIA's, under the license beside them. Model weights are not part of Straydog: each bundle is downloaded from its
publisher and is under that publisher's license.

Straydog v0.1.3 (pre-release)

Straydog 0.1.3. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes.

The executables of v0.1.0 and v0.1.1 named paths on the machine that built them. Those binaries were withdrawn, and v0.1.2
was withdrawn as well. This release replaces them and holds their fixes.

Changes

• Windows: a model on a second drive starts. The contained worker no longer opens and resolves the model and drafter
  directories itself; it takes the paths the server resolved. With the model on a second drive, the engine could stop at
  start with "Access is denied (os error 5)".

• Windows: a refused start says why. When the worker is refused access at start, the error names the cause: a drive
  that keeps no file permissions (exFAT, FAT32), or a path the worker has no read permission on.

• Release executables and archives name no builder paths. The GPU library's __FILE__ is relative to its folder, the
  release builds map the Rust dependencies' paths to ~, and the Linux tarball's files are owned by root.

• straydog setup --force. With --quant, it sets up the named bundle even when it needs more free VRAM than there is
  now, for example while other programs hold the GPU. It sets the smallest context and says so, and at load it waits for
  the VRAM. A machine with no GPU is still refused.

From v0.1.1:

• DeepSeek-V4.1-Flash tool calls with no arguments are read as calls, as DeepSeek's reference parser reads them.
• /props and /slots report DSpark's drafting.

Downloads

• straydog-v0.1.3-linux-x86_64-hip.tar.gz: the Linux executable with LICENSE, README.md and
  THIRD_PARTY_LICENSES.txt.
• straydog-linux-x86_64-hip: the Linux executable alone.
• straydog-v0.1.3-windows-x86_64-cuda.zip: the Windows executable with NVIDIA's cuBLAS DLLs (cublas64_13.dll,
  cublasLt64_13.dll, CUDA 13.3) and their license. It runs as is.
• straydog-windows-x86_64-cuda.exe: the Windows executable alone. It needs the CUDA 13 runtime's cuBLAS on PATH.
• THIRD_PARTY_LICENSES.txt: the notices and the licenses of every crate linked into either executable.
• SHA256SUMS: the checksums of these files.

The Linux build needs:

• x86_64 and glibc 2.39 or later (Ubuntu 24.04 or later);
• the ROCm 10 runtime (libamdhip64, hipBLAS, hipBLASLt);
• bubblewrap;
• an RDNA4 GPU (gfx1201: Radeon AI PRO R9700, RX 9070 series).

For other AMD GPUs, build from source with STRAYDOG_GPU_ARCHITECTURES.

The Windows build runs on NVIDIA GPUs only; AMD GPUs on Windows are not supported. It needs:

• x86_64 Windows;
• an NVIDIA driver that supports CUDA 13;
• an RTX 20, 30, 40 or 50 series GPU (75-real;86-real;89-real;120).

tar xzf straydog-v0.1.3-linux-x86_64-hip.tar.gz && cd straydog-v0.1.3-linux-x86_64-hip
./straydog setup    # choose a model and where it goes; downloads and authenticates it
./straydog          # serves http://127.0.0.1:8080

Known limitations

• Windows runs on NVIDIA GPUs only. AMD GPUs are supported on Linux (ROCm), not on Windows.
• DeepSeek-V4.1-Flash has run only on Linux with ROCm. Qwen3.8-Flash-Next runs on Windows with CUDA as well, down to 5-6 GB
  of VRAM and 32 GB of RAM.
• The released Linux executable carries GPU code for gfx1201 only.

License

MIT. THIRD_PARTY_LICENSES.txt holds code and data from other projects, and the licenses of the Rust crates linked into
the Linux and Windows executables; the source has the same notices in THIRD_PARTY_NOTICES.md. The Windows zip's cuBLAS
DLLs are NVIDIA's, under the license beside them. Model weights are not part of Straydog: each bundle is downloaded from its
publisher and is under that publisher's license.

Straydog v0.1.1 (pre-release)

Binaries withdrawn. The executables of this release named paths on the machine that built them, so they were removed on 2026-11-03. Use v0.1.2 or later.

A fix release of Straydog 0.1.0. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes.

Fixes

• DeepSeek-V4.1-Flash tool calls with no arguments. The model writes a call to a tool that takes no parameters with a
  blank line before its close, as the chat template does. Straydog dropped that call, so a client saw no call and the
  model's turn ended. It is now a call, as DeepSeek's reference parser reads it (a9cd367).
• /props and /slots while DSpark drafts. They said "speculative": false and "speculative.n_max": 0 for
  DeepSeek-V4.1-Flash with its DSpark drafter placed. They now report its 3 drafts a pass (160f161).
• The README says that Windows runs on NVIDIA GPUs only (0a9048b).

Downloads

• straydog-v0.1.1-linux-x86_64-hip.tar.gz: the Linux executable with LICENSE, README.md and
  THIRD_PARTY_LICENSES.txt.
• straydog-linux-x86_64-hip: the Linux executable alone.
• straydog-v0.1.1-windows-x86_64-cuda.zip: the Windows executable with NVIDIA's cuBLAS DLLs (cublas64_13.dll,
  cublasLt64_13.dll, CUDA 13.3) and their license. It runs as is.
• straydog-windows-x86_64-cuda.exe: the Windows executable alone. It needs the CUDA 13 runtime's cuBLAS on PATH.
• THIRD_PARTY_LICENSES.txt: the notices and the licenses of every crate linked into either executable.
• SHA256SUMS: the checksums of these files.

The Linux build needs:

• x86_64 and glibc 2.39 or later (Ubuntu 24.04 or later);
• the ROCm 10 runtime (libamdhip64, hipBLAS, hipBLASLt);
• bubblewrap;
• an RDNA4 GPU (gfx1201: Radeon AI PRO R9700, RX 9070 series).

For other AMD GPUs, build from source with STRAYDOG_GPU_ARCHITECTURES.

The Windows build runs on NVIDIA GPUs only; AMD GPUs on Windows are not supported. It needs:

• x86_64 Windows;
• an NVIDIA driver that supports CUDA 13;
• an RTX 20, 30, 40 or 50 series GPU (75-real;86-real;89-real;120).

tar xzf straydog-v0.1.1-linux-x86_64-hip.tar.gz && cd straydog-v0.1.1-linux-x86_64-hip
./straydog setup    # choose a model and where it goes; downloads and authenticates it
./straydog          # serves http://127.0.0.1:8080

Known limitations

• Windows runs on NVIDIA GPUs only. AMD GPUs are supported on Linux (ROCm), not on Windows.
• DeepSeek-V4.1-Flash has run only on Linux with ROCm. Qwen3.8-Flash-Next runs on Windows with CUDA as well, down to 5-6 GB
  of VRAM and 32 GB of RAM.
• The released Linux executable carries GPU code for gfx1201 only.

License

MIT. THIRD_PARTY_LICENSES.txt holds code and data from other projects, and the licenses of the Rust crates linked into
the Linux and Windows executables; the source has the same notices in THIRD_PARTY_NOTICES.md. The Windows zip's cuBLAS
DLLs are NVIDIA's, under the license beside them. Model weights are not part of Straydog: each bundle is downloaded from its
publisher and is under that publisher's license.