Straydog v0.1.8

Straydog 0.1.8. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes.

This release reads DeepSeek-V4.1 prompts much faster when VRAM and host memory only just hold the model, uses DeepSeek's DSpark drafter
with every bundle, and reads long conversations with their K/V in host memory without falling off a cliff past the VRAM stage.

Changes

DeepSeek-V4.1 when the model only just fits

On two Radeon AI PRO R9700s, Unsloth UD-IQ2_XXS keeps 5,003 experts in VRAM and the rest in a 102 GiB host tier: the two hold the model
with almost nothing to spare. Prompts there waited on the bus and on the drive far more than they had to.

• A fresh 7.8K-token prompt reads in 5.6 s (7.6-8.8 s before), five agent turns of 1.6-5.8K new tokens in 15.1 s (17.3-17.9 s), and a
  fresh 93K-token prompt in 50.8 s (87.6 s with only the first of these changes in).
• No more write-back chains. While a prompt streamed a layer's experts through VRAM, a full host tier gave a displaced resident the
  host copy of an expert being streamed, whose release then had to be written back too, and so on to the end of the pass: 17,696 of a
  7.8K prompt's 19,804 write-backs, and 3.3 s of waiting. A write-back now never takes a streamed expert's copy.
• What is let go of is what the prompt wants last. With no room left, the store lets go of experts of the layers the pass has done
  (first those it did not route to, then the latest), never of the layers still to come: a 93K prompt reads 74 GB from the drive, 283 GB
  before.
• Linux no longer stops the GPUs. The host tier is page-locked for the GPUs, and each time Linux moved one of its pages (compaction,
  khugepaged, swap) the driver stopped every GPU queue of the process, for 0.03-2.5 s: some 20 s over a 93K prompt and the turns before
  it. The host tier now takes huge pages (compaction passes them by), and other allocations no longer ask for huge pages (asking had Linux
  compact memory whenever it was short): 0.35 s. Loading takes 12 s, 17 s before.
• These change no bit of what is computed. UD-IQ1_M (with more room to spare) is as fast as before or faster: five agent turns 17.4 s,
  code 78 tokens/s.

DSpark with every DeepSeek-V4.1 bundle

• The DSpark drafter was used only with the bundles listed beside it; any DeepSeek-V4.1 bundle now uses it when its dspark/ folder is
  there. UD-IQ2_XXS: code 45.2 to 55.7 tokens/s, thinking 44.5 to 60.8.
• Its three stages keep 48, 28 and 16 of their most used experts in VRAM (87, 50 and 29 before): the drafts are kept about as often,
  and the target gets 138 more expert slots (UD-IQ2_XXS code 51.7 to 55.7 tokens/s). DS41_DSPARK_RESIDENT still sets them.

Long conversations with the K/V in host memory

• Past the VRAM stage (some 45K tokens at a 6 GiB budget), each token of a prompt batch read its 2,052 selected positions straight from
  host memory: an RTX 4060 read a token of a 48K conversation in 52-68 ms against 4.3 ms below the stage. A batch now copies the union
  of its selections into the stage once and reads them there, the same bits, on CUDA and on RDNA4's batched attention alike.

Long prompts on two GPUs (Qwen3.8)

• A prompt batch's rows go through page-locked buffers of their own, so the GPUs no longer wait for each other between batches: a 95K
  prompt on two R9700s (UD-IQ3_XXS, --vram 30) 21.5 to 21.0 s, a 52K one on an RTX 5070 Ti at 5.5 GiB 61.9 to 52.5 s, the same bits.
  (Served, a 95K prompt took 22.5 s with 0.1.7's huge pages for every allocation, and takes 22.7 s now.)
• On an RTX 5070 Ti at 5.5 GiB (UD-IQ1_S, K/V in host memory), a 42K-token conversation's first prompt reads in 31.3 s and each
  640-token turn after it, up to 48K, in 1.2-1.4 s (39.6 s and 1.7 s with only the K/V stage's fix above).

Drafts by what a pass costs (partially resident models)

• How many drafts pay depends on the machine: where the GPU's part of a layer is the longer, a second and third row cost little; where
  the CPU's experts are the limit, each row doubles their routes. A partially resident model now times its passes by their rows and
  takes the width that is fastest by measure, and says so: Straydog drafts: 0 18.2 ms/token, 1 16.9, 2 17.4 (accepted 0.93, 0.71); keeping 1.

Linux with NVIDIA GPUs (new; tested under WSL2)

• A Linux executable for NVIDIA GPUs (CUDA 13), straydog-v0.1.8-linux-x86_64-cuda.tar.gz. It needs the NVIDIA driver, the CUDA 13
  runtime (libcublas.so.13 and libcublasLt.so.13 in /usr/local/cuda/lib64, /usr/local/cuda-13/lib64 or wherever ldconfig
  lists them), bubblewrap and glibc 2.39 or later (Ubuntu 24.04 and later, as the ROCm build), and it is built for RTX 20 to 50 series
  GPUs (as the Windows build; data-center GPUs need a build of their own). It has been run under WSL2 (Ubuntu 24.04, RTX 5070 Ti): the GPU tests as on Windows, the same logits bit
  for bit as the Windows build on a 32K-token prompt, Qwen3.8 and DeepSeek-V4.1 served isolated. Nothing in it is WSL-only: the
  isolated worker is given the NVIDIA device nodes the host has (/dev/nvidia*, and /dev/dxg under WSL2), so it is meant for
  native Linux as well, where it has not been run yet.

Windows: the conversation kept across a restart (off by default)

• With --keep-conversation (or keep_conversation = true), a server stopped on purpose seals the state of the conversation it held
  last for the Windows user (AES-256-GCM, its key wrapped with DPAPI) and the next start takes that conversation's next request from
  there. A 39K-token conversation went on with its first token in 2.2 s, against 48.9 s read again (RTX 5070 Ti, UD-IQ1_S).

Fixes

• DeepSeek-V4.1: lending the cards' spare VRAM to the expert store could use up the budget, and the next request that allocated a buffer
  failed for good. Lending leaves 128 MiB; a budget error says what was asked and what was left.
• A frame's body crosses the worker's pipe in 4 MiB reads (a kept conversation's 1 GB took 7.5 s in 4 KiB ones).