Straydog v0.1.7

Straydog 0.1.7. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes.

This release reads long prompts faster on two GPUs, holds Qwen3.8's K/V in half the memory, and says at the start what a little
more VRAM would have bought.

Changes

Faster long prompts on two GPUs (Qwen3.8)

• A 95K-token prompt on two Radeon AI PRO R9700s (UD-IQ3_XXS, MTP drafter, prefill batch 2048) reads in 23.3 s, 25.5 s before; a
  32K one in 8.2 s, 8.9 s before. At the batch the placement takes itself (4096 at 262144 tokens, --vram 30), a 95K prompt reads
  in 22.4 s.
  • A DeltaNet layer's scan spreads a head's columns over four blocks and stages 16 tokens at a time: 1.9 times as fast, the same
    bits.
  • The two GPUs' exchanges move 16 bytes a thread. A scatter's residual update also normalizes the rows for the HyperConnection
    after it, whose projections then read FP16 values only. The MTP layer reads a prompt batch at once rather than 512 rows at a
    time.
  • These change no bit of the prompt's outputs; the MTP layer's change moves only how many drafts are accepted.

Qwen3.8's K/V in FP16

• The K/V holds FP16 values: half the bytes of F32 in VRAM, in host memory, in the VRAM stage and its mirrors (which hold twice the
  positions) and in a conversation's saved positions. With its K/V in host memory at 262144 tokens, a two-GPU placement's state
  falls from 14.4 to 7.4 GB.
• A long prompt's attention rounded keys and values to FP16 already; decode and verification now read them so. Against F32 on
  UD-IQ3_XXS, each row's logits on 255-token replies: KL mean 3.6e-4, 4.6e-4 and 6.6e-4 at 8K, 32K and 95K tokens, the same top token
  on 762 of 765 rows (the others near ties). A 4-row verification pass at 95K takes 22.85 ms, 24.09 ms before.

The ready line says what more VRAM would buy

• Qwen3.8's placement takes the first of: full residency with the K/V in VRAM, then with the K/V in host memory at prefill batches of
  4096 down to 1024, with the MTP drafter and then without, then partial residency. A few hundred MB decide which, and the speed
  with it: on two R9700s an 8K prompt takes 2.5 s fully resident at --vram 30, 6.0 s partially resident at --vram 26. When the
  first choice did not fit, the ready line now says what it would have needed, for example prefill batch 4096 (K/V in VRAM, prefill batch 4096, MTP drafter would need 2.6 GiB more VRAM), or with partial residency (fully resident: K/V in host memory, prefill batch 1024 would need 2.3 GiB more VRAM).

Machines short of VRAM and RAM

• straydog setup adds Qwen3.8's MTP drafter by default from a 5 GiB budget (it was 5.5), and --mtp adds it whatever the
  budget. On an RTX 5070 Ti at 5.5 GiB with UD-IQ1_S, 41.9-49.1 tokens/s without the drafter, 53.9-60.0 with it.
• Memory freed after the start is used. A partially resident model planned its page-locked layers, and under --vram auto its
  VRAM budget, once at the start. After each request it now locks the layers the memory available has room for, and under auto
  raises its budget and places its expert slots again. An RTX 5070 Ti started beside a program holding 1 GiB of VRAM went from 672
  to 1021 slots once it was gone (44.9 tokens/s, then 52.2 and 48.0).
• CUDA: a decode step's head waits once. The mix, the output projection and a check of the logits run in one queue with one wait:
  the step's head 1.11 ms to 0.89 (RTX 5070 Ti, UD-IQ1_S).
• CUDA: a partially resident layer routes in one kernel. The router, the shared expert's gate, the top-10 and the export to the
  CPU were four launches; the GPU's wait for the layers falls from 6.93 to 6.51 ms a token.
• The request line splits the GPU's wait by the GPU's own clock: GPU wait 8.8 (the GPU's layer work 8.2, its experts 7.6 beside the CPU).

Fixes

• CUDA: a partially resident model no longer fails after its expert slots are placed again. Raising the budget or ending a long
  prompt allocated the slots anew while captured graphs still held their old address; the next request ended with an illegal memory
  access. The graphs are dropped and captured again.
• The VRAM budget's changes and the warning that no more layers could be locked reach the log without --verbose.