Skip to content

Straydog v0.1.10

Pre-release
Pre-release

Choose a tag to compare

@rewroy rewroy released this 09 Nov 15:33

Straydog 0.1.10. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes; what 0.1.9 fixed is in its
notes.

This release is about waits and failures rather than steady speed: decode and prefill rates on the setups we measure are where 0.1.9
left them.

Shorter waits

  • A prompt stopped while it was being read keeps what it read (Qwen3.8). A request whose client left or was stopped during a long
    prefill used to throw away every batch it had read, and sending the same prompt again read it all from the start. The cancel is now
    seen between batches, and the next request that shares the prompt goes on from the last whole batch. A 32,483-token prompt cut at
    4,096 tokens, sent again: 22.5 s before, 17.1 s now (one R9700, partially resident). With two GPUs and the MTP drafter the request
    that goes on leaves the same tokens, logits and state as one that read the prompt whole.
  • DeepSeek-V4.1 starts sooner when memory is in pieces (Linux, AMD). After another large model's run had filled the page cache,
    page-locking the host tier while it was still being allocated made Linux move locked pages, and every move stopped the GPUs' queues.
    The host tier is now allocated in full before most of it is locked. Two R9700s, UD-IQ2_XXS: ready in 16 s instead of 29 to 38 s, and
    the queue stops in the first 150 s after ready cut from 15 to 27 s to 7 to 12 s.
  • Windows, partially resident: the read-ahead hints no longer hold up decode. The hints that ask Windows to read the CPU's experts
    ahead are now issued by a thread of their own. RTX 5070 Ti, Qwen3.8 UD-IQ1_M at 9.5 GiB: 42.7 tok/s before and 48 to 52 tok/s now
    while the hints are on; unchanged while they are off.
  • Prefill. An expert cache that starts empty is filled after the first batch has shown what the prompt routes to; the cache is
    left alone when the batch is already as wide as borrowing could make it; the first dense weights are counted before the expert
    slots are sized; on two GPUs, prompts of 64K tokens or more borrow wider workspaces and the caches come back on both GPUs at once.

Fixes

  • A prefill that failed and whose caches then could not be restored reports the first failure, not only the second.
  • Windows: a refused allocation or thread when the read-ahead thread starts is reported as an error instead of ending the process.

Diagnostics

  • Windows: a crash says what and where before the process ends, on stderr: Straydog crashed: access violation (0xC0000005) reading 0x10 at module.dll+0x1A2B3C. A program that runs Straydog and shows its last line now shows that instead of an exit code alone.
  • CUDA: the start says, for each GPU, whether it supports read-only host registration (the capability behind
    #2).
  • The dashboard shows the drive's reads, and labels the host expert budget and the model's weights separately in its memory figures.
  • The read-ahead hints switching on and off is logged only with --verbose; each request's line still counts them.