Skip to content

Straydog v0.1.5

Pre-release
Pre-release

Choose a tag to compare

View all tags
@rewroy rewroy released this 06 Nov 02:56
v0.1.5
5e83c53

Straydog 0.1.5. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes.

Changes

Qwen3.8 on machines short of VRAM (partially resident)

  • Ready sooner. A partially resident model now says it is ready before it page-locks its host banks and warms its GPU code; both
    go on behind, and the GPU's share of host experts turns on layer by layer as the banks are locked. On a Radeon AI PRO R9700 with
    UD-IQ1_S at 5.5 GiB, ready came after 3.0 s instead of 17.2 s; on an RTX 5070 Ti, 3.5-3.8 s instead of 12.9 s. The locking
    runs at a low priority and pauses between layers while a request generates (for at most 20 s at a time).
  • A cold start reads its banks ahead. Right after ready, Straydog asks the system to read the banks it is about to lock, 256 MiB
    at a time at a low priority, ahead of the locking, so a first request does not wait on the drive page by page. On the R9700 with
    the model's pages out of the page cache, a request 12 s after ready: first token 1.67 -> 0.65 s, 33.3 -> 48.6 tok/s (49 warm),
    and the locking was over 31 s after ready instead of 39. On an RTX 5070 Ti, 33-39 -> 43-47 tok/s. On Windows the reading and the
    locking keep their memory priority: at the background priority Windows put the pages they read first in line to be dropped.
  • The GPU's share of host experts is chosen by timing. A partially resident layer can have the GPU read a few of the
    experts kept in page-locked host memory instead of leaving them to the CPU. How many was fixed, which could slow a machine with
    a narrow bus (a PCIe 4.0 x8 card). Now Straydog times the decode passes at each level (0, 2 or 4 a layer with ROCm; 0, 1
    or 2 with CUDA) and moves to another level only when it is at least 2% faster by more than twice the measurements' noise;
    the log's Straydog assist: line gives the times. On a Radeon AI PRO R9700 it keeps
    4, as before (45.1 tok/s against 44.3).
  • A warm start. Straydog counts which experts each layer routes to and keeps the counts beside its other state, one file
    per bundle (written after requests, at most every 120 s, and at exit). At the next start it fills the GPU's expert slots
    from the counts and asks the system to read the most-used experts' pages ahead. On an RTX 5070 Ti with UD-IQ1_M, a first
    request sent 12 s after the start (as a person types) began 30% sooner (4.00 -> 2.81 s) and decoded 10% faster
    (34.8 -> 38.5 tok/s); one sent within a second of the start gained little (first token 4.06 s). Later requests were no
    slower.
  • The CPU's experts are computed on 8-bit activations (AVX-512 with VBMI: Ryzen 7000 and later, Intel's 11th generation and
    others; or AVX2: Intel's 12th to 14th generation, Zen 3): activations rounded to 8-bit integers a block, exact integer products,
    one F32 scale a block, as llama.cpp's CPU path and Straydog's GPU GEMVs do. For IQ2_XXS, IQ2_S and Q2_0, the formats of
    Unsloth's UD-IQ1_S bundle. Where the CPU has VNNI (AVX512-VNNI: Intel's 10th and 11th generation, Zen 4 and 5; AVX-VNNI: Intel's
    12th generation on, Zen 5), its dot instructions take the products, with the same results (the log names avx512-vnni-int8). On a Ryzen 5 7500F, six threads, experts cold: 48.0 -> 35.2 us a route on its IQ2_XXS layers,
    46.0 -> 36.9 us on its IQ2_S layers; served at 5.5 GiB of VRAM, 48.4 -> 50.4 tok/s with the CPU's time a token 9.7 -> 8.4 ms.
    Held to AVX2, served: 42.1 -> 48.4 tok/s. On a Ryzen 7 9700X a route takes 32.6 us instead of 48.8. VNNI on top: on the 7500F
    the CPU's time a token 8.7 / 8.2 -> 8.2 / 7.8 ms; on a Ryzen 7 9700X with an RTX 5070 Ti, 11.9 -> 9.6 ms, 44.7-47.3 -> 50.2-51.9 tok/s. The replies change by KL
    0.0022 against F32 (two F32 runs differ by 0.0009). Other formats keep F32, and STRAYDOG_CPU_INT8=0 keeps it everywhere.
  • Routers in half the bytes. Unsloth's bundles store Qwen3.8's routers in F32, though every value is a BF16 value; they now go
    up as BF16, the same values, so a decode step reads 126 MB less, and the VRAM they free holds more experts (UD-IQ1_S at
    5.5 GiB: 953 -> 1040 expert slots). This helps cards whose memory, not their CPU, is the limit.
  • A prompt's rarely used experts on the CPU. A prompt batch streamed every expert the GPU did not hold over the bus, whatever
    its rows, while the CPU waited. Now the experts with the fewest rows go to the CPU, which computes their routes while the bus
    streams the rest; how many is set by how the last batches' CPU and GPU finished, so that both end about together. The request
    line counts them (prompt streamed N experts and the CPU M routes). On the R9700 at 5.5 GiB, prompts of 2,101, 2,397 and 3,394
    tokens: 2.1 -> 1.7 s, 2.1 -> 1.7 s and 2.4 -> 2.1 s; on an RTX 5070 Ti, 5-7% faster. Which experts go where depends on timing, so
    such a prompt is not computed alike from run to run; STRAYDOG_PROMPT_CPU=0 streams them all.
  • Streamed experts in fewer copies. The experts a prompt batch streams were copied a matrix at a time, three copies of about
    0.4 MB an expert; a run of consecutive experts is now one copy a matrix kind, and the bus moved 22 GB/s instead of 19 on the R9700:
    a 10,609-token prompt 6.1 -> 5.8 s.
  • A verification's CPU work in one pass. With MTP drafts, the rows a pass verifies used to reach the CPU one at a time; now
    they go together, and an expert two rows share is read from memory once. Each row's result is the same as before, bit for
    bit. On an R9700 at 5.5 GiB with the drafter: 55.9 -> 57.2 tok/s.
  • The request line splits the CPU's time into gate and up, the products and down, and the log names the CPU kernels at load
    (Straydog CPU experts: avx512-int8 kernels).
  • The request line says what "other" is made of. A partially resident model's line follows the time a token spends outside
    the GPU wait and the CPU with its parts (head, first layer, PLE rows wait, PLE layer, the rest of the layers), timed by the
    host's clock without extra GPU synchronization. A request that verified drafts has no breakdown, and parts under 0.05 ms
    are left out.

Windows

  • Reads in parallel. On Windows, reads from one file handle without FILE_FLAG_OVERLAPPED run one at a time, whatever the number
    of threads issuing them; the PLE rows, the expert reads, loading and the verity trees all shared one such handle. They now use an
    overlapped handle: twelve 4 KiB reads took 246 us instead of 1354, and a token's wait for its PLE rows fell from 1.1 to 0.8 ms.
  • The worker is not slowed down as a background process. Windows 11 may lower the clocks of a process it judges to be in the
    background, which a worker without a window is. The worker now asks the system not to (ProcessPowerThrottling). On an RTX
    5070 Ti machine at 5.5 GiB: +2 to 8% tok/s.
  • Qwen3.8's dense projections decode on 8-bit activations with CUDA too (dp4a: the kernels, the rounding and the order of
    every sum are those of the ROCm build's). On an RTX 5070 Ti the dense front of a layer takes 40 us instead of 56 (the card
    streams those bytes in 38), and served at 5.5 GiB the GPU's wait a token 8.5 -> 6.7 ms, 40-44 -> 44-49 tok/s. KL 1.1e-3
    against F32 (356 decode steps). STRAYDOG_Q8=0 keeps F32.
  • The next start answers its first request at once. After the first reply, Straydog keeps the state of the request's system
    block (the system prompt and its tools) in its data folder, sealed for the user: AES-256-GCM under a key that Windows' DPAPI
    keeps, written to a temporary file and renamed, at most once a start and two files a bundle. The next start reads it back after
    it is ready (under a second), so a first request with the same system block does not read it again: on an RTX 5070 Ti with
    Qwen3.8 UD-IQ1_S at 5.5 GiB and a 4,048-token system block, the first token came after 1.63 s instead of 12.35 s.
    prefix_cache = false in the settings file, or --no-prefix-cache, turns it off. Linux keeps no such state.

Fixes

  • Speed first, the engram table switch and STRAYDOG_CPU_INT8 reach a Linux worker. The sandbox (bubblewrap) cleared every
    variable not on its own list, and these were on the server's list alone: on Linux, --expert-floor and STRAYDOG_ENGRAM did
    nothing. One list now serves both systems' sandboxes and the server.

DeepSeek-V4.1

  • Long prompts wait less for Engram rows. Before each pass of a long prompt Straydog reads the Engram rows of that pass and
    the next; the first Engram layer waited for both passes' rows. The current pass's rows are now read first: a fresh
    47.8K-token prompt's first token 24.7 -> 22.6 s on two Radeon AI PRO R9700s. Outputs are unchanged.

Dependencies

  • rustls-webpki 0.103.16 and webpki-roots 1.0.11, past the certificate advisories RUSTSEC-2026-0382 and
    RUSTSEC-2026-0381. Straydog uses them when it downloads a bundle, and checks every file it downloads against its pinned
    SHA-256 as before.

Downloads

  • straydog-v0.1.5-linux-x86_64-hip.tar.gz: the Linux executable with LICENSE, README.md and
    THIRD_PARTY_LICENSES.txt.
  • straydog-linux-x86_64-hip: the Linux executable alone.
  • straydog-v0.1.5-windows-x86_64-cuda.zip: the Windows executable with NVIDIA's cuBLAS DLLs (cublas64_13.dll,
    cublasLt64_13.dll, CUDA 13.3) and their license. It runs as is.
  • straydog-windows-x86_64-cuda.exe: the Windows executable alone. It needs the CUDA 13 runtime's cuBLAS on PATH.
  • THIRD_PARTY_LICENSES.txt: the notices and the licenses of every crate linked into either executable.
  • SHA256SUMS: the checksums of these files.

The Linux build needs:

  • x86_64 and glibc 2.39 or later (Ubuntu 24.04 or later);
  • the ROCm 10 runtime (libamdhip64, hipBLAS, hipBLASLt);
  • bubblewrap;
  • an RDNA4 GPU (gfx1201: Radeon AI PRO R9700, RX 9070 series).

For other AMD GPUs, build from source with STRAYDOG_GPU_ARCHITECTURES.

The Windows build runs on NVIDIA GPUs only; AMD GPUs on Windows are not supported. It needs:

  • x86_64 Windows;
  • an NVIDIA driver that supports CUDA 13;
  • an RTX 20, 30, 40 or 50 series GPU (75-real;86-real;89-real;120).
tar xzf straydog-v0.1.5-linux-x86_64-hip.tar.gz && cd straydog-v0.1.5-linux-x86_64-hip
./straydog setup    # choose a model and where it goes; downloads and authenticates it
./straydog          # serves http://127.0.0.1:8080

Known limitations

  • Windows runs on NVIDIA GPUs only. AMD GPUs are supported on Linux (ROCm), not on Windows.
  • DeepSeek-V4.1-Flash has run only on Linux with ROCm. Qwen3.8-Flash-Next runs on Windows with CUDA as well, down to 5-6 GB
    of VRAM and 32 GB of RAM.
  • The released Linux executable carries GPU code for gfx1201 only.

License

MIT. THIRD_PARTY_LICENSES.txt holds code and data from other projects, and the licenses of the Rust crates linked into
the Linux and Windows executables; the source has the same notices in THIRD_PARTY_NOTICES.md. The Windows zip's cuBLAS
DLLs are NVIDIA's, under the license beside them. Model weights are not part of Straydog: each bundle is downloaded from its
publisher and is under that publisher's license.

Assets 8
Loading