Skip to content

Straydog v0.1.4

Pre-release
Pre-release

Choose a tag to compare

View all tags
@rewroy rewroy released this 05 Nov 08:43
v0.1.4
144faf5

Straydog 0.1.4. It is still a pre-release. What Straydog does and how fast it runs are in the
v0.1.0 notes.

Changes

Bundles

  • A bundle is its model, provider and quantization. --model-name, --provider and --quant choose it, and setup --list
    shows all three. Earlier names are still read.
  • Unsloth's Qwen3.8-Flash-Next bundles UD-IQ1_S, UD-IQ1_M, UD-IQ2_XS and UD-IQ3_XXS are pinned, and setup lists and
    recommends Unsloth's bundles first.
  • Engram tables. A Qwen3.8 bundle reads its own engram table unless --engram names another (BF16, Q8_0, Q5_1); the
    ready line says which table it read.
  • Setup counts the RAM other programs leave free, not the machine's total.

DeepSeek-V4.1

  • A new conversation does not wait for the last one's checkpoint copies. After a long conversation, a new one began only
    once the earlier checkpoints were copied to host memory (up to 2.6 s before its first token); now it starts at once.
  • Long prompts compute about 5% less: the index queries are prepared once per row, a layer's engram_kv expansion is kept
    for its attention batches, and linears of one input round it to FP16 once. Outputs are unchanged, bit for bit.

Speed first

  • --expert-floor WEIGHT (the settings file's expert_floor) leaves out the routes a partially resident model would run on
    the CPU when their router weight is below WEIGHT. Off by default. At 0.04 the replies stay close (Qwen3.8 UD-IQ2_XS: KL 0.002
    against every route); it saves the reads of experts a machine short of RAM keeps on its drive.

Windows

  • Partially resident models read cold experts ahead while the system is reading the drive for page faults: on a 16 GB
    RTX 5070 Ti with 62 GB of RAM, half of it held by other programs, warm-up decode 27.7 -> 36.0 tok/s and later requests
    +6 to 10%.
  • Expert reads that end inside a file's last block no longer fail (os error 87).
  • The load says how many layers' expert banks were page-locked, and why no more.

Other

  • A worker that panics at start is reported with its panic.
  • The request log line names the id a client sent in X-Request-Id.

Downloads

  • straydog-v0.1.4-linux-x86_64-hip.tar.gz: the Linux executable with LICENSE, README.md and
    THIRD_PARTY_LICENSES.txt.
  • straydog-linux-x86_64-hip: the Linux executable alone.
  • straydog-v0.1.4-windows-x86_64-cuda.zip: the Windows executable with NVIDIA's cuBLAS DLLs (cublas64_13.dll,
    cublasLt64_13.dll, CUDA 13.3) and their license. It runs as is.
  • straydog-windows-x86_64-cuda.exe: the Windows executable alone. It needs the CUDA 13 runtime's cuBLAS on PATH.
  • THIRD_PARTY_LICENSES.txt: the notices and the licenses of every crate linked into either executable.
  • SHA256SUMS: the checksums of these files.

The Linux build needs:

  • x86_64 and glibc 2.39 or later (Ubuntu 24.04 or later);
  • the ROCm 10 runtime (libamdhip64, hipBLAS, hipBLASLt);
  • bubblewrap;
  • an RDNA4 GPU (gfx1201: Radeon AI PRO R9700, RX 9070 series).

For other AMD GPUs, build from source with STRAYDOG_GPU_ARCHITECTURES.

The Windows build runs on NVIDIA GPUs only; AMD GPUs on Windows are not supported. It needs:

  • x86_64 Windows;
  • an NVIDIA driver that supports CUDA 13;
  • an RTX 20, 30, 40 or 50 series GPU (75-real;86-real;89-real;120).
tar xzf straydog-v0.1.4-linux-x86_64-hip.tar.gz && cd straydog-v0.1.4-linux-x86_64-hip
./straydog setup    # choose a model and where it goes; downloads and authenticates it
./straydog          # serves http://127.0.0.1:8080

Known limitations

  • Windows runs on NVIDIA GPUs only. AMD GPUs are supported on Linux (ROCm), not on Windows.
  • DeepSeek-V4.1-Flash has run only on Linux with ROCm. Qwen3.8-Flash-Next runs on Windows with CUDA as well, down to 5-6 GB
    of VRAM and 32 GB of RAM.
  • The released Linux executable carries GPU code for gfx1201 only.

License

MIT. THIRD_PARTY_LICENSES.txt holds code and data from other projects, and the licenses of the Rust crates linked into
the Linux and Windows executables; the source has the same notices in THIRD_PARTY_NOTICES.md. The Windows zip's cuBLAS
DLLs are NVIDIA's, under the license beside them. Model weights are not part of Straydog: each bundle is downloaded from its
publisher and is under that publisher's license.

Assets 8
Loading