Skip to content

Straydog v0.1.0 (pre-release)

Pre-release
Pre-release

Choose a tag to compare

View all tags
@rewroy rewroy released this 03 Nov 09:47
v0.1.0
33b45b3

Binaries withdrawn. The executables of this release named paths on the machine that built them, so they were removed on 2026-11-03. Use v0.1.2 or later.

The first public release of Straydog: a local inference engine with an OpenAI-compatible server, in one
executable. This is a pre-release. Options, the API's extensions and the pinned bundles may still change.

What it does

  • Two models, pinned. It runs DeepSeek-V4.1-Flash (748B total, 16B active) and Qwen3.8-Flash-Next from pinned GGUF bundles
    (ISTA's GSQ-RCO and Unsloth's builds). straydog setup looks at the machine, recommends what fits, then downloads and
    authenticates it.
  • OpenAI-compatible API. It supports streaming, reasoning, tool calls, response_format (JSON and JSON schema),
    tool_choice and images. Like llama-server, it also serves /props and /slots.
  • Every rounding named. Weights run in the format their GGUF ships, decoded exactly, and every routed expert is computed.
    Speculative decoding (DSpark, MTP, n-gram lookup) never changes an answer. The README's Numerics lists each speed-for-precision
    trade, with its switch.
  • Isolated. The engine runs in a worker with no network access (bubblewrap on Linux, an AppContainer on Windows). Every model
    file is pinned by SHA-256 and checked against an fs-verity tree.
  • Fast when the model does not fit. A model larger than VRAM keeps its routed experts in a store: VRAM slots over a
    page-locked host tier over the drive. Two GPUs split every layer between them. Conversation checkpoints let a returning
    conversation skip its prompt.
  • Dashboard at /dashboard: GPU, CPU and memory charts, speeds, and a chat. It is in English by default; Japanese is one
    click away, or /dashboard?lang=ja.

Measured

On two Radeon AI PRO R9700s with 125 GiB of RAM, with drafting:

Model Decode, 20K context First token
DeepSeek-V4.1-Flash GSQ-RCO-IQ2_XXS 74-80 tokens/s 0.3-0.6 s for a follow-up turn; 12-21 s for a fresh 20K prompt
Qwen3.8-Flash-Next IQ3_S 81-98 tokens/s 5.4 s for a fresh 20K prompt

Downloads

  • straydog-v0.1.0-linux-x86_64-hip.tar.gz: the Linux executable with LICENSE, README.md and
    THIRD_PARTY_LICENSES.txt.
  • straydog-linux-x86_64-hip: the Linux executable alone.
  • straydog-v0.1.0-windows-x86_64-cuda.zip: the Windows executable with NVIDIA's cuBLAS DLLs (cublas64_13.dll,
    cublasLt64_13.dll, CUDA 13.3) and their license. It runs as is.
  • straydog-windows-x86_64-cuda.exe: the Windows executable alone. It needs the CUDA 13 runtime's cuBLAS on PATH.
  • THIRD_PARTY_LICENSES.txt: the notices and the licenses of every crate linked into either executable.
  • SHA256SUMS: the checksums of these files.

The Linux build needs:

  • x86_64 and glibc 2.39 or later (Ubuntu 24.04 or later);
  • the ROCm 10 runtime (libamdhip64, hipBLAS, hipBLASLt);
  • bubblewrap;
  • an RDNA4 GPU (gfx1201: Radeon AI PRO R9700, RX 9070 series).

For other AMD GPUs, build from source with STRAYDOG_GPU_ARCHITECTURES.

The Windows build runs on NVIDIA GPUs only; AMD GPUs on Windows are not supported. It needs:

  • x86_64 Windows;
  • an NVIDIA driver that supports CUDA 13;
  • an RTX 20, 30, 40 or 50 series GPU (75-real;86-real;89-real;120).
tar xzf straydog-v0.1.0-linux-x86_64-hip.tar.gz && cd straydog-v0.1.0-linux-x86_64-hip
./straydog setup    # choose a model and where it goes; downloads and authenticates it
./straydog          # serves http://127.0.0.1:8080

Known limitations

  • Windows runs on NVIDIA GPUs only. AMD GPUs are supported on Linux (ROCm), not on Windows.
  • DeepSeek-V4.1-Flash has run only on Linux with ROCm. Qwen3.8-Flash-Next runs on Windows with CUDA as well, down to 5-6 GB
    of VRAM and 32 GB of RAM.
  • The released Linux executable carries GPU code for gfx1201 only.

License

MIT. THIRD_PARTY_LICENSES.txt holds code and data from other projects, and the licenses of the Rust crates linked into
the Linux and Windows executables; the source has the same notices in THIRD_PARTY_NOTICES.md. The Windows zip's cuBLAS
DLLs are NVIDIA's, under the license beside them. Model weights are not part of Straydog: each bundle is downloaded from its
publisher and is under that publisher's license.

Assets 3
Loading