Skip to content

Releases: wici-ai/inferssd

Release list

v0.1.1

v0.1.1 Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 31 Oct 14:00
v0.1.1
d326616
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

This release fixes the Linux source install with recent CUDA toolkits.

Fixes

  • Build with CCCL 3.4 and later: the installer failed to compile with CCCL 3.4.0 and later, including the CCCL 3.4.3 bundled with CUDA 13.4.3 and CCCL 3.5.0 on Arch Linux. Thanks to @rokam for the report (#95).
  • inferssd profile after a one-command install: the command now finds the installed probes instead of reporting them as missing. Thanks to @HamedShams for the report (#97).

Install or upgrade

curl -fsSL https://github.com/wici-ai/inferssd/releases/download/v0.1.1/install.sh | sh

Run the same command to upgrade from v0.1.0. Downloaded models are not touched. The known limitations of v0.1.0 still apply.

Contributors

rokam and HamedShams
Assets 3
Loading

v0.1.0

v0.1.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 30 Oct 17:19
v0.1.0
b9cc72d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

First public release of InferSSD, which runs mixture-of-experts models larger than your RAM or VRAM from SSD. InferSSD builds on llama.cpp, reads GGUF weights directly, and serves browser chat and an OpenAI-compatible API.

Highlights

  • Models: DeepSeek-V4.1-Flash-1017 (Q8_0) and Muse-Spark-Open (UD-Q4_K_XL), with routed experts streamed from SSD through bounded RAM and VRAM caches. See the benchmarks.
  • Platforms: macOS on Apple Silicon with Metal, and Linux with NVIDIA CUDA on x86_64 and ARM64. Windows support is in progress.
  • One-command install and launch: the installer sets up a private environment, and inferssd cli and inferssd serve download a tested model and start it.
  • Automatic setup: the first launch measures the machine, and the engine sizes its caches from the memory available.

Install

curl -fsSL https://github.com/wici-ai/inferssd/releases/download/v0.1.0/install.sh | sh

This release installs from source, so the engine is compiled for your machine's CPU and GPU. Prebuilt packages are not available yet. Building takes about 2 to 12 minutes, depending on the machine.

The installer needs Git, Python 3.9 or later, and a C/C++ compiler. macOS also needs the Xcode Command Line Tools; Linux also needs the CUDA toolkit, liburing headers, and systemd with cgroup v2.

Then start a model. The first run downloads about 470 GiB of model files.

# Chat in your terminal
inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

# Or start the web UI and an OpenAI-compatible API at http://127.0.0.1:8080
inferssd serve -hf unsloth/Muse-Spark-Open-816B-A42B-GGUF

Known limitations

  • -hf accepts only the two tested repositories above. Use -m for local GGUF files.
  • If the CUDA runtime is outside the system library path, the first launch cannot measure the GPU, and prefill runs on the CPU.

License

InferSSD is licensed under Apache-2.0. Bundled llama.cpp/ggml and other third-party components keep their original licenses and notices.

Loading