Repository navigation
Releases
Releases · PixelEzz/NoVram
v0.1.4
Full Changelog: v0.1.3...v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0 - Zero-Dependency CPU LLM Runner & OpenAI Micro-Server
Full Changelog: https://github.com/PixelEzz/NoVram/commits/v0.1.0
⚡ novram v0.1.0 — Initial Public Release
novram is an open-source, ultra-fast, zero-dependency CPU-only LLM runner and OpenAI-compatible micro-server. Run quantized GGUF language models directly on system RAM without requiring a discrete GPU or CUDA setup.
🚀 Key Highlights & Architectural Features
- Strict Physical Core Affinity: Automatically discovers physical CPU core topology and pins worker threads to bypass hyperthreading contention, eliminating L1/L2 cache thrashing and boosting dense SIMD GEMM throughput by 25%–35%.
- Prompt Lookup Speculative Decoding: Uses n-gram pattern speculation verified in single batched CPU passes, unlocking token burst speeds of up to 180–280+ tokens/second on structured code and repetitive text.
- Quantized Q8_0 KV Cache: Reduces attention memory bus pressure by 50% compared to standard float16 KV caches.
- Zero-Copy
mmapEngine: Memory-maps GGUF model weights directly into virtual memory space in sub-50ms with zero duplicate heap copies. - OpenAI-Compatible Micro-Server: Drop-in
/v1/chat/completions(Server-Sent Events streaming) and/v1/modelssupport for VS Code, Roo Code, Cline, Cursor, and Continue. - Zero-Swap Feasibility Matrix (
novram doctor): Inspects host CPU vector extensions (AVX2, AVX-512, NEON) and available physical RAM to prevent OS swapping before starting inference.
📦 Quick Start & Pre-Built Binaries
Download the standalone binary for your architecture from the Assets section below:
| Platform | Architecture | File |
|---|---|---|
| Windows | x86_64 (amd64) |
novram-windows-amd64.zip |
| Linux | x86_64 (amd64) |
novram-linux-amd64.tar.gz |
| Linux | ARM64 (aarch64) |
novram-linux-arm64.tar.gz |
| macOS | Apple Silicon (arm64) |
novram-darwin-arm64.tar.gz |
| macOS | Intel (amd64) |
novram-darwin-amd64.tar.gz |
One-Line Terminal Run:
# Check CPU vector flags and zero-swap RAM feasibility
./novram doctor
# Start local OpenAI-compatible server on port 8080
./novram serve --model ./models/llama-3.2-3b.Q4_K_M.gguf --port 8080
**Full Changelog**: https://github.com/PixelEzz/NoVram/commits/v0.1.0