What's New
⚡ Flash Attention
Memory-efficient fused attention kernel using online softmax (Dao et al., 2022). O(1) extra memory per head instead of O(seq_len). Critical for long-context inference on memory-constrained devices. Drop-in replacement with --flash-attention flag.
📊 Structured Benchmark Suite
Machine-readable JSON output for CI/CD regression testing. 6 scenarios: GGUF parse, cold/warm load, sequential streaming, forward pass estimation, cache budget analysis. Use ssd-llm bench model.gguf --json for automated performance tracking.
🏥 Health & Metrics API
Production monitoring endpoints:
GET /health— readiness probe (model status, uptime)GET /metrics— JSON metrics (throughput, latency p50/p95/p99, cache stats, SSD I/O)- Prometheus-compatible text output
Stats
- 74 tests, all passing
- ~8200 lines of Rust
- Full clippy-clean codebase
Next: v1.0
Production-ready release with end-to-end benchmarks against llama.cpp.