Skip to content

v0.1.3

Choose a tag to compare

@TheSylvester TheSylvester released this 31 May 09:05

Fixes

  • gpu: enable Flash Attention (-fa) for the embed server, the one-shot path, and the install probe. The 8192-token attention compute buffer is otherwise O(n²) (~3.7 GB on GPU → ~5.7 GB total), which OOMs cards <6 GB and forces CPU fallback. -fa makes it O(n) (~0.8 GB → ~2.9 GB total), so 4 GB GPUs now run on GPU at the full 8192 context. Embeddings are numerically identical (verified cosine 1.0 vs non-FA) and throughput improves (~26 → ~32 msg/s on an RTX 2060). The probe must mirror runtime, or it OOMs on a 4 GB card and falsely rejects a working GPU.

Builds on v0.1.2 (bin/ CUDA staging, real-device probe, embed resilience, codex array output). CUDA backend asset is byte-identical (llama.cpp b5300).