Skip to content

v0.25.2

Choose a tag to compare

@srvsngh99 srvsngh99 released this 21 Oct 05:24
· 38 commits to main since this release

Embedding servers rest light and stay under 1 GB. Embedding and reranker models now unload after the idle keep-alive (default 5 minutes, same keep_alive / KRILL_KEEP_ALIVE as chat), so a resting server drops to about 55 MB; the next request reloads in under a second. Requests are split into bounded sub-batches, long inputs run attention in blocks, and the image and audio towers unload on their own: the quantized EmbeddingGemma 2 builds now peak under 800 MB for bulk indexing (was 1.5-3 GB) and an 8,192-token input peaks at 1.2 GB instead of 5 GB. New max_length (or options.num_ctx) and KRILL_EMBED_MAX_TOKENS cap input length. The full-precision build indexes faster (29 passages/s, was 26) at half the memory. Vectors are unchanged.

See CHANGELOG.md for details (#340, #341).