Skip to content

v0.9.0

Choose a tag to compare

@bluesky585 bluesky585 released this 19 Sep 16:12
· 7 commits to main since this release
7262cc6

This release completes the retrieval story around the chunker: index management, filtered search, and a faster, half-sized vector pipeline.

Source management (#67)

Indexed chunks can now carry a source — a file path, a URL, a caller-chosen key.

  • CLI: -source with -index, -list-sources, -delete-source
  • API: source on the index body, GET /v1/sources, DELETE /v1/sources/{name}
  • Chunks are keyed by text and offsets, so re-indexing a document under a new source moves it there — replace without rebuilding the index. Deleting to zero is reported as 0, not an error.

Source-filtered search (#68)

  • CLI: -source on -query; API: optional source on the search body.
  • The filter lands before scoring, so k counts filtered hits. Absent searches everything; an empty string matches the records indexed without a source.

float32 vectors end to end (#69) — breaking change for Go consumers

embed.Embedder.Embed now returns [][]float32, and Chunk.Embedding / Record.Vector are []float32. Scoring runs over the stored bytes directly: memory halves and the per-query float64 conversion is gone.

  • The SQLite file format is unchanged — existing databases open as they are. The JSONL and wire formats are unchanged; the TypeScript client needs no update.
  • The cosine scan gained 4-lane accumulators (the profiler showed accumulator serialization, not memory bandwidth, was the bound): 200k chunks at 1536 dimensions went 591ms → 297ms; 50k went 143ms → 81ms.
  • Ranking agreement with the previous float64 scoring was checked over 500 randomized trials — no top-5 disagreement.

rag-search example (#70)

go run ./examples/rag-search walks the retrieval lifecycle in one program: chunk with a real chunker, index into SQLite under per-document sources, hybrid search, source-filtered search, delete a source, and search again.

Full changes: v0.8.0...v0.9.0