Skip to content

fidx v0.1.0 — local semantic search in one SQLite file

Latest
Latest

Choose a tag to compare

View all tags
@williamliu-ai williamliu-ai released this 04 Jul 15:15
· 28 commits to main since this release
v0.1.0
78c4094

fidx v0.1.0 — local semantic search in one SQLite file

Install: uv tool install fmdidx (or pipx install fmdidx) — the
installed command is fidx. The PyPI name fidx belongs to an unrelated
package, so fidx distributes as fmdidx ("fast markdown index").
macOS: use Homebrew Python (see the README — fidx doctor explains why).

What it is

Hybrid retrieval for your local files — SQLite FTS5 BM25 for exact names,
768-dim ONNX embeddings for "that doc that discussed X", fused with
reciprocal-rank fusion. No LLM in the query path: one embedding pass is the
only model work per query. The entire index (documents, BM25, vectors via
sqlite-vec) is a single SQLite file. CPU-only; the embedding model downloads
once, then fully offline. No telemetry.

Highlights

  • fidx collection add / index / search / get — incremental indexing,
    collection scoping, --json and --files agent-friendly output
  • fidx serve — warm unix-socket daemon; 18–49 ms hybrid queries (p50,
    single ONNX thread) on 2k–19k-doc corpora
  • fidx doctor — host capability checks with exact fix instructions
  • --truncate knee|mad|... deterministic result-list truncation and
    fidx calibrate corpus-adaptive score floors (both opt-in)
  • Verified on Linux x86_64, macOS arm64 (Homebrew Python), Windows x86_64
    via a wheel-install e2e matrix in CI

Benchmarks — reproducible, with the datasets attached

docs/BENCHMARKS.md
has the full methodology, recall+purity tables, and threats-to-validity.

The prepared text corpora and the exact 500-query sets used in the
benchmark are attached to this release so you can reproduce without the
fetch/build step (see fidx-bench-DATASETS.md for provenance/licensing):

  • fidx-bench-docs-small.tar.gz — 2,000-doc 20 Newsgroups subset
  • fidx-bench-docs.tar.gz — 18,821-doc 20 Newsgroups corpus
  • fidx-bench-chat.tar.gz — ~8,000 synthetic WhatsApp-style chat exports
    (Cornell Movie-Dialogs; no real chats)
  • fidx-bench-queries.tar.gz — every generated known-item + no-match query
    set (queries-*.jsonl)
  • SHA256SUMS

Quick start:

mkdir -p bench/data && tar xzf fidx-bench-docs-small.tar.gz -C bench/data
tar xzf fidx-bench-queries.tar.gz -C bench/data
uv run python bench/run_bench.py run --engine fidx --corpus docs-small

The 92k-file code corpus is not attached (it is 10 third-party repos at
pinned tags, ~4 GB) — uv run python bench/corpora.py rebuilds it
deterministically, and bench/README.md documents the pins.

Notes

  • License: MIT with an AI-attribution addendum (LICENSES/).
  • Trusted publishing: this release was built once in CI, e2e-tested on the
    exact artifact, and published to PyPI via OIDC — no API tokens exist.
Assets 12
Loading