Skip to content

Releases: mrsladoje/sweet-search

v2.6.9 — output-invariant performance pass (FTS5 once per build, merkle once per tick, stmt/view reuse)

Choose a tag to compare

@mrsladoje mrsladoje released this 03 Jul 10:18

Performance pass — zero behavioral change

Every change in this release is output-invariant: same index, same ss-* tool output, just faster and leaner. Verified with a three-layer parity protocol on a 226-file fixture corpus — the post-change build's logical dump (vector rows incl. embedding-blob hashes, entities, relationship content, FTS5 integrity-check, 12 MATCH probes) is exactly identical to a pre-change same-path rerun, and an 18-command ss-search/ss-grep/ss-find/ss-semantic/ss-read/ss-trace battery against a frozen index is byte-identical (only latency/pid telemetry fields move).

Indexing

  • FTS5 mirrors rebuilt once per graph build instead of once per 100-file batch. 'rebuild' reconstructs the whole external-content index, so per-batch rebuilds were O(entities × batches) work discarded by the next batch. Code Graph phase 4.1s → 1.9s on the parity corpus; the win grows with repo size.
  • File discovery stats and the full-reindex hash pass now run in order-preserving batches (discovery 126ms → 45ms, state save 57ms → 33ms).
  • Duplicate per-chunk embedding-text build eliminated (li_greedy_text reuses embedding_text under the production variant); the full-index path skips the never-persisted dedup fingerprint; LI skip-policy globs are precompiled.

Incremental reconcile

  • merkle-state.json parsed once per tick instead of ~100× (it was re-read per dirty file + per modified file + per cutoff-skip; multi-MB on real repos, every tick).
  • Fixed-SQL statements compiled once per tick connection via a new per-connection prepareCached helper.

Query path

  • Prepared-statement reuse across all 17 code-graph repository call sites — ss-trace 2.2–2.7× faster in the parity battery (256/378/258ms → 117/147/97ms).
  • Hoisted/memoized per-result work in boost, fusion, MMR, and file-kind ranking; identifier-anchored retrieval now buckets LI chunks by file (O(entities × chunks) → O(chunks + entities)); ss-semantic reads the reconcile manifest once per request (was 3×).

Vector / storage layer

  • WASM memory views cached in all 7 distance kernels (was a fresh typed-array allocation per Hamming/int8/MaxSim call — millions per HNSW build).
  • FloatVectorStore.load adopts the read buffer instead of copying the whole (~99MB on large repos) vector file; HNSW search heaps pooled; tombstone bitmap loads as a view.
  • Native maxsim_score_batch(+pertoken): Buffer→Vec<i8> is now a single bit-identical memcpy — ships in this release's napi builds.

Verification

  • 7,060-test suite: the only full-suite failures are pre-existing daemon-lifecycle flakes that reproduce (worse) on a clean-HEAD worktree under identical load; all affected files pass standalone on both trees.
  • Parity artifacts (logical dumps, noise-envelope calibration runs, query-battery outputs) captured before/after on identical corpora.

No index rebuild required — existing indexes are fully compatible.

v2.6.8 — index Zeek DSLs + secondary source/DSL/template/build extensions

Choose a tag to compare

@mrsladoje mrsladoje released this 02 Jul 13:18

Extension-coverage widening

An extension-coverage audit of all 200 SWE-rebench-V2 task-bench repos — every repo's file tree (at exact base_commit) and every gold-patch fix file cross-checked against the production indexing admission gate — found exactly one task whose entire bug fix lived in a file sweet-search could not index: zeek-1734, fixed in src/broker/comm.bif. .zeek/.bro were allowlisted (~2.6.2), but Zeek's build-time DSLs were not.

Added to the indexing allowlist (FILE_PATTERNS.include + EXTENSION_MAP/FILENAME_MAP)

  • Zeek DSLs: .bif .pac .pac2 (→zeek), .spicy .evt .hlt (→spicy)
  • Cython: .pyx .pxd .pxi (→python)
  • Scala build/worksheets: .sbt .sc (→scala)
  • Ruby build/manifest: .rake .gemspec .podspec .ru (→ruby), Rakefile, Gemfile
  • CoffeeScript; .pcss / .styl
  • Templating: twig liquid njk hbs handlebars mustache gohtml eex heex leex (→html); pug jade haml slim; jinja jinja2 j2 tera tmpl tpl gotmpl (generic)
  • .NET / MSBuild: .props .targets .proj .vbproj .fsproj .resx .nuspec (→xml)
  • Autotools / scripts: .m4 .ac (→m4), .am (→makefile), .awk, .sed, *.in
  • Misc: .sln, go.mod, go.work, .ipynb (generic fixed-window chunking; image-heavy notebooks self-filter via the 1 MB size gate)

Notes

  • Discovery-only change (adds files to the crawl) — not a ranking signal, so it cannot regress GCSN.
  • Post-change re-check: full-blind fix tasks 0/200 (was 1); 157 fully clean.
  • .mp4 / .parquet / .csv / .png and other binary/data types remain rejected.
  • ⚠️ Existing indexes must be rebuilt for the new coverage to take effect.