Skip to content

v0.8.0

Choose a tag to compare

View all tags
@clemlesne clemlesne released this 14 Jun 20:12
v0.8.0
16a11e9

Two big wins for steady-state cost. Zoekt delta indexing replaces full shard rebuilds, and a new cache GC keeps ~/Library/Caches/seek (or ${XDG_CACHE_HOME}/seek) bounded over time.

seek gc                 # opportunistic eviction (default)
seek gc --dry-run       # report only
seek gc --force         # ignore throttle gate
seek gc --full          # evict everything

Highlights

  • Zoekt IsDelta enabled for committed and uncommitted shards. Only changed blobs are touched per cycle; renames and removals land as tombstones in the .meta sidecar.
  • Committed indexer flips to delta with a 64-shard fallback threshold.
  • Uncommitted indexer mirrors the folder pattern: per-cycle manifest (path + size + mtime + inode) tagged with state hash, sorted-merge diff against working tree, tombstones on rename/delete.
  • New seek gc subcommand with --force, --dry-run, --full.
  • Opportunistic GC runs after each successful search, gated by throttle and NFS detection (skips network filesystems).
  • Suffix-only shard cleanup preserves Zoekt's contiguous shard numbering (fixes a latent folder-corpus bug too).
  • stateVersion bumps v5 → v6 to force a one-time clean re-index for upgraded binaries.

Performance

Measured on a real kubernetes/kubernetes clone (~17k Go files, 1.6 GB), M1 Max, n=3 medians, via SEEK_BENCH_REPO-gated benches:

Bench Before After Delta
LargeRepo_CommittedAdvance wall 8.97 s 2.43 s 3.7× faster
LargeRepo_CommittedAdvance RAM ~3.2 GB ~250 MB ~13× less
LargeRepo_CommittedAdvance allocs ~19 M ~320 k ~60× fewer
LargeRepo_UncommittedRealistic 97.3 ms 104.1 ms +7% (noise)
Cold index (warm page cache) 10.04 s 9.21 s −8%

Small-fixture benches (1-file, M1 Max, n=5 medians):

  • Win: PlannedGitCorpus_ColdIndex 312 → 272 ms (−13%), PlannedGitCorpus_WarmSearch 28.0 → 25.6 ms (−9%).
  • Regression: PlannedGitCorpus_DirtyReindex 53.6 → 62.0 ms (+16%), indexCommitted_incremental 306 → 364 µs (+19%), indexUncommitted_1file 26.3 → 27.8 ms (+6%), FolderCorpus_DirtyReindex_1File 30.1 → 39.6 ms (+32%).
  • New steady-state: GitCommitted_1CommitAhead 96 ms, GitUncommitted_RapidEdits_N16 912 ms (~57 ms/cycle), GitBranch_Switch_Unrelated 79 ms (80% churn), SearchTombstoneCost 859 µs over 16 delta shards.

Uncommitted is neutral on real repos: rebuilding one dirty file is dominated by ctags + Zoekt builder init (~30 ms), so manifest + per-shard probe overhead cancels the savings. Delta path still helps by keeping each shard small and bounding tombstone growth.

Development

  • New delta.go shared helpers, uncommitted_manifest.go for state hashing.
  • 18 correctness tests covering modify/delete/rename tombstoning, HEAD rewind, rebase, branch switch, threshold fallback, manifest corruption, dirty-to-committed transitions, drift mid-build, concurrent search.
  • 5 delta-focused benches plus LargeRepo_CommittedAdvance and LargeRepo_UncommittedRealistic (gated by SEEK_BENCH_REPO).
  • gc.go, gc_cmd.go, gc_test.go (1217 lines test coverage), platform-specific NFS detection (nfs_darwin.go, nfs_linux.go, nfs_other.go).

Notes

  • An earlier draft claimed "order-of-magnitude" committed-advance gain from extrapolation; actual measured speedup on kubernetes is 3.7×. A 20× number from prior runs came from a reflog oscillation bug (HEAD@{1} alternating between two distant positions). LargeRepo_CommittedAdvance now uses git rev-list --reverse HEAD~10..HEAD and asserts shard count grows under IsDelta so silent fallback cannot pass.

Full changelog: v0.7.1...v0.8.0

Assets 12
Loading