v0.5.0: two-phase compute/store indexing, per-chunk prepare, FilterByRatio cost floor

A minor bump because one signature breaks: FilterByRatio gains a parameter.
Everything else is additive, and the fused AddFile/Reindex/AppendChunks
family behaves as before.

Breaking:

• FilterByRatio(maxRatio) becomes FilterByRatio(maxRatio, minCount)
  (9310fba, R675). A trigram is skipped only when it is both
  non-discriminating (over maxRatio) and expensive to scan (at least minCount
  postings). minCount of 0 or 1 reproduces the old behaviour exactly.

New:

• Two-phase indexing (3bb59d5, R676-R689). PrepareFile/PrepareContent/
  PrepareAppend compute chunks without a transaction, from any goroutine;
  StorePrepared/ReindexPrepared/AppendPrepared store them. The opaque
  single-use handles PreparedFile and PreparedAppend join the two. The store
  allocates fileids, derives the reindex removal set in its own transaction,
  and rejects an append whose tail moved (ErrAppendTailMoved).
• Per-chunk compute (06492ec, R690-R699). SplitFile/SplitContent/SplitAppend
  return SplitChunk{Chunk, Hash} plus FileMeta; PrepareChunk computes one
  chunk's grams, content-pure and cacheable by hash; AssembleFile/
  AssembleAppend package PreparedEntry values into the handles the store
  accepts. Positions always come from the entry's chunk, so reused grams are
  safe across files and scans.
• SplitAppend takes AppendOption and honours WithBaseLine (8362b4a, R700), so
  appended chunks get absolute line ranges. Existing four-argument callers are
  unchanged.

Also: mini-spec config moves to .minispec/config.toml.

v0.4.4: deleted-feature registry, retired bigram spec restored

v0.4.4: deleted-feature registry, retired bigram spec restored

Documentation only. The Go library is byte-identical to v0.4.3 -- no .go file
changed.

New since v0.4.3:

• specs/deleted.md (07557d5): a registry of capabilities microfts2
  deliberately does not have. A feature whose requirements are ALL retired has
  no live spec to cite, and pointing it at the spec that replaced it misstates
  what that spec covers -- api.md no longer describes Options.MaxDBs,
  storage.md no longer describes big-endian C records. Four features now cite
  deleted.md instead: Index Computation (R36), MaxDBs Option (R101/T56),
  Encoding (R123), Bigram Index (R379-R416, T17-T54).

  Each entry records what the feature was, when and why it went, and what
  replaced it, so an approach that was tried and rejected cannot be quietly
  re-proposed. The bigram entry carries the measured cost that killed it --
  2.5s on 74K chunks, 1.7x index size.

• specs/migrations/complete/002-bigram-index-removed.md (9f2f6a9): the bigram
  spec restored verbatim from c8d1a19^. The requirements had cited BIGRAM.md,
  a path never tracked in this repo; the real spec was specs/bigram.md,
  deleted with the feature. Its number is recording order, not landing order
  -- the removal (2026-03-22) predates 001-lmdb-to-bbolt (2026-06-12).

• gitignore CLAUDE.local.md (7e2d3aa).

phase spec OK, query unindexed-specs empty, full suite green. phase
requirements fails on one approved item only: A2, the deliberately
non-sequential numbering that is never renumbered.

Daneel of Claude Opus 5

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

v0.4.0

Headline: LMDB → BBolt storage migration

The storage backend moved from LMDB (cgo) to BBolt (pure Go) — no more C dependency, simpler builds, fully portable.

⚠️ On-disk format break. Existing indexes are not readable by this version. Consumers must rebuild their index.

Chunker enhancements

• FileChunker interface for binary formats — reads the file directly with hash-based skip, distinct from the content-based Chunker.
• AppendAware chunkers with ErrAppendBoundary guard, enabling correct incremental >>-style appends.
• ChunkerMetadata optional interface for chunkers to surface metadata.
• Chunk locator and refcounted C-record fileids (a chunk's content can be shared across files via dedup).
• DB.SetFileLength.

API

• Exported ReadCRecord for power-user CRecord lookups.

Cleanup

• Retired stale bigram requirements (the bigram index was removed earlier; SearchFuzzy's trigram OR-union covers typo tolerance).
• A per-chunker content-transform hook was introduced and then fully retired within this cycle — the trigram index stays full-text (indexed verbatim; nothing stripped).

---

Full diff: v0.3.0...v0.4.0

v0.3.0

What's New

RandomAccessChunker interface

• New RandomAccessChunker interface for O(1) chunk retrieval by range label
• Built-in chunkers (LineChunk, MarkdownChunker, BracketChunker, IndentChunker) use lazy line-offset tables
• ChunkCache dispatches to the fast path when available, falls back to streaming
• Breaking: ChunkTexter interface removed — replaced by RandomAccessChunker

FileChunker interface split

• FileChunker is now a separate interface (FileChunks method) for binary formats that read files directly
• A single type can implement both Chunker and FileChunker
• Overlay (tmp://) requires Chunker; FileChunker-only chunkers error on tmp:// paths

Markdown headline merging

• Heading chunks now absorb following tag-only chunks (lines starting with @) and one content chunk
• Internal blank lines between merged sections are included in the chunk content
• Produces richer heading contexts for downstream consumers (e.g. ark search)

Other changes

• ContentLen field in C records — chunk byte length without re-reading files
• ChunkContentLens(fileid) method for per-file chunk size statistics
• TmpFileInfo for overlay document metadata
• Overlay chunk-level UTF-8 validation

v0.2.0

What's New

Search

• SearchFuzzy — typo-tolerant search via trigram OR-union tally, no bigram index needed
• SearchMulti — collect candidates once, score per strategy (map[string]ScoreFunc)
• WithLoose() — OR across terms (AND within each term)
• SearchRegex with trigram-accelerated boolean evaluation

tmp:// Overlay

• In-memory documents searchable alongside the LMDB index
• AddTmpFile, UpdateTmpFile, AppendTmpFile (shell >> semantics), RemoveTmpFile

ChunkCache

• NewChunkCache() — per-query cache for file content and chunked data
• Retrieve() for cached chunk content
• Copy() and InvalidateCaches() for read/write path separation

Chunkers

• BracketChunker — table-driven, line-oriented depth tracking
• IndentChunker — indentation-scoped, reuses BracketLang config
• MarkdownChunkFunc — honors fenced code blocks
• ChunkCallback API for streaming chunk processing
• CopyPairs exported; Attrs surfaced in ChunkResult

Performance

• NewSearchCache with pathToID reverse cache — >2x search speedup
• Coalesced T/W batch writes on add

Architecture

• Single LMDB subdatabase with prefix-distinguished records
• Chunk deduplication via SHA-256 content hash
• TxnHolder interface — records carry transaction context
• searchConfig receiver refactor for cleaner search internals
• RecordCounts, FileIDPaths, UnmarshalFHeader utilities

Removed

• Bigram index (too slow, too fat — SearchFuzzy handles the use case)

v0.1

Initial release of microfts2 — a dynamic LMDB trigram index in Go.

Features

• Trigram indexing with configurable character sets (6-bit encoding, 18-bit trigrams)
• Two-tree LMDB design — content DB is durable, index DB is rebuildable
• Pluggable chunking strategies — any command that outputs byte offsets
• Three built-in chunkers — line-based, overlapping lines, overlapping words
• Staleness detection — tracks file mod time and SHA-256 hash; -r flag auto-refreshes before any command
• Active trigram cutoff — configurable frequency percentile controls index selectivity
• Long filename support — key chaining for paths exceeding LMDB's 511-byte limit
• Library API — usable as a Go package, not just a CLI