Skip to content

v0.5.3

Latest

Choose a tag to compare

@avanrossum avanrossum released this 03 Apr 13:52
· 2 commits to main since this release

Minimum chunk size merging

Tiny header sections (e.g. single-word changelog entries, stub headings) were becoming standalone chunks with no semantic value, polluting search results.

Added

  • min_chunk_size config option (default: 50 words) — sections below this threshold merge with adjacent sections instead of becoming standalone chunks
  • _merge_small_sections() in indexer — merges forward, backward, and chains
  • 6 new tests (33 total)

Migration

Existing indexes should be rebuilt with pmem index --force to re-chunk with the new merge logic.