Skip to content

v0.3.0

Choose a tag to compare

@zot zot released this 24 Apr 14:15
· 36 commits to main since this release

What's New

RandomAccessChunker interface

  • New RandomAccessChunker interface for O(1) chunk retrieval by range label
  • Built-in chunkers (LineChunk, MarkdownChunker, BracketChunker, IndentChunker) use lazy line-offset tables
  • ChunkCache dispatches to the fast path when available, falls back to streaming
  • Breaking: ChunkTexter interface removed — replaced by RandomAccessChunker

FileChunker interface split

  • FileChunker is now a separate interface (FileChunks method) for binary formats that read files directly
  • A single type can implement both Chunker and FileChunker
  • Overlay (tmp://) requires Chunker; FileChunker-only chunkers error on tmp:// paths

Markdown headline merging

  • Heading chunks now absorb following tag-only chunks (lines starting with @) and one content chunk
  • Internal blank lines between merged sections are included in the chunk content
  • Produces richer heading contexts for downstream consumers (e.g. ark search)

Other changes

  • ContentLen field in C records — chunk byte length without re-reading files
  • ChunkContentLens(fileid) method for per-file chunk size statistics
  • TmpFileInfo for overlay document metadata
  • Overlay chunk-level UTF-8 validation