Releases
v0.3.0
Compare
Sorry, something went wrong.
No results found
zot
released this
24 Apr 14:15
What's New
RandomAccessChunker interface
New RandomAccessChunker interface for O(1) chunk retrieval by range label
Built-in chunkers (LineChunk, MarkdownChunker, BracketChunker, IndentChunker) use lazy line-offset tables
ChunkCache dispatches to the fast path when available, falls back to streaming
Breaking: ChunkTexter interface removed — replaced by RandomAccessChunker
FileChunker interface split
FileChunker is now a separate interface (FileChunks method) for binary formats that read files directly
A single type can implement both Chunker and FileChunker
Overlay (tmp://) requires Chunker; FileChunker-only chunkers error on tmp:// paths
Markdown headline merging
Heading chunks now absorb following tag-only chunks (lines starting with @) and one content chunk
Internal blank lines between merged sections are included in the chunk content
Produces richer heading contexts for downstream consumers (e.g. ark search)
Other changes
ContentLen field in C records — chunk byte length without re-reading files
ChunkContentLens(fileid) method for per-file chunk size statistics
TmpFileInfo for overlay document metadata
Overlay chunk-level UTF-8 validation
You can’t perform that action at this time.