Repository navigation
v0.3.0
A real token budget, and the fuzzing that proves the split invariants.
-tokenizer tiktoken counts with a real BPE vocabulary, so -size is a model's token budget rather than a rune count. The vocabulary is not embedded: the first use downloads and caches it, and a run that never selects this tokenizer never touches the network. It is the one third-party dependency, quarantined in pkg/tokenizer/tiktoken so that every chunker, and pkg/tokenizer itself, still build with none.
Pieces are cut on character boundaries rather than raw token boundaries, because a BPE token can end inside a character and offsets are rune ranges.
Fixed: when one character is wider than the whole budget, a window of nothing but those boundary pieces spans no runes, and was emitted as an empty chunk carrying a non-empty token count. A bare four-byte han at -size 2, or any multi-token character at -size 1, hit it. Such a window is now skipped.
internal/chunkerfuzz adds five fuzz targets over every chunker. The BPE target found that empty chunk on its first seed; the other four cover a rune ruler, a word ruler, the byte-budget chunker, token overlap, and neighbor-token context, and need no vocabulary.
-dir with -embed now embeds every input in one pass, in bounded batches, instead of one request per file.
go get github.com/bluesky585/nibble@v0.3.0
go install github.com/bluesky585/nibble/cmd/nibble@v0.3.0
go install github.com/bluesky585/nibble/cmd/nibble-api@v0.3.0No API removal: this is additive over v0.2.0. Still not a production RAG platform. Hashing embeddings are bag-of-words, the JSONL store is scanned linearly, the HTTP index is in-process only, and the default tokenizer still counts runes.