Skip to content

v0.3.0

Choose a tag to compare

@bluesky585 bluesky585 released this 15 Sep 01:02
· 42 commits to main since this release
07889fe

A real token budget, and the fuzzing that proves the split invariants.

-tokenizer tiktoken counts with a real BPE vocabulary, so -size is a model's token budget rather than a rune count. The vocabulary is not embedded: the first use downloads and caches it, and a run that never selects this tokenizer never touches the network. It is the one third-party dependency, quarantined in pkg/tokenizer/tiktoken so that every chunker, and pkg/tokenizer itself, still build with none.

Pieces are cut on character boundaries rather than raw token boundaries, because a BPE token can end inside a character and offsets are rune ranges.

Fixed: when one character is wider than the whole budget, a window of nothing but those boundary pieces spans no runes, and was emitted as an empty chunk carrying a non-empty token count. A bare four-byte han at -size 2, or any multi-token character at -size 1, hit it. Such a window is now skipped.

internal/chunkerfuzz adds five fuzz targets over every chunker. The BPE target found that empty chunk on its first seed; the other four cover a rune ruler, a word ruler, the byte-budget chunker, token overlap, and neighbor-token context, and need no vocabulary.

-dir with -embed now embeds every input in one pass, in bounded batches, instead of one request per file.

go get github.com/bluesky585/nibble@v0.3.0
go install github.com/bluesky585/nibble/cmd/nibble@v0.3.0
go install github.com/bluesky585/nibble/cmd/nibble-api@v0.3.0

No API removal: this is additive over v0.2.0. Still not a production RAG platform. Hashing embeddings are bag-of-words, the JSONL store is scanned linearly, the HTTP index is in-process only, and the default tokenizer still counts runes.