v0.10.0: CJK retrieval, one-step re-index, rune fast path

This release makes CJK text a first-class retrieval citizen, collapses re-indexing to one step, and cuts the token chunker's allocations by two orders of magnitude.

Shared CJK bigram segmentation (#76) — breaking change

Han, kana, and hangul are written without word boundaries, and until now only the BM25 term splitter handled that. Everything else saw a CJK sentence as an undivided lump.

• New tokenizer.Bigram: tiles CJK runs into non-overlapping pairs so a size budget measured in it bounds retrieval terms, not sentences. Latin and digit runs stay whole.
• tokenizer.Terms is now the one definition of a term: scoring.WordTerms delegates to it, and embed.Hashing hashes it.
• Breaking: Hashing embeds terms instead of whitespace fields. CJK vectors change fundamentally — from one sentence-slot to sliding bigrams, giving dense/hybrid a real signal (a phrase query scores 0.46 vs 0.26 where it previously carried none). Latin vectors change rarely in ranking but not bit-for-bit. Indexes built before this release must be rebuilt — the one-step re-index below makes that one command per source.
• CLI flag help, the API's tokenizer field, and the TypeScript Tokenizer union all accept "bigram".

One-step re-index (#74)

• CLI: -source a.md with -index now replaces that source's contents (delete, then index). -append opts back into adding.
• API: "replace": true on the index body, opt-in for the same reason.
• Library: store.ReplaceSourceLabeled / store.ReplaceSourceEmbeddedLabeled.

Rune tokenizer substring fast path (#75)

A tokenizer whose tokens are exactly runes no longer pays for per-rune strings and window joins: the token chunker slices windows out of the original text on a new tokenizer.Runes capability check. The table chunker's benchmark went 3785 → 20 allocs/op and 210µs → 50µs; the token chunker's own benchmark holds at 2 allocs/op. word and tiktoken are unaffected.

Bug fix (#73)

store.Records returned nothing for the in-memory store through the Store interface — a missing case in its type switch made bm25 and hybrid silently empty on memory-backed indexes. Found while testing #74; reported as #72.

Full changes: v0.9.0...v0.10.0

v0.9.0

This release completes the retrieval story around the chunker: index management, filtered search, and a faster, half-sized vector pipeline.

Source management (#67)

Indexed chunks can now carry a source — a file path, a URL, a caller-chosen key.

• CLI: -source with -index, -list-sources, -delete-source
• API: source on the index body, GET /v1/sources, DELETE /v1/sources/{name}
• Chunks are keyed by text and offsets, so re-indexing a document under a new source moves it there — replace without rebuilding the index. Deleting to zero is reported as 0, not an error.

Source-filtered search (#68)

• CLI: -source on -query; API: optional source on the search body.
• The filter lands before scoring, so k counts filtered hits. Absent searches everything; an empty string matches the records indexed without a source.

float32 vectors end to end (#69) — breaking change for Go consumers

embed.Embedder.Embed now returns [][]float32, and Chunk.Embedding / Record.Vector are []float32. Scoring runs over the stored bytes directly: memory halves and the per-query float64 conversion is gone.

• The SQLite file format is unchanged — existing databases open as they are. The JSONL and wire formats are unchanged; the TypeScript client needs no update.
• The cosine scan gained 4-lane accumulators (the profiler showed accumulator serialization, not memory bandwidth, was the bound): 200k chunks at 1536 dimensions went 591ms → 297ms; 50k went 143ms → 81ms.
• Ranking agreement with the previous float64 scoring was checked over 500 randomized trials — no top-5 disagreement.

rag-search example (#70)

go run ./examples/rag-search walks the retrieval lifecycle in one program: chunk with a real chunker, index into SQLite under per-document sources, hybrid search, source-filtered search, delete a source, and search again.

Full changes: v0.8.0...v0.9.0

v0.8.0

Two retrieval-quality fixes since v0.7.0. No API breaks; drop-in upgrade.

BM25 now works for Chinese, Japanese, and Korean (#63)

Sparse search previously split a run of Han/kana/hangul whole, so a
paraphrased CJK query scored zero on every document. CJK text is now
segmented into sliding bigrams — the unit Lucene's CJK analyzers use,
no dictionary needed — while Latin words pass through unchanged.
Bigrams never cross a script boundary.

查询「如何提高检索的效果」此前在含「提高检索质量」的文档上得 0 分，现在正常命中。

Over-budget sentences are re-cut at their clauses (#65)

sentence and semantic sent a sentence that exceeded the size budget
straight to token windows, cutting words wherever the window boundary
landed. The sentence now gets a second, finer split first — commas,
then whitespace, then tokens — via the new shared recursive.FallbackRules().
Text with no finer delimiter behaves exactly as before.

Also

• examples/ — six runnable programs, one per idea (#64)
• Fuzz targets for the new paths: 90s BPE + 60s standard, no findings
• Verified against three real scraped corpora (English docs, Chinese article, Markdown syntax)

Full changelog: v0.7.0...v0.8.0

v0.7.0

Retrieval becomes a complete story: chunk, embed, persist on either front end, rank three ways.

Store

• SQLite index store (#58): nibble -index docs.db writes chunks plus vectors to a SQLite database — one row per chunk, keyed by text and offsets so a re-upserted chunk updates in place. Vectors are float32, halving the file with the ranking preserved. Incremental: an upsert writes the rows it touched, not the whole file, and the file reopens across processes. Same exact brute-force cosine ranking as the JSONL store; the driver is pure Go, no CGO.

Chunking

• Recursive overlap (#59): recursive accepts -overlap — the next chunk repeats the previous chunk's tail in its own text, the way token windows widen, so a retrieval hit on either side of a cut can see across it. Chunks stay exact slices of the source; the last still ends where the input does.

HTTP API

• Scoring modes (#60): POST /v1/search takes scoring (dense default, bm25, hybrid) and hybrid_weight, matching the CLI. Only dense and hybrid embed the query; bm25 needs no embedder at all. The ranking lives in pkg/store, shared by both front ends, so they cannot drift. The TypeScript client carries the same options.
• Persistent index (#61): nibble-api -index docs.db backs the API's index with the same SQLite file shape the CLI writes — created when missing, resumed by the next server on the same path. Without the flag, the memory default is unchanged.

Full change list

• #58 store: SQLite index store with incremental upsert
• #59 recursive: overlap repeats the previous chunk's tail
• #60 httpapi: scoring modes for /v1/search, shared with the CLI
• #61 httpapi: -index backs the API's index with a SQLite file
• #62 Pin the install commands to v0.7.0

v0.6.0

New features

• CSV and TSV input. Tabular files are read wherever a file or -dir entry is accepted: one chunk per data row, column names carried in context the way the table chunker carries a GFM header. The row is never split, and the header row is metadata, not a chunk. (#54)
• BM25 and hybrid retrieval. -scoring dense|bm25|hybrid picks how -query ranks hits. BM25 needs no embedder at all — it works on an index built without vectors — and hybrid blends both paths, with the sparse side normalized first since BM25 scores are unbounded. (#55)
• HTML and EPUB extraction. Container formats are pulled to plain text before chunking: block tags become the paragraph breaks the recursive rules split on, script and style are dropped, and an EPUB's spine orders its documents. Uses the official golang.org/x/net parser, quarantined in pkg/extract so every chunker still builds with no dependency. (#56)

Notes

All three were found by studying how txtai, a full-stack RAG framework in Python, structures the same problems; the cuts that made sense for a small core were kept, the rest (agents, graph search, thirteen ANN backends) were deliberately not taken.

Full changelog: v0.5.1...v0.6.0

v0.5.1

Bug fix

• recursive: no more whitespace-only chunks. When a paragraph exactly fills the budget, its trailing blank-line separator used to hard-split into a chunk holding nothing but whitespace — pure retrieval noise for RAG. Found by validating against a real 142KB corpus, where a 67-document sweep produced 28 of them. The separator is now merged into the neighboring chunk: reconstruct still holds, and the absorbing neighbor may exceed the size by the whitespace it absorbed. (#52)

Full changelog: v0.5.0...v0.5.1

v0.5.0

New opt-in APIs, all additive — nothing existing changes behavior.

Chunker options

• sentencechunker.MinRunes(n) / semantic.MinRunes(n): merge sentence pieces shorter than n runes, absorbing abbreviation fragments such as the e. that e.g. this yields. Opt-in and off by default: in CJK a 2-rune sentence is complete, and merging it with its neighbor destroys a real boundary.
• semantic.SimilarityWindow(n): widen the cut test from adjacent sentence pairs to the mean vectors of n sentences on each side. Smooths single-sentence wording jitter that would otherwise split an unchanged topic. Consecutive below-threshold points are one boundary, cut at the deepest.
• recursive.RulesFromJSON: the rule hierarchy from JSON (-rules file on the CLI, a rules field on the HTTP API). attach accepts only prev, because that is how nibble's split always attaches a delimiter.

Text handling

• overlap.Merge: fold neighbor context into chunk text itself, in prefix, suffix, and justified modes. Breaks the reconstruct guarantee by design; Prefix/Suffix remain for reconstruct-safe context.
• tokenizer.CountBatch: measure many pieces in one call.

Throughput

• batch.Chunk: chunk many texts through a worker pool (min(8, NumCPU*3/4)). Results line up with inputs by index; errors report the lowest failing index. The CLI chunks -dir inputs through it.

Tooling

• Benchmark baselines for the core packages against a fixed corpus (go test -run xxx -bench=. -benchmem ./pkg/...), documented as a manual before/after comparison rather than a CI gate.

v0.4.1

Documentation only. No code changed, so nothing to upgrade for, but the README is the first thing anyone reads and it was getting in the way.

The README now opens with a quick start instead of an argument. It was six design principles and a screen of limitations before the first runnable command, and the first sentence calls this a Go library while every code block in the file was bash, json, or typescript — there was no example of using it as a library at all. The new section has a complete main.go with its real output, so the shape of a Chunk is visible before any JSON is explained; a table of all eight chunkers with their constructors and what each one cuts on, which is what you need to pick one, since buildchunk maps CLI flags to chunkers and is internal; and a one-line command-line example that pipes text in, so it runs with no input file to create first. Principles and Status are unchanged and now come after the working example.

pkg/overlap's package doc said it "copies neighboring tokens into Chunk.Context", which is accurate and stopped there. The package is named overlap and there is a -overlap flag, so the natural reading was that this is where -overlap lives. It is not: -overlap widens each token window and is implemented in pkg/tokenchunker, while this package implements -context, where nothing repeats and Chunk.Text stays a slice of the source. The doc now names both flags. No behavior changes.

go get github.com/bluesky585/nibble@v0.4.1
go install github.com/bluesky585/nibble/cmd/nibble@v0.4.1
go install github.com/bluesky585/nibble/cmd/nibble-api@v0.4.1

Still not a production RAG platform. Hashing embeddings are bag-of-words, the JSONL store is scanned linearly, the HTTP index is in-process only, and the default tokenizer still counts runes.

v0.4.0

Python source, and a TypeScript client for the HTTP API.

-chunker code no longer parses Go only. Python is cut on its top-level declarations by a line scanner that tracks bracket depth, comments, and string literals; a doc comment or decorator stays with the declaration it belongs to, and anything the scanner is unsure about stays in the piece it is in rather than risking a cut through the middle of a statement. Go keeps its standard library parser, so a declaration's span is still exact. -lang go|python names the language when the caller knows it and empty detects it; a name nibble cannot cut is an error rather than a silent fallback to token windows. markdown now reads each fence's info string, so a ```python block is cut by Python rules instead of being parsed as Go — that string was already parsed and then thrown away.

ts/ adds a thin TypeScript client for the HTTP API. It is thin on purpose: the chunkers stay Go, so nothing there reimplements splitting, and adding a chunker to the library gives the client that chunker for free. Node 23.6.0+ runs the .ts files directly by stripping types, so there is no build step and no npm package. One thing worth knowing first: a chunk's start and end are rune offsets while JavaScript indexes UTF-16 code units, so text.slice(start, end) is wrong for text holding an astral character.

No API removal: codechunker.New takes its options variadically, so New(tok, size) still compiles.

go get github.com/bluesky585/nibble@v0.4.0
go install github.com/bluesky585/nibble/cmd/nibble@v0.4.0
go install github.com/bluesky585/nibble/cmd/nibble-api@v0.4.0

Still not a production RAG platform. Hashing embeddings are bag-of-words, the JSONL store is scanned linearly, the HTTP index is in-process only, and the default tokenizer still counts runes.

v0.3.0

A real token budget, and the fuzzing that proves the split invariants.

-tokenizer tiktoken counts with a real BPE vocabulary, so -size is a model's token budget rather than a rune count. The vocabulary is not embedded: the first use downloads and caches it, and a run that never selects this tokenizer never touches the network. It is the one third-party dependency, quarantined in pkg/tokenizer/tiktoken so that every chunker, and pkg/tokenizer itself, still build with none.

Pieces are cut on character boundaries rather than raw token boundaries, because a BPE token can end inside a character and offsets are rune ranges.

Fixed: when one character is wider than the whole budget, a window of nothing but those boundary pieces spans no runes, and was emitted as an empty chunk carrying a non-empty token count. A bare four-byte han at -size 2, or any multi-token character at -size 1, hit it. Such a window is now skipped.

internal/chunkerfuzz adds five fuzz targets over every chunker. The BPE target found that empty chunk on its first seed; the other four cover a rune ruler, a word ruler, the byte-budget chunker, token overlap, and neighbor-token context, and need no vocabulary.

-dir with -embed now embeds every input in one pass, in bounded batches, instead of one request per file.

go get github.com/bluesky585/nibble@v0.3.0
go install github.com/bluesky585/nibble/cmd/nibble@v0.3.0
go install github.com/bluesky585/nibble/cmd/nibble-api@v0.3.0

No API removal: this is additive over v0.2.0. Still not a production RAG platform. Hashing embeddings are bag-of-words, the JSONL store is scanned linearly, the HTTP index is in-process only, and the default tokenizer still counts runes.