Find the chunk size your retrieval actually needs — measured on your own documents, in your browser.
Live: https://chunkforge.benrichardson.dev
Your RAG assistant cites the wrong page and you cannot tell whether that is the model, the prompt, or the fact that the answer got sliced in half at chunk 47. Every guide hands you a number — 512 with 50 overlap, 1000 with 200 — and none of them has seen your documents.
chunkforge lets you measure it. Drop your own PDFs or Markdown, type the queries that are actually
failing, click the passage each one should have returned, and drag the chunk-size slider. Every move
re-chunks, re-embeds and re-ranks the whole corpus in front of you. You leave with chunks.jsonl, a
splitter configuration with your numbers filled in, and a comparison of every setting it tried.
Everything happens in the tab. The documents are the kind you cannot upload — contracts, board packs, internal handbooks — and the queries are often more revealing than the documents, because they are what someone was actually trying to find out.
drop → extract → tokenize ONCE → [ split → embed → rank → score ] ← the slider re-runs this bracket
in ~220 ms for 3,000 chunks
Extraction. pdf.js's text layer, reassembled: lines grouped by baseline, running headers and page numbers removed, hyphens broken across lines rejoined, headings kept as structure. Lines stay in the producer's content-stream order rather than being sorted by y — see the note on two columns below.
Tokenization. Once per document, in the tokenizer of the embedding model you are chunking for, cached as a token→character offset array. Every subsequent slider move is array slicing and a binary search, never re-tokenization.
Scoring. sentence-transformers/static-retrieval-mrl-en-v1, whose entire ONNX graph is
Gather → DequantizeLinear → ReduceMean. It is a lookup table, not a network: an embedding is the
mean of some rows. chunkforge reads the quantised table straight out of the ONNX file with a ~200-line
protobuf reader and ships no inference runtime at all.
This is the distinction the whole tool rests on.
- The boundary tokenizer decides what "512 tokens" means. It is your embedding model's own
tokenizer, fetched as
tokenizer.jsonfrom that model's Hugging Face repository. WordPiece and byte-level BPE are both implemented exactly; the BPE path was checked token for token againstgpt-tokenizerfor cl100k_base and o200k_base, across ten edge cases and an 11,003-token document, first token to last. A tokenizer that cannot be reproduced exactly — SentencePiece/Unigram models, gated repositories — is refused by name rather than approximated, because the artefact is a number you paste into your pipeline. - The scoring tokenizer is the static model's own WordPiece vocabulary. It exists only to turn text into rows of a lookup table. Its token counts are never shown.
A hit rate is a score against ground truth. With no labels there is no ground truth, so until you mark a passage chunkforge shows the ranking moving and claims nothing — no margin, no self-consistency, no cross-retriever agreement. Those proxies measure a retriever's confidence, which is highest exactly when it is confidently wrong.
Once you mark a passage:
- A hit is binary character overlap between a retrieved chunk and a marked span. Not "what fraction of the span came back": people mark a paragraph when they meant a sentence, and measured against that sloppiness a fractional score moves by −0.26 while binary overlap moves by +0.02.
- k is derived from a context budget, not fixed. You set the tokens you are willing to paste into
your model;
k = floor(budget ÷ chunk size). At a fixed top-10, 128-token chunks return ~1,300 tokens and 8,192-token chunks return ~54,000, so "recall rises with chunk size" would be arithmetic rather than a finding. - A chance level is drawn beside every score — exactly what random selection would score, in
closed form:
1 − C(n−h, k)/C(n, k). On the shipped sample, 96-token chunks with a 4,096-token budget score a perfect 3 of 3 at a 52% chance level. Without that column that row reads exactly like the genuinely good one. - Rank is never summed into a headline, because it is not comparable across chunkings. It is shown per query, beside the number of chunks it was drawn from.
- No winner is declared. Separating two close settings takes around six queries that disagree one way about them; with the two to five queries anyone types, most pairs tie. chunkforge reports the cheapest setting that returned everything you marked, as a description of what happened.
Prior art for span-level ground truth is Smith and Troynikov, Evaluating Chunking Strategies for Retrieval (Chroma, July 2024). Their metrics need pre-annotated relevant excerpts, which chunkforge deliberately does not collect, so this is a simpler measurement and does not borrow their names.
| strategy | what it does | fidelity |
|---|---|---|
| Fixed-size | token windows with overlap, ignoring all structure | exact by construction |
| Recursive | LangChain's RecursiveCharacterTextSplitter |
verified chunk-for-chunk against langchain-text-splitters 0.3.11 across seven size and overlap settings, including the degenerate ones where it emits chunks larger than the limit |
| Heading-aware | cuts at headings, prepends the heading trail before embedding | no stock splitter reproduces it, and the exported config says so |
The recursive splitter's separators are ["\n\n", "\n", " ", ""] — read out of the library, not
remembered. Note what is not in that list: there is no sentence separator, so the default splitter
will cut a sentence in half once a paragraph does not fit. Adding a sentence rung here would make
chunkforge's chunks nicer than the config it hands you, which is the one thing this code cannot do.
- The modern pdf.js build is a white page on iOS below 18.4.
build/pdf.mjscontainsif (typeof Iterator.prototype.join !== "function")unguarded at module top level;typeofprotects a bare identifier, not a property access on one. chunkforge ships the legacy build. The cost is about 16 kB gzipped. - Without
cMapUrl, a CJK PDF returns zero text items and no error. The predefined CMaps are served from this origin and fetched only by documents that need one. - IDF-weighted pooling is a regression. It was in an earlier design on the strength of one anecdote. On BEIR SciFact — 300 queries, harness first validated by reproducing this model's own published MTEB figures — plain mean scores nDCG@10 0.5938 against 0.5648 weighted, paired bootstrap P(weighted better) = 0.007. Worse, weights drawn from the chunking move with the slider: eight points of instrument drift, the same size as the effect being measured.
- Sorting lines by y breaks two-column PDFs. Real two-column producers emit the left column contiguously and then the right, so document order is already reading order. Column detectors built to repair the damage were measured finding zero columns on a genuine two-column paper and up to sixteen spurious ones on pages containing tables. No column detector ships.
- A paragraph threshold of 1.55× the line pitch merged four paragraphs into one 1,537-character block on the shipped sample. It is 1.3×.
- The sample's own demo plan was wrong. It was written so one query would favour small chunks; measured, it favoured large ones, because the 900 tokens surrounding that answer are all on the query's topic. The sample now demonstrates what is actually true of it — see below.
public/samples/handbook.pdf is a 19-page fictional employee handbook written for this tool, with
three pre-filled queries and their answers pre-marked. Measured on it:
| setting | answers | retrieved tokens/query | chance level |
|---|---|---|---|
| recursive 1024/200 — the most-copied config anywhere | 3 of 3 | 3,724 | 29% |
| heading-aware 1024 | 3 of 3 | 1,146 | 7% |
| fixed 96 | 3 of 3 | 4,015 | 52% |
Same three answers, a third of the prompt. That comparison is only possible because recall is reported with its price rather than on its own.
- Web Workers — the embedding table, the token indices and every re-score live off the main thread
- Cache API — the model and tokenizer files, so every run after the first is fully offline
- File API + drag-and-drop +
<input type=file>— iOS Safari cannot drag and drop at all - Clipboard, Web Share, anchor download — one zip from one click, because Safari suppresses every programmatic download after the first in a single gesture
- Service Worker (vite-plugin-pwa) — installable, offline app shell with the sample precached
Protected
- Documents and queries never leave the device. There is no upload endpoint in the code.
- Nothing is written to disk: no IndexedDB, no Origin Private File System, no cookies. Two settings go
in
localStorage; document text never does. A build gate overdist/**/*.jsasserts the storage APIs stayed out. - Every export is assembled in the tab.
Not protected
- Which model you pick is visible to Hugging Face. The scoring model and your chosen tokenizer are anonymous GETs of constant URLs with credentials omitted. That reveals your IP and which file you asked for, and nothing about your documents. After the first run they come from cache and the tool works with the network off.
- Anything you type into the feedback form, and only when you press Send.
- Your own exports.
chunks.jsonlcontains your document text by design.
Trust model
- The static bundle served by GitHub Pages, and the TLS chain to it.
huggingface.cofor two public model files.- A Cloudflare Web Analytics beacon records anonymous page views — no cookies, no fingerprinting, no cross-site tracking; your files and data are never sent to it.
The claim is "your documents never leave the tab", not "no network". Two model files are fetched.
- Vite 7 + vanilla TypeScript, no framework
pdfjs-dist(legacy build) for PDF text,fflatefor the export zip- First-party: the ONNX table reader, the WordPiece and byte-level BPE tokenizers, the three splitters, the metric, the CSV writer
- Vitest; GitHub Pages via GitHub Actions
No inference runtime, no WASM of our own, no SharedArrayBuffer, and therefore no COOP/COEP — which
is why it runs on GitHub Pages unmodified.
npm install
npm run dev # vite dev server on :5173
npm test # vitest
npm run build # produce dist/
npm run preview # serve dist/ locally
npm run sample # regenerate public/samples/handbook.pdf
npm run notices # regenerate THIRD-PARTY-NOTICES.md from dist/ source mapsSome tests need the 31 MB scoring model and are skipped without it:
CHUNKFORGE_MODEL_DIR=/path/with/model_int8.onnx+tokenizer.json npm testA push to main triggers .github/workflows/deploy.yml, which builds, runs the tests (the build runs
first, because several tests are gates over dist/ and would otherwise skip silently and report
green), and deploys dist/ to GitHub Pages. The custom domain is pinned by public/CNAME.
GNU Affero General Public License v3.0 or later, with an attribution requirement added under section 7(b) — see ADDITIONAL-TERMS.md.
In short: you may run, modify, redistribute and even sell this, but if you distribute it — or run a modified version where other people can reach it — you have to publish your source under the same licence and keep the attribution. A separate commercial licence without those obligations is available on request: hi@ben.gy.
Third-party components keep their own licences — see THIRD-PARTY-NOTICES.md.