Better Vector Search for Long Documents: Chunking Inside Manticore Search

[githubmanticore]

Originally posted by Dmitrii Kuzmenkov on https://mnt.cr/go/Ku6gwD on September 15, 2026

Better Vector Search for Long Documents: Chunking Inside Manticore Search

An embedding model reads only the first few hundred tokens of a document and silently drops the rest. Manticore Search now splits long documents for you at INSERT time: add chunk_strategy to the vector column and pick one of five strategies. No ingest pipeline, no splitter library. On our own manual, recall@5 for deep content went from 55% to 83%.

Say you are building search over your team's internal documentation — guides, runbooks, postmortems. You have a table with auto embeddings: you insert text, Manticore runs the model and fills the vector column for you. (If that is new to you, start with vector search in Manticore.) You load a 4,000-word document. The insert succeeds. The search works. Everything looks fine.

Except the model you picked has a 512-token input window, and that document is about 5,000 tokens long. The model read the first 380 words and threw away the other 3,600. Nothing in the document past that point can ever be retrieved, and nothing anywhere told you. The embedding may not represent the document as a whole either.

Until now, you would usually split the document into several pieces yourself, create embeddings for each one, and then work out how to combine the results if you wanted document search rather than chunk search. Manticore now handles this in the table definition: add chunk_strategy to the vector column in CREATE TABLE, and Manticore splits each document into chunks, embeds every chunk, and searches all of them:

DROP TABLE IF EXISTS docs;

CREATE TABLE docs (
  title text,
  content text,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,content'
    chunk_strategy='sentence' max_tokens='256' overlap_tokens='32'
);

That is the whole feature. No ingest pipeline, no splitter library, no second table for chunks, no GROUP BY to fold chunk hits back into documents.

TL;DR

• Five strategies: truncate (the old default), mean, fixed, recursive, sentence. Set with chunk_strategy on a model-backed vector column.
• truncate and mean produce one vector per document and work on a float_vector column. fixed, recursive and sentence produce many, so they need a float_vector_array column.
• A document is still one search result. Chunks compete individually, and Manticore returns the document once, with knn_dist() reporting the distance to its closest chunk. k counts documents, not chunks.
• Tuning knobs: max_tokens (chunk size), overlap_tokens (shared tokens between neighbors), max_chunks (ceiling per document).
• Measured on the Manticore manual (189 pages, ~298k words): for content buried past the model's window, recall@5 went from 55.1% → 83.3% and MRR from 0.44 → 0.70, at ~2.5× the RAM and ~4× the ingest time.
• Queries are never chunked. A query is short enough to embed as a whole; only stored documents are split.

The problem, shown with a small example

Suppose you have four documents:

1. Backup and restore runbook — about 700 words, roughly 900 tokens. Backup schedules, retention, restore drills, credentials, capacity planning. The last section explains how to rotate the TLS certificate used by the replication port.
2. Monitoring and alerting guide — unrelated.
3. Getting started with the CLI — unrelated.
4. TLS and certificates for the HTTP API — a short page that is entirely about certificates, and never mentions rotation or replication.

You can create the table and add the documents using the commands below.

So, what we have is: one table, three vector columns with the same source text — one column per strategy. A single INSERT fills all three, so the comparison conditions are identical:

DROP TABLE IF EXISTS docs;
CREATE TABLE docs (
  title text,
  body text,
  v_truncate float_vector       knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,body',
  v_mean     float_vector       knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,body' chunk_strategy='mean',
  v_sentence float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,body'
    chunk_strategy='sentence' max_tokens='128' overlap_tokens='32'
);

Insert the four documents

INSERT INTO docs (id, title, body) VALUES
  (1, 'Backup and restore runbook',
   'Nightly backups run at 02:00 UTC from the standby node. The job snapshots every table directory, writes a manifest, and uploads the result to object storage. Retention is thirty daily copies, twelve monthly copies, and one yearly copy. A restore drill runs on the first Monday of each month against a scratch cluster. The drill counts as passed only when a full-text search over the restored data returns the same document count as production. Anything less is treated as a failed drill and investigated the same week. Before a restore, freeze the target cluster so that no writes land while files are being replaced. Copy the manifest first and verify its checksum. If the checksum does not match, stop: a partial restore is worse than no restore, because the cluster will start and silently serve half the corpus. After the files are in place, unfreeze and let replication catch up. Watch the queue depth. If it does not drain within ten minutes, the node is probably still reading from cold storage and needs a warm-up pass before it can serve traffic. Backup failures page the on-call engineer. The three most common causes are an expired object storage credential, a disk that filled up while the snapshot was being written, and a table left frozen by a previous failed run. All three are recoverable without data loss. Check the job log first, then the disk, then the freeze state of every table. Capacity planning for backups is boring but it matters. A daily copy of the search cluster is roughly the size of the data directory plus fifteen percent for the manifest and metadata. Multiply by the retention count, add the transfer cost, and you have the monthly bill. Most teams discover too late that the yearly copies dominate the storage line. Object storage lifecycle rules do most of the retention work. Daily copies move to infrequent access after seven days and expire after thirty. Monthly copies move to archive after sixty days. Yearly copies never expire automatically; deleting one is a manual action that requires a second approver. Credentials for the backup job live in the secret manager and are issued to a role, not to a person. The role can write new objects and list the bucket. It cannot delete, and it cannot read objects older than the current day. That last restriction is the cheapest defence against a compromised backup runner turning into a data exfiltration path. Documentation for each table lives next to its schema: what the table is for, who owns it, how large it is expected to get, and whether it can be rebuilt from an upstream source. A table that can be rebuilt does not need thirty daily copies. Roughly half of most clusters turns out to be derived data that nobody had marked as derived. Verification is not the same as the job exiting zero. The job can succeed while producing an unusable copy: an empty table, a truncated upload, a manifest that references a file that was never written. The verification step reads the manifest back, checks every referenced object exists and matches its recorded size, and compares row counts on three sampled tables against production. Rotating the replication TLS certificate is a separate procedure and the step people most often get wrong. The certificate that secures the replication port is not the same as the one the HTTP API uses, and replacing one does not replace the other. Generate the new key and signing request on the node that will be rotated first, sign them with the cluster certificate authority, and place the files next to the existing ones rather than on top of them. Then update the node configuration to point at the new paths and reload. Do one node at a time and confirm that the cluster reports every peer as synced before moving on. A half-rotated cluster where two nodes trust different authorities will keep accepting writes on both sides and diverge quietly. When every node has been rotated, remove the old key material and revoke the retired certificate at the authority.'),
  (2, 'Monitoring and alerting guide',
   'Every node exports metrics over an HTTP endpoint that a scraper collects once per fifteen seconds. The dashboards are grouped into four rows: traffic, latency, saturation, and errors. Traffic is queries per second broken down by table. Latency is the ninety-fifth and ninety-ninth percentile of query time, measured server side. Alerting is deliberately thin. Paging alerts fire on sustained error rate above one percent for five minutes, on ninety-ninth percentile latency above two seconds for ten minutes, and on a node dropping out of the cluster. Everything else is a ticket, not a page. Teams that page on every anomaly stop reading pages within a month. Log retention is fourteen days hot and ninety days cold. The query log records the query text, the table, the match count, and the elapsed time. Turning it on costs a few percent of throughput and is almost always worth it, because most performance investigations start with a slow query nobody knew was being issued.'),
  (3, 'Getting started with the CLI',
   'The command line client connects over the MySQL wire protocol, so any MySQL client works and you do not need to install anything special. Point it at port 9306 and you get an interactive shell. The shell understands the usual conveniences: history, tab completion of table names, and vertical output when a row is too wide for the terminal. Start by listing tables, then look at one with SHOW CREATE TABLE. The output is the exact statement that would recreate the table, including every option that was applied implicitly, which makes it the fastest way to find out what a table actually does rather than what someone documented two years ago. Bulk loading from the shell is possible but rarely what you want. For anything above a few thousand rows, use the HTTP bulk endpoint or one of the log shipper integrations, both of which batch and retry for you.'),
  (4, 'TLS and certificates for the HTTP API',
   'The HTTP API can be served over TLS. You supply a certificate, a private key, and optionally a chain file, and the listener starts speaking HTTPS instead of HTTP. Clients that present a certificate of their own can be authenticated by it, which is the usual way to lock an internal API down without putting a password in every config file. Certificates for the HTTP API come from wherever your organisation gets certificates: a public authority, an internal authority, or an automated issuer. The file format is PEM. Both the certificate and the key must be readable by the user the server runs as, and the key must not be world readable or the listener refuses to start. Debugging TLS problems is mostly about reading the handshake. A client that reports an unknown authority is missing the chain. A client that reports a hostname mismatch is connecting by an address that is not in the certificate. A client that hangs is usually talking TLS to a plaintext port.');

Now ask a question whose answer lives in the runbook's last section, once per strategy:

SELECT title, knn_dist() FROM docs
WHERE knn(v_truncate, 4, 'how do I rotate the TLS certificate used for replication');

SELECT title, knn_dist() FROM docs
WHERE knn(v_mean, 4, 'how do I rotate the TLS certificate used for replication');

SELECT title, knn_dist() FROM docs
WHERE knn(v_sentence, 4, 'how do I rotate the TLS certificate used for replication');

Strategy	1st result	2nd result
truncate (default)	TLS and certificates for the HTTP API — 0.762	Backup and restore runbook — 0.936
mean	Backup and restore runbook — 0.656	TLS and certificates for the HTTP API — 0.762
sentence, 128 tokens, 32 overlap	Backup and restore runbook — 0.254	TLS and certificates for the HTTP API — 0.700

With truncate, the document that actually answers the question loses to a decoy that merely looks like it is about certificates. The runbook's single vector was built from its opening pages on backup schedules and restore drills, because that is all the model was allowed to read.

With sentence chunking, the runbook is stored as nine vectors instead of one:

SELECT id, title, LENGTH(v_sentence) AS chunks FROM docs ORDER BY id ASC;

+------+---------------------------------------+--------+
| id   | title                                 | chunks |
+------+---------------------------------------+--------+
|    1 | Backup and restore runbook            |      9 |
|    2 | Monitoring and alerting guide         |      2 |
|    3 | Getting started with the CLI          |      2 |
|    4 | TLS and certificates for the HTTP API |      2 |
+------+---------------------------------------+--------+

One of those nine is the certificate-rotation paragraph. It matches the query almost exactly, so the document wins by a wide margin: 0.254 against 0.700.

More about the chunking strategies

Strategy	Vectors per document	Column type	What it does
truncate	1	float_vector	Embeds as much as fits the model's window, drops the rest. The only mode available in older versions, and still the default.
mean	1	float_vector	Splits the whole document, embeds every piece, averages them into one vector.
fixed	N	float_vector_array	Fixed windows of max_tokens tokens.
recursive	N	float_vector_array	Splits on a separator hierarchy — paragraph, then line, then sentence, then space — keeping each piece within max_tokens.
sentence	N	float_vector_array	Sentence boundaries (Unicode UAX #29), packed up to max_tokens.

The important distinction is not how the text is cut. It is what a match means.

With one vector per document, search asks: "is this document, as a whole, similar to the query?" A single relevant paragraph is diluted by everything around it, and a document that covers five topics ends up not really matching any of them.

With one vector per chunk, search asks: "does this document contain something similar?" Each chunk competes on its own merits, and Manticore returns the document once, scored by its best chunk.

truncate — keep it when your documents are short

title text,
v float_vector knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='title'

This is what you already have; chunk_strategy='truncate' is the default and you never have to write it. It's the right choice — and the fastest, and the smallest — whenever your text genuinely fits the model's window: product titles, short descriptions, tags, chat messages, log lines, search queries, commit subjects.

How much fits? More than most people assume, and less than they hope. all-MiniLM-L6-v2 takes 512 tokens, roughly 380 English words. text-embedding-3-small takes 8,192. If your 95th-percentile document is comfortably under the limit, stop reading and keep truncate.

When it hurts: anything long-form. Documentation pages, knowledge-base articles, contracts, transcripts, email threads, wiki pages, README files, incident postmortems.

mean — one vector, but the whole document

v float_vector knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='title,content'
  chunk_strategy='mean'

Manticore splits the document, embeds every chunk, and averages the chunk vectors into a single normalized vector. Storage and search cost are identical to truncate — one vector per document, one HNSW node — but nothing is thrown away.

Use it when:

• You want the tail to count but can't afford more vectors — a very large corpus where index RAM is the binding constraint.
• The column is a plain float_vector and you can't change the type (for example you're adding the column to an existing table with ALTER, which multi-vector strategies don't support).
• Your documents are about one thing, just long. A single product's full description, one recipe, one job posting.

Do not use it when a document covers several unrelated topics. Averaging a legal contract's indemnity clause with its payment terms produces a vector that sits between them and is close to neither. In our benchmark below, mean recovered about a third of the gap that chunking closes — a real improvement, and clearly not the same thing.

fixed — predictable, cheapest to reason about

v float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='content'
  chunk_strategy='fixed' max_tokens='256' overlap_tokens='32'

Cut every max_tokens tokens, no matter what the text is doing at that point. Chunk count is a straight function of document length, so index size is predictable before you load anything.

Use it when the text has no reliable structure to exploit: OCR output, scraped HTML that lost its paragraphs, machine transcripts without punctuation, log dumps, minified content. Also a fine default when you simply want the cheapest thing that stops truncation.

The cost: a boundary can land mid-sentence, and a chunk that begins in the middle of a thought embeds badly. That is exactly what overlap_tokens is for — see below.

recursive — the best general default for prose

v float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='title,content'
  chunk_strategy='recursive' max_tokens='256' overlap_tokens='32'

Same token budget as fixed, but each cut is pulled back to the nearest natural boundary: a blank line first, then a line break, then a sentence end, then a space. A chunk stops where the text stops, not where the counter runs out. The boundary is never dragged back past the midpoint of the chunk, so you don't get a stream of tiny fragments.

If you have used LangChain's RecursiveCharacterTextSplitter, this is the same idea, except it runs inside the database on the model's real tokens instead of characters, and there is nothing to install.

Use it for: Markdown and HTML documentation, wiki pages, knowledge bases, blog posts, README files, structured reports — anything written by a human in paragraphs. This scored highest on deep content in our benchmark.

sentence — when a chunk must be a complete thought

v float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='content'
  chunk_strategy='sentence' max_tokens='256' overlap_tokens='32'

Detects sentence boundaries with Unicode UAX #29, then greedily packs whole sentences until the token budget is reached. A chunk never starts or ends mid-sentence. A single sentence longer than the budget is split by the token window, as a last resort.

Use it for: support tickets and email threads, chat and meeting transcripts, legal and policy text, news, customer reviews, medical and scientific abstracts — anything where a fragment of a sentence changes or destroys the meaning. It is also the strategy to pick when chunks will be fed to an LLM afterwards, because a chunk that ends mid-clause reads badly in a prompt.

sentence is a little more conservative than recursive: it produced fewer, cleaner chunks in our tests and scored about the same on recall@5.

The three knobs

chunk_strategy  = truncate | mean | fixed | recursive | sentence
max_tokens      = chunk size in tokens; 0 (default) = the model's own limit
overlap_tokens  = tokens shared between consecutive chunks; needs a non-zero max_tokens
max_chunks      = ceiling on vectors per document; 0 (default) = unlimited

max_tokens is capped at what the model can actually accept — ask for 4,096 on a 512-token model and you still get 512, not an error. Smaller chunks mean sharper matches and more vectors; larger chunks mean more context per vector and fewer of them. For English prose, 128–512 covers almost every use case; we used 256 throughout the benchmark.

overlap_tokens repeats the tail of each chunk at the head of the next, so a sentence that straddles a boundary still appears intact somewhere. 10–20% of max_tokens is the usual setting. Manticore guarantees forward progress: fixed and recursive cap the overlap at half the chunk size, and sentence re-seeds the next chunk with at most overlap_tokens worth of trailing whole sentences while always advancing by at least one sentence. It requires an explicit non-zero max_tokens — overlap against "whatever the model's limit happens to be" isn't a meaningful setting, so Manticore rejects it.

max_chunks limits the impact of unusually large documents. Without it, a 400-page PDF pasted into one row becomes thousands of HNSW nodes. With it, Manticore merges the overflow into the last kept chunk, then truncates it to the model's window when embedding:

-- a ~600-token document, chunked at 64 tokens
chunk_strategy='fixed' max_tokens='64'                  -- 22 vectors
chunk_strategy='fixed' max_tokens='64' max_chunks='3'   --  3 vectors

Use it as a guard rail against outliers, not as a way to save memory across the board.

What search looks like

Nothing about your query changes from before. There is no chunk table, no nested field, no join, no GROUP BY. Here is the complete example:

DROP TABLE IF EXISTS notes;
CREATE TABLE notes (
  title text,
  body text,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,body'
    chunk_strategy='sentence' max_tokens='32'
);

INSERT INTO notes (id, title, body) VALUES
  (1, 'Certificate rotation',
   'The replication certificate is not the one the HTTP API uses. Generate the new key on the node being rotated and sign it with the cluster authority. Update the paths and reload, one node at a time, confirming every peer reports as synced before you move on.'),
  (2, 'Disk pressure',
   'When a data directory crosses eighty percent the merge scheduler stops compacting and the node starts refusing writes. Free space first, then trigger a manual OPTIMIZE. Adding a disk without draining the queue only postpones the problem.'),
  (3, 'Slow queries',
   'Turn the query log on before guessing. Most investigations end at a single query nobody knew was being issued, usually one that sorts on an unindexed attribute over the whole table.');

SELECT id, title, knn_dist() FROM notes
WHERE knn(chunks, 3, 'how do I replace an expiring certificate on every node');

+------+----------------------+------------+
| id   | title                | knn_dist() |
+------+----------------------+------------+
|    1 | Certificate rotation | 0.51039070 |
|    2 | Disk pressure        | 0.91703475 |
|    3 | Slow queries         | 1.02606630 |
+------+----------------------+------------+

max_tokens='32' is small on purpose here, so that these short notes actually split and you can see the multi-vector behaviour on a toy dataset. LENGTH() on the vector column shows how each document was divided:

SELECT id, title, LENGTH(chunks) AS n FROM notes ORDER BY n DESC LIMIT 5;

+------+----------------------+------+
| id   | title                | n    |
+------+----------------------+------+
|    1 | Certificate rotation |    2 |
|    2 | Disk pressure        |    2 |
|    3 | Slow queries         |    2 |
+------+----------------------+------+

Six vectors, three rows back. Search follows the rules described in Multiple vectors per document:

• A document matches if any of its vectors is near the query vector.
• Manticore returns each match exactly once. knn_dist() is the distance to its closest chunk.
• k counts documents, not vectors. knn(chunks, 3, ...) means three documents.
• A document with no vectors is never returned.

The same query over HTTP:

POST /search
{
  "table": "notes",
  "knn": {
    "field": "chunks",
    "query": "how do I replace an expiring certificate on every node",
    "k": 3
  },
  "_source": ["title"]
}

...
{
  "_id": 1,
  "_score": 1,
  "_knn_dist": 0.51039070,
  "_source": { "title": "Certificate rotation" }
}
...

Everything else on the KNN page keeps working as before: filtering, prefilter and postfilter strategies, quantization, early termination, and rescoring.

Does it actually help? Numbers on our own manual

We tested the feature on the Manticore English manual — 189 pages and about 298,000 words, ranging from a two-paragraph note to a 39,000-word changelog.

The query set is generated mechanically, not hand-picked. For every page we took its section headings, kept only headings that are unique across the whole manual, and split them in two:

• Deep-content queries (419) — headings that appear after the first ~1,200 characters of their page. That is a deliberately conservative line: the model's window is 512 tokens, roughly 2,000 characters, so a few of these still point at text truncate can partly see. The gap below is therefore an understatement, not an exaggeration.
• Head-content queries (88) — headings inside the first ~1,200 characters. The control group: content truncate can already see.

A query is a hit if KNN returns the page the heading came from, within the top k. Model: Xenova/all-MiniLM-L6-v2 (384 dims, 512-token window) running on Manticore's ONNX backend. Hardware: 32 threads. max_tokens='256', overlap_tokens='32' for the multi-vector strategies. Quality numbers are deterministic for a given index; the timings are a single run per strategy on an otherwise idle box.

Deep content — what chunking is for

Strategy	Vectors	Ingest	Index RAM	hit@1	hit@5	hit@10	MRR
truncate	189	21 s	4.2 MB	33.7%	55.1%	63.2%	0.44
mean	189	72 s	4.2 MB	43.9%	65.2%	74.7%	0.54
fixed	3,430	73 s	9.5 MB	56.3%	81.1%	86.2%	0.68
recursive	4,664	86 s	11.7 MB	58.7%	83.3%	89.5%	0.70
sentence	4,041	79 s	10.6 MB	55.4%	83.5%	89.0%	0.68

Chunking turns a coin flip into a working search. recall@5 goes from 55.1% to 83.3%, and the rank of the right answer improves just as much — MRR 0.44 → 0.70. Of the queries truncate could not answer in the top 5 at all, recursive recovers roughly two thirds.

mean lands where you would expect: it recovers about a third of the gap for free, because it costs exactly nothing extra to store or search.

Head content — the control group

Strategy	hit@1	hit@5	MRR
truncate	65.9%	86.4%	0.74
mean	59.1%	83.0%	0.69
fixed	60.2%	83.0%	0.71
recursive	58.0%	86.4%	0.70
sentence	56.8%	85.2%	0.69

For completeness, the control group is worth reading carefully. For content that the model could already see, truncate is still the most precise at rank 1 — 65.9% against 58.0% for recursive. A whole-document vector carries the page's overall topic, and when the query is about the page's opening subject, that context helps.

By rank 5 the difference is gone: recursive matches truncate exactly at 86.4%. So the trade is a few points of top-1 precision on content near the beginning, in exchange for +28 points of recall on everything else. For a documentation search, a help center, or any RAG retriever that feeds 5–10 passages to an LLM, that is not a close call.

Cost

• Index RAM: 4.2 MB → 11.7 MB, about 2.5×, for ~25× as many vectors. Vectors are only part of what an RT table stores. The HNSW graph over those vectors also takes longer to build during chunk saves and OPTIMIZE, though Manticore builds it across all your cores.
• Data load: 21 s → 86 s for 189 documents. Chunking means embedding the whole corpus instead of the first 380 words of each document, and the time scales with it. This is embedding cost, not chunking cost — the splitting itself is not measurable next to inference.
• Query response time: 6.3 ms → 8.5 ms at p50. HNSW handles 4,664 vectors about as easily as 189 — see 2-pass HNSW, batched distances and AVX-512 for what carries that.

If you use a paid embedding API, read that ingest number as a bill: chunking sends your whole corpus to the model instead of the head of each document, and you pay for every token of it. Local ONNX models have no per-token cost, which is a large part of why we made them fast.

Recommendations for choosing a chunking strategy

Your data	Start with
Titles, names, short descriptions, tags, log lines	truncate
Long but single-topic; or RAM is the hard limit; or the column is an existing float_vector	mean
Documentation, wikis, knowledge bases, articles, READMEs	recursive, max_tokens 128–256
Support tickets, email, transcripts, legal text, reviews	sentence, max_tokens 128–256
OCR, scraped HTML, machine transcripts, unstructured dumps	fixed, max_tokens 256, plus overlap
Chunks will be passed to an LLM as context	sentence, max_tokens 384–512

Overlap is deliberately absent from most of those: our sweep below could not measure a benefit from it on structured prose, and it costs vectors. Add it when a thought routinely straddles a boundary — unstructured transcripts, OCR, long narrative without paragraph breaks.

How big should a chunk be?

Chunk size is the setting that actually affects your results. The trade is direct: a smaller chunk is a sharper match on one idea, a larger chunk carries more context but dilutes each idea inside it. A paragraph buried in a long document only becomes findable once the chunk size is small enough to give it a vector of its own.

We ran another test: recursive on the same 189-page manual, three chunk sizes × three overlap settings, and the same 419 deep queries. Two trends stand out: quality rises as chunks get smaller, while overlap adds cost without improving quality much.

Note that the Y axis starts at 78%, not zero — the whole spread is about six points, so a zero-based axis would flatten it into a straight line. The numbers behind the chart:

max_tokens	overlap_tokens	Vectors	Index RAM	deep hit@5	deep MRR
128	0	8,256	17.3 MB	85.2%	0.718
128	13	9,328	18.7 MB	85.7%	0.705
128	32	11,623	22.6 MB	85.2%	0.694
256	0	3,984	10.0 MB	83.1%	0.657
256	26	4,525	10.9 MB	84.5%	0.681
256	64	5,569	12.5 MB	83.5%	0.689
512	0	1,973	6.9 MB	80.2%	0.655
512	51	2,191	7.2 MB	79.2%	0.651
512	128	2,666	8.0 MB	79.7%	0.660

What we see:

Smaller chunks win, consistently. Going from 512 to 128 tokens buys about five points of recall@5 (80.2% → 85.2%) and a large jump in ranking quality (MRR 0.655 → 0.718). It costs 4× the vectors and 2.5× the index RAM. Below 128 the chunks stop containing a whole thought, so this is not a slope you ride forever — but on long technical prose, 128–256 beat 512 every time.

Overlap did essentially nothing for quality, and was not free. At 128 tokens, going from no overlap to 25% overlap moved recall@5 from 85.2% to 85.2% while adding 41% more vectors and 5 MB of RAM. The pattern holds at every size: the spread across overlap settings (±1.5 points) is within the noise of a 419-query set, while the cost is not. This lines up with Chroma's chunking evaluation, where plain recursive splitting at 200 tokens with no overlap scored 88.1% recall — within a few points of an LLM-driven splitter at 91.9% — and it is the opposite of the "always use 10–20% overlap" advice you will read in most RAG guides.

The honest caveat: this is one corpus, one model, and queries that look like section headings. Overlap earns its keep when a single fact routinely straddles a boundary — long unbroken narrative, transcripts without structure — and recursive already snaps cuts to paragraph and sentence boundaries, which does much of the same work. So treat "start at 128–256 with no overlap, add overlap only if you can measure it helping" as the default, and check it on your own data with the recipe below.

Comparing settings

You do not have to guess, and you do not need two tables. A table can carry several model-backed vector columns, each with its own strategy, all filled from the same fields on the same INSERT:

CREATE TABLE ab (
  title text,
  body text,
  sent_256 float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,body'
    chunk_strategy='sentence' max_tokens='256' overlap_tokens='32',
  rec_128 float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,body'
    chunk_strategy='recursive' max_tokens='128' overlap_tokens='16'
);

Load your corpus once, then run the same query against each column and compare. For example:

SELECT id, LENGTH(sent_256) AS sent_chunks, LENGTH(rec_128) AS rec_chunks FROM ab;
SELECT id, knn_dist() FROM ab WHERE knn(sent_256, 5, 'how do I rotate the replication certificate');
SELECT id, knn_dist() FROM ab WHERE knn(rec_128,  5, 'how do I rotate the replication certificate');

On a short runbook whose certificate section sits at the end, sentence/256 fits the whole document in a single chunk and answers at distance 0.515; recursive/128 splits it in two, isolates the certificate paragraph, and answers at 0.310. Same row, same model, same query — only the chunk size differs.

Build a dataset of real queries with answers you trust — even 50 is enough — and compare recall@5 across two or three columns, exactly as we did on the manual above. Then drop the losing column with ALTER TABLE ... DROP COLUMN and keep the winner.

Recipes

Documentation and help center search. Long Markdown pages, users asking questions in their own words. Chunk on structure and search across it:

CREATE TABLE docs (
  url string,
  title text,
  body text,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,body'
    chunk_strategy='recursive' max_tokens='192'
);

Note from='title,body': the fields are joined before chunking, so the page title lands in the first chunk and gives it context. For a worked end-to-end example of this shape, see Vector search on GitHub.

Support tickets and email threads. A thread is a sequence of complete messages; cutting one mid-sentence loses the fact you need. Keep the chunk count bounded, because threads have no natural length limit:

CREATE TABLE tickets (
  ticket_id bigint,
  customer string,
  status string,
  thread text,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='thread'
    chunk_strategy='sentence' max_tokens='256' overlap_tokens='32' max_chunks='64'
);

SELECT ticket_id, knn_dist() FROM tickets
WHERE knn(chunks, 10, 'customer was charged twice after upgrading')
  AND status = 'closed';

Filtering works exactly as it does for a single-vector column.

Contracts and policy documents. Clause-level retrieval is the entire point — nobody wants "the contract" back, they want the indemnity clause. Smaller chunks, generous overlap:

chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='body'
  chunk_strategy='sentence' max_tokens='128' overlap_tokens='32'

Product catalog with long descriptions. One product is one topic, and catalogs are large, so pay nothing extra:

embedding float_vector knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='name,description'
  chunk_strategy='mean'

RAG: retrieval for an LLM. Whatever you retrieve gets pasted into a prompt, so chunks should read as prose — this is the retrieval half of conversational search. Larger chunks, sentence boundaries, and ask for more of them:

chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='title,body'
  chunk_strategy='sentence' max_tokens='512' overlap_tokens='64'

Adding chunking to a table you already have. Multi-vector columns can't be added by ALTER — existing rows have no vectors and there's no way to backfill them yet. A single-vector strategy can:

ALTER TABLE docs ADD COLUMN v2 float_vector knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='title,body' chunk_strategy='mean';

ALTER TABLE docs REBUILD EMBEDDINGS v2;

For a multi-vector column, create the new table with the column in place and reindex into it.

How other engines handle this

Every vector engine now generates embeddings for you. Far fewer will split your document before doing it — and of those, most make you assemble it out of pipeline stages.

Engine	Embeds in-engine	Chunks in-engine	Strategies	One row per document at search time
Manticore Search	Yes — local + OpenAI / Voyage / Jina	Yes — chunk_strategy on the vector column	truncate, mean, fixed, recursive, sentence	Yes, native
Elasticsearch	Yes — inference endpoints	Yes	sentence (default), word, recursive (9.1+), none	Yes — semantic_text hides the chunks
OpenSearch	Yes — ML Commons	Yes — separate ingest processor	fixed_token_length, fixed_char_length, delimiter	Needs a nested field + nested query
Vespa	Yes — built-in embedders	Yes — indexing expression	fixed-length, sentence, custom	Yes
Azure AI Search	Yes — integrated vectorization	Yes — Split skill in a skillset	pages (chars), sentences	No — one row per chunk
PostgreSQL + pgai	Yes — background worker	Yes	character, recursive character	No — separate table, join and dedupe
Milvus / Zilliz	Yes — Function (2.6+)	No — app-side	—	—
Qdrant	Yes — Cloud Inference	No — app-side	—	—
Weaviate	Yes — vectorizer modules	No — app-side	—	—
Meilisearch	Yes — embedders	No — app-side	—	—
Typesense	Yes	No — open request	—	—
Apache Solr	Yes — LLM module (9.8+)	No	—	—
Pinecone	Yes — integrated inference	No — app-side	—	—
MongoDB Atlas	Yes — Automated Embedding	No — app-side	—	—

Every cell in the chunking column links to a source. A “yes” links to the feature's own documentation. An “app-side” links to that vendor's own guidance on chunking in your application — which is what they publish instead of an in-engine option. If we missed a feature, or one has shipped since, tell us and we will fix it.

Versions checked - 4 September 2026

The latest stable release of each product available that day:

Product	Version
Elasticsearch	9.5.3
OpenSearch	3.8.0
Vespa	8.750.13
Azure AI Search	REST API 2026-04-01
PostgreSQL + pgai	extension 0.11.2
Milvus / Zilliz	2.6.23
Qdrant	1.19.0
Weaviate	1.38.13
Meilisearch	1.53.1
Typesense	30.2
Apache Solr	10.0.0
Pinecone, MongoDB Atlas	hosted services, no version to pin

Two things stand out.

Chunking in-engine is still rare. Milvus, Qdrant, Weaviate, Pinecone, MongoDB Atlas, Typesense, Meilisearch and — since the 9.8 LLM module — Apache Solr will all run the embedding model for you, and every one of them will happily truncate your 4,000-word document without saying so. The splitting is your problem, in your application, in a language and a library that has no idea what tokenizer the model uses.

Where chunking exists, the plumbing usually leaks. OpenSearch gets you there with a text_chunking processor feeding a text_embedding processor writing into a nested field, queried with a nested query and a score mode. Azure AI Search wants a skillset with a Split skill, an embedding skill and index projections — and returns one result row per chunk, so grouping back to documents is on you. pgai Vectorizer writes chunks to a second table, so every query is a join plus a DISTINCT ON. Elasticsearch's semantic_text is genuinely close to Manticore's model: chunking settings on the inference endpoint, chunks hidden inside the field, one hit per document.

Manticore does the same thing with less surface area: the strategy is an option on the column, the chunks are the column's value, and search returns documents. If you are weighing the whole stack rather than this one feature, we have written up the comparison with Elasticsearch and with Turbopuffer too.

What chunking does not fix

Chunking solves one problem well — a document longer than the model's window is no longer half-invisible. It does not make retrieval perfect, and two known gaps are worth naming.

A chunk does not know where it came from. Split a document and you get a paragraph that says "do one node at a time and confirm every peer reports as synced" with no indication of what is being rotated, or which product it belongs to. Anthropic's contextual retrieval work put numbers on this: prepending a short, chunk-specific description of the surrounding document before embedding cut top-20 retrieval failures by 35%, and by 49% combined with a contextual BM25 index.

Manticore does not do this for you. FROM joins its fields with a space before chunking, so listing title first puts the title at the head of the text that gets split — which means it lands in the first chunk and only that one. Every chunk after it is on its own:

-- 'title' leads, so its words are in chunk 1; chunks 2..N never see them
from='title,body'

If you need every chunk to carry context, you have to build it into the stored text yourself before inserting — for example by repeating a short heading at the start of each section of body. There is no per-chunk prefix option today.

Chunk boundaries are decided before the model sees the text. Manticore splits, then embeds each piece independently — the standard approach, and what every engine with in-engine chunking in the table above does. An alternative called late chunking inverts it: run a long-context model over the whole document first, then pool the token embeddings into chunks, so each chunk vector carries context from the rest of the document. It needs a long-context model and more compute per document, and Manticore does not do it today. If your documents depend heavily on cross-paragraph context, it is worth knowing the option exists.

Neither gap changes the basic result: for long documents, chunked retrieval beats truncated retrieval by a wide margin, and reaching it means adding chunk_strategy to one column.

Limits and gotchas

max_chunks discards text. Manticore merges overflow into the last kept chunk, then truncates it to the model's window. Nothing warns you. It's a guard rail for outliers, not a way to save memory across the board.

Remote models chunk by bytes, not tokens. OpenAI, Voyage and Jina have no local tokenizer, so Manticore falls back to a deliberately conservative estimate of 3 bytes per token — a chunk lands under the provider's cap rather than over it. In practice max_tokens='N' becomes an N × 3-byte window. We measured it against a stub endpoint with a 3,599-byte document and the fixed strategy:

max_tokens	Byte window	Chunks produced
100	300	12
200	600	6
400	1,200	3

English prose runs closer to 4 bytes per token, so on a remote model you get chunks roughly a quarter smaller than the number you asked for — set max_tokens about 30% higher than you would for a local model to land in the same place. If exact boundaries matter, use a local model, where splitting is done on the model's real tokens.

Multi-vector columns can't be added with ALTER. ALTER TABLE ... ADD COLUMN and ALTER TABLE ... REBUILD EMBEDDINGS on a model-backed float_vector_array are rejected. Recreate the table instead. Both work normally on a float_vector, including with mean.

Chunking applies to auto embeddings only. Vectors you insert yourself are stored exactly as given — Manticore never re-cuts data you supplied. chunk_strategy without model_name is a DDL error, on purpose.

embeddings is a reserved word. EMBEDDINGS is a DDL keyword (ALTER TABLE ... REBUILD EMBEDDINGS), so a column literally named embeddings is a syntax error unless escaped. Use escaping if you need that name.

Queries are not chunked. A query is embedded whole, as a single vector. That is what you want: chunking exists to make a long document findable, not to split a fifteen-word question.

The DDL tells you when a combination is wrong, at CREATE TABLE time rather than at the first insert:

mysql> CREATE TABLE t (title text, v float_vector ... chunk_strategy='sentence');
ERROR 1064: chunk_strategy='sentence' produces several vectors per document
            and requires a float_vector_array attribute

mysql> ... chunk_strategy='fixed' overlap_tokens='32');
ERROR 1064: overlap_tokens requires an explicit non-zero max_tokens

mysql> ... chunk_strategy='paragraph');
ERROR 1064: unknown chunk_strategy 'paragraph'; expected truncate, mean, fixed,
            recursive or sentence

mysql> ... chunk_strategy='truncate' max_tokens='128');
ERROR 1064: chunk_strategy='truncate' ignores max_tokens, overlap_tokens and max_chunks

Try it

The shortest path to a working chunked semantic search:

DROP TABLE IF EXISTS docs;

CREATE TABLE docs (
  title text,
  content text,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,content'
    chunk_strategy='recursive' max_tokens='256' overlap_tokens='32'
);

INSERT INTO docs (id, title, content) VALUES
  (1, 'Backup and restore runbook', 'Nightly backups run at 02:00 UTC ... ');

SELECT id, title, knn_dist() FROM docs
WHERE knn(chunks, 5, 'how do I rotate the replication certificate');

No model to download by hand, no splitter to pick, no pipeline to maintain. One column option, and the parts of your documents that used to be invisible start showing up in results.

Full reference: Chunking strategies and Multiple vectors per document in the manual. Questions and bug reports on GitHub.