docs(search-semantic): correct the boundary-aware chunker settings

[marevol]

Documents the boundary-aware chunker from codelibs/fess#3211. Tracked as codelibs/fess#3231 (filed there because issues are disabled on this repository).

Rebased onto the current master. #475 landed the same three settings while this was open, so this PR is now a correction and extension of that text rather than the original addition. The overlapping notes the two branches each added are merged into one.

Corrections to what #475 landed

All three checked against LengthChunker / ChunkBoundaryFinder / ChunkVectorHelper.

1. The worst case is 840 characters, not "about 841". The bound is chunk_size + max(lookahead, 2 * ChunkBoundaryFinder.MAX_CLUSTER_ADJUST) = 800 + max(40, 32). The further + 1 branch in findChunkEnd requires ideal == start + 1, which MIN_CHUNK_SIZE makes unreachable from LengthChunker. The floor — 800 - 160 — was not stated at all, so the table now gives the full range: 640 to 840 characters. That matters to anyone who sized chunk_size against an embedding model's token budget.

2. A cut does not move to "the nearest suitable break … preferring" one kind over another. Candidates are tiered, and the nearest candidate of the highest tier present wins: a line break or sentence end beats any clause separator or space no matter how much farther back it is, and those beat a writing-system change. This is the part the upstream PR got wrong three times in its own prose.

3. content_chunker.length.overlap snaps its restart point to a boundary too. snapOverlapStart can only move it earlier, and its window is capped at the configured overlap, so the effective overlap is between the configured value and twice it. Not documented at all before.

Additions

content_chunker.max_chunks_per_document — a document that crosses the cap is marked skipped and receives no embeddings at all: it stays keyword-searchable but drops out of semantic search. Boundary-aware splitting also makes chunks shorter, so a document yields roughly 3% to 25% more chunks than a fixed-length split (measured: +2.7% on Japanese prose, +9.3% on English prose, +24.8% worst case) — the cap is easier to reach than it was. The row now says both.

Re-chunking requires a recrawl — changing any of these settings only affects documents chunked afterwards; a document already stored as a chunk array keeps its boundaries until it is re-crawled.

Kept from #475

The guarantee that only the cut point moves, so concatenating a document's chunks still reproduces its content exactly, and the description of the independent grapheme-cluster overshoot that ignores lookahead_percent.

Scope

All seven languages: en, ja, de, es, fr, ko, zh-cn.

Verification

A full Sphinx build is not reproducible from this repository alone (conf/conf.py is templated by the builder), so the checks are structural:

• every content_chunker.* list-table still has the same shape as master (23 rows / 69 cells) in each of the seven languages;
• the number of .. note:: blocks per file is unchanged from master (the two overlapping notes were merged, not stacked);
• all three boundary keys are present in each of the seven;
• docutils parses all seven files with a diagnostic set identical to master (zero non-Sphinx diagnostics);
• no line exceeds the 123-column display width each file already had (East-Asian-Width aware).