Repository navigation
v0.8.4 — Chunker whitespace fix and retuned PDF eval
Patch release with a RecursiveChunker correctness fix and a retuned PDF GraphRAG evaluation.
Chunker
Recursive splitting could emit a piece consisting only of a separator — for example a lone "\n" when pending splits were flushed ahead of an oversized one — which then became a ChunkSpec carrying no content. Those pieces are now dropped.
Skipped pieces still advance the internal search floor. Without that, the following piece could re-match an earlier occurrence of its own text and leave real characters uncovered by any chunk.
Contract change: splitting a whitespace-only text now returns no specs at all, where it previously returned one. The module docstring documents this.
PDF GraphRAG evaluation
The notebook and evaluation script previously raised max_chars to 1400 while dropping chunk overlap, to stop graph node labels from looking like duplicated mid-sentence fragments. The larger budget cost retrieval quality: section_hit_rate fell from 1.00 to 0.83 for the reading-expansion strategies, avg_ce_max dropped across all ten strategies, and context grew 39%.
Measuring the two knobs separately shows max_chars was the culprit, not the overlap — 1100/0 and 1100/120 score alike, while 1400/0 and 1400/120 both fail. Larger chunks dilute the evidence, which lowers the cross-encoder score and moves the top-1 vector hit into the wrong section.
Both now use 1100 characters with no overlap: labels stay readable, section_hit_rate returns to 1.00, avg_ce_max to 4.48 (from 4.54), and context is slightly cheaper than before the change. Only avg_term_recall gives up about a point, which is what the overlap was buying.
Evaluation outputs were regenerated and the notebook re-executed against this configuration.
Full Changelog: v0.8.3...v0.8.4