Repository navigation
Replies: 1 comment
|
Real-world data point for this idea. Setup: GBrain 0.60.553 on Postgres, local embeddinggemma (768). One note is a running log of about 200 short, unrelated facts (about 120 KB). It splits into 47 chunks of about 2,600 characters, so each chunk mixes several facts. Test: 20 questions, each answered by a single fact in that note. Score is the right fact in the top 3 hybrid results, live brain:
On a throwaway copy, GBRAIN_MAX_CHUNK_TOKENS=64 (the only knob; process-wide, about 20x the chunks across the brain) scored 5/20, so shrinking every chunk is not the fix. Caveat: the questions were written after reading the facts, so the absolute scores are probably optimistic. The gap is the point. Two observations from the source (checked against v0.60.613.0):
A per-page opt-in, for example honoring |
Uh oh!
There was an error while loading. Please reload this page.
Moved from #7118, opened by @DaCal90 on 2026-10-20. Feature requests now live in Discussions > Ideas.
Setup. gbrain 0.60.306.0 on Postgres 17.11 with pgvector 0.8.7 HNSW and voyage-4 (1,024 dimensions). One page of about 530,000 characters, stored inline (under 1 MiB): a register of about 660 dated entries, one line each, most of them short; the kind of page your memory-boundaries guide (Large pages) says to keep whole: "A page stays one page whatever its size: one slug, one title, one set of links. Size no longer decides whether search can find it." (
docs/guides/memory-boundaries.mdlines 89 to 90). Your recursive chunker cuts it into about 220 chunks (about 2,400 characters of the page each, plus the overlap): it splits at blank lines, then at line breaks, merges whole lines until a chunk would pass 450 words (1.5 times its 300 word target), then overlaps chunks by 50 words. So chunks of short entries each hold several unrelated ones (a few dozen long entries span chunks of their own). Every chunk has its vector and the page is current.What we see. We asked 16 questions, each about one entry of this page, through
hybridSearch(limit 20, cache off) withexplain_targeton the page. In this run 13 of them are not returned: for 11 it reportstarget_not_retrieved(no arm has the page) and for 2target_beyond_limit(the vector arm has it, cut at the limit); 3 come back. For the 9 of those 11 whose best chunk is below the vector arm's cutoff or level with it (in the other 2 it clears the cutoff and the HNSW walk loses the page, a separate matter), the cosine similarity of the query vector to:The index is not the cause for the 8 below the cutoff: the cutoff is the lowest chunk the arm's own walk kept, so an exact search could only raise it; and the chunks holding the 9 entries each come back first when searched by their own vectors (all but 7 of the page's chunks do at the default ef_search 40, all but 1 at 400). The keyword arm never matches either, because no chunk of the page holds every word of the question (the missing words are the question's framing, not the entry's); that is its all-terms design, and we do not ask to change it. Seven of the 16 keyword arms also hit their time limit, a separate matter.
Why, as we read your code.
greedyMergeinsrc/core/chunkers/recursive.tsmerges line pieces until a chunk would pass 1.5 times the 300 word target, with no notion that the lines are independent, andapplyOverlapthen carries 50 words of the previous chunk into each. The one cap,GBRAIN_MAX_CHUNK_TOKENS(src/core/embedding-input-limit.ts, floor 64 tokens), is meant for embedding models with input limits: it applies to every page any process writes or embeds while it is set (embed passes split stored chunks too), a process run without it cuts the page back at its next write, and it slices the merged text by length (capByChars), not at line breaks. The same guide says "Don't split a page by hand to make it searchable." (line 94), and we found no supported way to keep such a page whole while each short entry ranks on its own vector.The ask. An entry-aware option for such a page, in its front matter: one entry per chunk at line breaks, entries joined only when a single entry is shorter than a floor, and no overlap across entries; long entries split as today. The page stays one page with one slug, and each short entry becomes a chunk whose vector is the entry's, with the page title as the title tier gives it. (The v0 design doc's
chunk_strategyfront matter key would be a natural home; it is not in the current code. Related: #1616, closed, named multi-chunk dilution in its eval design.) If there is a supported pattern for this that we missed, a pointer would be enough: as we read the code, the timeline goes through the same chunker, andsearch.long_page_unitwidens a hit already found rather than finding one.All reactions