Refine the Flash tree with merge - #375
Merged
Merged
Conversation
Carried from bench/flash-cost; benchmark artifacts left there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes to how the Flash tree is refined.
1. The default path merges instead of thinning
Both operations collapse subtrees whose structure does not earn its keep. Merge
subsumes thinning and is better on the two things that matter.
Thinning collapses a subtree spanning 1 page, or 2 pages on a tree of more than
20 nodes. Merge collapses when
S(v) <= tree_cost(v), and any node with childrencosts at least
R + 1 = 2, so everything thinning would collapse is collapsed bymerge as well. Merge additionally collapses larger subtrees that do not beat a
linear scan, on a principled threshold rather than a node-count heuristic.
Thinning also discarded the headings of a collapsed subtree. Merge keeps them on
the parent as
key_items. On the 2023 annual report thinning lost 55 titles;merge loses none.
utils.page_level_thinningis retained as legacy with no callers.Measured over the nine documents in
examples/documents/:validate()after mergemerge_treeidempotent2. Same-page merge under
--optimize(@张鸣天)Retrieval is page-granular, so frontier siblings covering identical pages cannot
be told apart: an agent routed to any of them reads the same text. They now
collapse into one node whose title is the union of theirs, which a leaf summary
call rewrites when the node is large enough to earn one. Deterministic, no LLM.
This runs inside
--optimize, so--optimize mergenow does strictly more thanthe default rather than producing the same tree.
3. Internal marker no longer reaches the output
Same-page merge tags a node with
_same_pageto tell summary generation thetitle was synthesised and may be rewritten. That marker was stripped on the two
page_index_flashpaths but not onoptimize_tree()or thepython3 -m pageindex.tree_optimizeCLI, so both emitted it into their trees. Itis now stripped at those exits too, while
optimize()still carries itinternally so summaries can retitle.