Say where the content sketch is stored, and why it has no table - #133
Merged
Conversation
Step 11 named the metric and parked its parameters, but never said what the sketch physically is or where it lives. A reader arriving at the step finds "shingled MinHash" and reasonably expects a table — the shape most MinHash implementations take, because most of them do corpus-wide nearest-neighbour search and need banding to make it tractable. Neither is true here. The sketch is one opaque BLOB in the note's own catalog row, which is Decision 7's "stored per note in the catalog beside the hash" taken literally, and the shingles themselves are never stored at all: the k-grams are transient inputs to the hashing, and only the fixed-width signature survives. Storing the shingle set would amount to a second copy of the note's content. The absence of banding is a consequence of the query, not an omission. Decision 7 compares an unmatched path against the sketches of recently-missing notes, so the candidate set is the handful that went missing in one scan rather than the engram; loading those rows and comparing pairwise is the whole algorithm. Records the coupling that follows — a candidate set that ever widens to the whole engram turns the column into a full scan, and a band table is the additive escape hatch — so the next person to want corpus-wide similarity finds the reasoning instead of rediscovering it. Also states why the column is an untyped BLOB: the width is this step's to decide, and step 3 stores the bytes without interpreting them. Docs only; no code and no user-facing change, so docs/manual-test-plan.md is untouched — no cases added, changed, or invalidated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Coverage after merging worktree-sketch-storage-note into main will be
Coverage Report |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Answers, in the plan itself, the question Step 11 provoked: where does the
content sketch actually live?
Step 11 named the metric and parked its parameters, but never said what the
sketch physically is or where it is stored. A reader arriving at "shingled
MinHash" reasonably expects a table — that is the shape most MinHash
implementations take, because most of them do corpus-wide nearest-neighbour
search and need banding to make it tractable.
Neither holds here, and now the step says so:
BLOBin the note's own catalog row, not a table of its own.That is Decision 7's "stored per note in the catalog beside the hash" taken
literally, and it is the
sketchcolumn Step 3 already specifies.hashing; only the fixed-width signature survives. Storing the shingle set
would amount to a second copy of the note's content, since a k-gram set is
roughly the size of the text it came from.
than an omission. Decision 7 compares an unmatched path against the sketches
of recently-missing notes, so the candidate set is the handful that went
missing in one scan — loading those rows and comparing pairwise is the whole
algorithm.
later feature ever wants similarity across the whole engram instead of
across the missing set, the
BLOBcolumn becomes a full scan and a bandtable keyed by band hash is the additive escape hatch. Not to be built
before something needs it.
BLOBwith no fixed length: the sketch widthis that same bullet's decision to make, and Step 3 round-trips the bytes
without interpreting them.
Note on ordering
This says "the
sketchcolumn that step 3 already created", which matches thecolumn list in the plan's own Step 3 section. The code behind it is in #131,
still open — if that PR's shape changes in review, this wording is worth a
second look, but it is written against the plan's spec rather than against the
implementation.
Docs only — no code, no user-facing change, so
docs/manual-test-plan.mdisuntouched: no cases added, changed, or invalidated.
🤖 Generated with Claude Code