Add index.subsample() to resize an index without re-encoding - #33
Open
wxiao0421 wants to merge 1 commit into
Open
Add index.subsample() to resize an index without re-encoding#33wxiao0421 wants to merge 1 commit into
wxiao0421 wants to merge 1 commit into
Conversation
This was referenced Jul 29, 2026
wxiao0421
force-pushed
the
feat/persist-corpus-and-deterministic-load
branch
from
July 29, 2026 20:07
d64f37e to
cb513d9
Compare
wxiao0421
force-pushed
the
feat/index-subsample
branch
from
July 29, 2026 20:07
ded6bd5 to
839e2ee
Compare
vcai4071
approved these changes
Jul 29, 2026
Comment on lines
+423
to
+434
| if n_keep is None or n_keep >= available: | ||
| if n_keep is not None and n_keep > available: | ||
| LOG.info( | ||
| "Requested %d %s examples but the index only has %d - keeping all of them.", | ||
| n_keep, | ||
| label, | ||
| available, | ||
| ) | ||
| # Copy the corpus list so callers cannot mutate the original through the copy. | ||
| return embeddings, (list(corpus) if corpus is not None else None) | ||
|
|
||
| indices = torch.randperm(available, generator=generator)[:n_keep] |
Contributor
There was a problem hiding this comment.
edge case: if we are doing a grid search with positive to negative ratio, if there is one run in the search where we want to keep all the positive rows, then torch.randperm(available, generator=generator) will not run, which means the stateful generator will be at a different state, and thus the negative embedding set will be different.
Encoding a sentence produces the same numbers regardless of which index it ends up in, so a small index is just a large one with rows removed. Today there is no way to exploit that: resizing means re-encoding the whole corpus from scratch, which makes sweeping index size - the highest-leverage knob there is - prohibitively expensive for anyone outside the internal pipeline. subsample() copies the rows you want instead. On the shipped example index, building a 3x3 grid of (n_positive, ratio) configurations takes 9 ms, against roughly 12.5 s to re-encode the same 16,100 rows. Design decisions worth reviewing: - Returns a new instance and never mutates the receiver. Callers loop over sizes, so if each call shrank the original, run 2 would start from run 1's leftovers and every result after the first would be silently wrong. This differs from _apply_negative_ratio, which mutates in place - fine there because it runs once inside load(). - Positives are chosen first, because the ratio is defined relative to how many positives survive. - Invalid sizes raise rather than warn-and-continue. A grid search that quietly ignored a bad argument would emit rows describing an index nobody asked for. - scale_fn, encoding kwargs and model card all carry over; a dropped scale_fn would silently change scores for models like E5. The sentence model is shared rather than reloaded, since reloading it would reintroduce the cost being removed. - Reuses the _take_rows helper and the local-torch.Generator seeding convention, so corpus alignment is handled in one place across the whole library. Adds 15 tests, including that the original index is provably unchanged after repeated calls, and that corpus texts still track their own embedding rows on both sides of the index. Co-authored-by: Cursor <cursoragent@cursor.com>
wxiao0421
force-pushed
the
feat/index-subsample
branch
from
July 29, 2026 23:49
839e2ee to
2287470
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Encoding a sentence produces the same numbers regardless of which index it ends up in, so a 1,000-example index is just a 10,000-example one with 9,000 rows deleted. There is currently no way to exploit that: resizing means re-encoding the whole corpus from scratch, which makes sweeping index size — the highest-leverage knob there is — too expensive to bother with outside the internal pipeline.
subsample()copies the rows you want instead:On the shipped
examples/hate_speech_modelindex, building a 3×3 grid takes 9 ms against roughly 12.5 s to re-encode the same 16,100 rows. Around 1,400× faster, and the gap widens with the grid.Design decisions worth reviewing
_apply_negative_ratio, which mutates in place — fine there, because it runs once insideload().neg_to_pos_ratio=Noneleaves negatives untouched.scale_fn, encoding kwargs and model card carry over, and the sentence model is shared rather than reloaded. A droppedscale_fnwould silently change scores for models like E5._take_rows()and the private-torch.Generatorseeding convention from Persist the corpus and make index loading deterministic #32, so corpus alignment is handled in one place.This is a new method: no existing signature or behaviour changes, and nothing calls it yet. The caller arrives in #34.
Test plan
15 new tests, 82 total (was 67). The load-bearing ones assert that the original index is unchanged after repeated calls, that repeated calls are independent of each other, and that corpus texts still track their own embedding rows on both sides. Also covers seeding, clipping when over-asking, carry-over of
scale_fn/kwargs/model card, corpus-less indices, andValueErroron invalid sizes.flake8output is unchanged from base anddocs/check_docs_sync.pyreports in sync.No CI checks appear because
.github/workflows/test.ymlonly triggers on pull requests targetingmain. Run locally on Python 3.10; the full 3.10/3.11/3.12 matrix runs once the stack retargets.