Additional index deserialization validation#4844
Closed
scsiguy wants to merge 2 commits intofacebookresearch:mainfrom
Closed
Additional index deserialization validation#4844scsiguy wants to merge 2 commits intofacebookresearch:mainfrom
scsiguy wants to merge 2 commits intofacebookresearch:mainfrom
Conversation
…search#4827) Summary: In validate_HNSW(), bounds check access to cum_nneighbor_per_level for each level of the deserialized HNSW. Reviewed By: junjieqi Differential Revision: D93903637
Summary:
Fix 1: IndexLattice r2 and dsq validation
- Tightens the r2 check from r2 >= 0 to r2 > 0. r2 must be greater
than zero to avoid a divide by zero during normalization.
- Adds a new check that 'dsq = d/nsq' is a power of 2 and >= 2. This
aligns with asserts in ZnSphereCodecRec and prevents an invalid
'cache_level' value of -1.
Fix 2: Binary hash invlists buffer validation
- Changes from pre-allocating the buffer with a computed size (then
overwriting via READVECTOR) to reading the buffer first and then
checking it is large enough. Since READVECTOR determines its size
from the serialized data stream, the previous logic didn't actually
verify that the buffer was fully filled with valid data.
Differential Revision: D94558548
Contributor
Contributor
|
This pull request has been merged in 6704852. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Fix 1: IndexLattice r2 and dsq validation
than zero to avoid a divide by zero during normalization.
aligns with asserts in ZnSphereCodecRec and prevents an invalid
'cache_level' value of -1.
Fix 2: Binary hash invlists buffer validation
overwriting via READVECTOR) to reading the buffer first and then
checking it is large enough. Since READVECTOR determines its size
from the serialized data stream, the previous logic didn't actually
verify that the buffer was fully filled with valid data.
Differential Revision: D94558548