Filter HNSW during BFS in a single pass #1940
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1940 +/- ##
==========================================
- Coverage 83.97% 83.93% -0.04%
==========================================
Files 329 329
Lines 18708 18708
==========================================
- Hits 15710 15703 -7
- Misses 2998 3005 +7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
javitonino
force-pushed
the
single_vectors_bfs_search
branch
from
6a54e6f
to
911a9cd
Compare
| // We just need to perform BFS, the replacement is the closest node to the actual | ||
| // best solution. This algorithm takes a lazy approach to computing the similarity of | ||
| // candidates. | ||
| let mut visited_nodes = HashSet::new(); | ||
| let mut candidates = VecDeque::from([x]); | ||
| let mut visited = HashSet::new(); |
There was a problem hiding this comment.
We can go ~30% faster by replacing this with a bitfield or similar. A HashSet is too slow in the case where we need to scan many nodes.
There was a problem hiding this comment.
I just did it for funsies 😝
There was a problem hiding this comment.
you can also use FxHashSet which is not cryptographically secure, so its way faster.
…b into single_vectors_bfs_search
…b into single_vectors_bfs_search
…b into single_vectors_bfs_search
| let mut neighbours = vec![(entry_point.node, 0.)]; | ||
| while crnt_layer != 0 { | ||
| for crnt_layer in (0..=entry_point.layer).rev() { |
There was a problem hiding this comment.
we were skipping layer 0
There was a problem hiding this comment.
hah, I thought it was on purpose. Should have asked!
Description
Describe the proposed changes made in this PR.
How was this PR tested?
Describe how you tested this PR.