Limit segment size for better indexing performance #135
Comments
generall
added a commit
that referenced
this issue
Jan 18, 2022
* add parameters to optimizer config * benchmark search speed in different segment sizes * use constructor for FilteredScorer * * Implement benchmarks for HNSW index search with different number of stored points * Fix minor issue in HNSW graph edge assignment * Update profiler with call-graph report generation * Add profiling guide * Add HNSW graph statistics test function (debug inly) * limit resulting segment size in merge optimizer * fix clippy * stop the music * fix clippy once again * fmt once again
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
Indexing speed of HNSW is not linear, so it might take a lot of time to build segment of size
2Nand relatively small time o build 2 segments of sizeN+N.Describe the solution you'd like
Limit maximal size of the segment to achieve faster indexing speed.
Merge optimizer should not perform merge with segments of size >=
max_size_limit.(Alternatively, it could check the size of resulting segment, but it might be a little harder to implement)
Describe alternatives you've considered
Check other existing OS implementations and investigate how do they solve the problem.
The text was updated successfully, but these errors were encountered: