Extract common code paths in indexing pipeline #2275
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #2275 +/- ##
============================================
+ Coverage 63.41% 63.82% +0.41%
- Complexity 1287 1296 +9
============================================
Files 196 197 +1
Lines 11463 11251 -212
Branches 1457 1421 -36
============================================
- Hits 7269 7181 -88
+ Misses 3672 3593 -79
+ Partials 522 477 -45 ☔ View full report in Codecov by Sentry. |
lintool
changed the title
| public int memoryBuffer = 16384; | ||
|
|
||
| @Option(name = "-threads", metaVar = "[num]", usage = "Number of indexing threads.") | ||
| public int threads = 4; |
There was a problem hiding this comment.
Thinking out loud: what about defaulting to Runtime.getRuntime().availableProcessors() so that indexing concurrency would automatically try to use all available resources, without requiring users to pass the right number of threads? (Possibly for another PR)
There was a problem hiding this comment.
Yes, potentially - but:
(1) defaulting to consuming all available resources can be rude ;)
(2) it depends on the corpus as well... e.g., if the corpus only has 10 files, then 16 threads isn't going to help since we only do per-file parallelism.
|
Tests are proceeding, but not finished yet. Stand by. |
Major refactoring of indexing pipeline (IndexCollection, IndexHnswDenseVectors, and IndexInvertedDenseVectors), extracting common code paths into AbstractIndexer.
@tteofili @MXueguang still running tests, but ready for review in parallel.