v0.14.0
What's Changed
Bug Fixes
- fix template function specialization missing in hnsw by @inabao in #383
- fix template compile error by @LHT129 in #391
- fix the incompatibility of v0.12 and v0.13 in HNSW by @inabao in #388
- fix occasional failures of unittests caused by a high filter rate by @inabao in #394
- fix: Add missing header for readerset.h by @Coien-rr in #398
- fix the incorrect assignment of prefetch_jump_code_size_ by @inabao in #403
- fix compiler issues when using clang make libcxx by @ShawnShawnYou in #407
- fix eval_performance calculates recall incorrectly by @wxyucs in #409
- calculates recall by distances in eval_performance by @inabao in #413
- fix hgraph deserialize bug by @LHT129 in #417
- fix sq4_uniform train bug on gist dataset by @LHT129 in #435
- fix bug in LocalFileReader when use async io in DiskANN by @inabao in #450
- fix direct io asan bug by @LHT129 in #472
- fix clang 13 compile error by @nedchu in #458
- fix comiple error due to dependency missing of spdlog by @inabao in #480
- fix ef search bug on hgraph by @LHT129 in #478
- fix tool eval_performance output nothing without exporters setting by @wxyucs in #483
- add hdf5 files' attribute existence check before opening by @wxyucs in #518
- fix illegal instruction happened on rabitq by @wxyucs in #527
- update flatten datacell prefetch length by @wxyucs in #538
- fix compile error in TestGetExtraInfoById by @inabao in #542
- fix compile error for diskann while static link by @inabao in #552
- fix the issue where train_data_type_ is not initialized by @inabao in #554
- fix coredump for searching empty HGraph index by @inabao in #559
New Features
- introduce bucket datacell parameter by @LHT129 in #362
- add batch scan for quantizer compute by @LHT129 in #359
- introduce buffer io by @LHT129 in #326
- introduce bf16 simd operators by @LHT129 in #415
- add bucket datacell by @LHT129 in #385
- introduce async io (libaio implement) by @LHT129 in #411
- introduce an implement of kmeans-cluster by @LHT129 in #390
- add sparse dtype in dataset by @ShawnShawnYou in #325
- enable libaio for hgraph by @LHT129 in #419
- introduce ivf classify strategy by @LHT129 in #423
- Implement the merge function in HNSW by @inabao in #432
- introduce bf16 quantization by @LHT129 in #418
- support add in pyramid by @inabao in #427
- Introduce fp16 simd operators by @LHT129 in #436
- Support fp16 quantizer on HGraph by @LHT129 in #454
- introduce IVF index implement by @LHT129 in #456
- Support IVF Index by @LHT129 in #505
- support build index for different levels by @inabao in #465
- hgraph support extra info by @lyxiong0 in #486
- Support GetExtraInfoByIds on HGraph by @LHT129 in #530
- Support PCA for rabitq by @ShawnShawnYou in #529
- Support Hgraph Iterator Filter Search by @Carrot-77 in #543
- Supports getting the maximum and minimum label interface from the label by @Carrot-77 in #512
Improvements
- introduce different kinds of heap by @LHT129 in #355
- unified all io serialize method by @LHT129 in #357
- use trunc train for sq as default by @LHT129 in #356
- add minor optimize on performance of hnswalg by @ShawnShawnYou in #396
- optimize filter in HNSW by reduce CheckValid by @inabao in #397
- add libaio deps for ubuntu and centos by @LHT129 in #402
- update hgraph doc with buffer io config by @LHT129 in #399
- add libaio-dev for dockerfile by @LHT129 in #412
- extract the pruning strategy from HGraph by @inabao in #373
- remove the brute-force strategy in filter search by @inabao in #414
- support filter & range for searcher by @LHT129 in #378
- support build graph from existing graph storage in ODescent by @inabao in #406
- refactor pyramid from being index-based to datacell-based by @inabao in #370
- use empty_index_set for serialize empty index by @LHT129 in #429
- add random allocator test for pyramid by @inabao in #426
- improve coverage rate for simd(avx) by @LHT129 in #438
- refactor utils by @LHT129 in #433
- introduce vsag exception to wrapper vsag::Error by @LHT129 in #430
- implement automatic update of the total_count in the graph datacell by @inabao in #428
- add filter for eval_performance by @inabao in #420
- add ceil_int function to calulate int64's ceil by @LHT129 in #440
- add arm asan build & test for github action by @LHT129 in #442
- add performance pipeline in GitHub CI by @wxyucs in #421
- remove arm ci on circle_ci by @LHT129 in #448
- Support random projection by @ShawnShawnYou in #445
- update clang-tidy performance and readability options by @wxyucs in #424
- introduce index_impl layer by @LHT129 in #439
- support build one vector index in diskann by @inabao in #444
- add const referance for shared_ptr by @LHT129 in #460
- support cal batch distance by id for hgraph by @LHT129 in #441
- speed up functests for ci by @LHT129 in #457
- update cache version to v4 by @LHT129 in #468
- [skip ci]disable performance check in pull request by @wxyucs in #461
- [skip ci]update ci image refresh condition by @wxyucs in #462
- add different simd operator test for github action by @LHT129 in #443
- introduce CreateFastDataset by @LHT129 in #469
- add member func and static method check by @LHT129 in #473
- add disk io in the example of DiskANN by @inabao in #452
- implement index interface for brute_force by @LHT129 in #466
- speed up cluster by @LHT129 in #476
- use using instead of typedef by @LHT129 in #471
- update performance pipeline by @wxyucs in #477
- add compatibility check by @LHT129 in #487
- disable simd ci when no changes on simd/ by @LHT129 in #488
- use searcher on HGraph by @LHT129 in #463
- use distanceHeap instead MaxHeap on bruteforce by @LHT129 in #492
- refactor some param, move bruteforce to algorithm by @LHT129 in #490
- ensure thread-safe reading and writing for pyramid by @inabao in #455
- optimize filter in searcher by @inabao in #499
- replace progschj::ThreadPool in hgraph by @LHT129 in #503
- simplify the feature test for index by @LHT129 in #493
- use inner_index for ivf router by @LHT129 in #500
- add ignore_reorder config to drop high_precise codes by @LHT129 in #498
- introduce random train for ivf router by @LHT129 in #502
- refactor some recall test on random dataset by @LHT129 in #520
- update ivf expected recall in functests by @wxyucs in #524
- refactor pyramid and add an example by @inabao in #519
- support multi-threading in eval_performance by @wxyucs in #489
- calculate recall by distance in test_performance by @inabao in #504
- refactor extra_info_size by @LHT129 in #525
- some simd optimize by @LHT129 in #545
- optimize prefetch operators by @LHT129 in #547
- refactor hgraph to add point one by one by @LHT129 in #514
- reduce searcher allocator by @LHT129 in #555
- optimiaze avx512 fp32 IP compute operator by @LHT129 in #549
Other Changes
- [skip ci] refactor hgraph param doc by @LHT129 in #386
- [skip ci] Refresh example code link in README by @Coien-rr in #392
- support rabitq quantizer basic funcs by @ShawnShawnYou in #431
- [skip ci]support fix lint command by @wxyucs in #474
- reduce unnecessary conversions in sparse_graph_datacell by @inabao in #470
- fix compile bug in avx512 about fp16 and bf16 by @ShawnShawnYou in #495
- remove unnecessary code for filter by @inabao in #501
- support rabitq in Hgraph by @ShawnShawnYou in #467
- add tsan pipeline by @inabao in #497
- support sparse index by @inabao in #491
- skip threshold Calculation method by @Carrot-77 in #511
- remove unused code by @inabao in #521
- support simd of rabitq-fp32 by @ShawnShawnYou in #507
- add iter filter context interface by @Carrot-77 in #515
- Hnsw iter filter search by @Carrot-77 in #526
- remove the internal default resource creation while constructing engine by @inabao in #532
New Contributors
Full Changelog: v0.13.0...v0.14.0
