v0.15.0

What's Changed

Bug Fixes

• fix rabitq recall decrease by @LHT129 in #565
• fix coredump when searching empty index by @inabao in #577
• fix bug due to narrow cast in HGraph by @inabao in #582
• [bugfix][hgraph] fix core on large dataset build by @LHT129 in #588
• fix abnormal latency statistics in eval_performance by @inabao in #594
• fix resize bug on hgraph concurrent add by @LHT129 in #602
• Update lint.yml to revert the changes by @antfin-oss in #647
• reduce resize memory on large size extrainfo by @LHT129 in #660
• fix compatibility of hnsw by @inabao in #624
• fix too much memory used on small dataset by @LHT129 in #673
• fix hnsw core dump when destructing by @ShawnShawnYou in #649
• fix curl related compile issue in cpr by @ShawnShawnYou in #665
• resolve initialization order issue for static variables by @inabao in #682
• init the cpuinfo before other simd functions by @LHT129 in #705
• disable simd instructions in diskann by @wxyucs in #698
• kmeans train for huge K and data count by @LHT129 in #717
• fix double assignment for max_degree in HNSW Merge by @inabao in #724
• resolve not properly handling invalid IDs in CalDistanceById by @inabao in #720
• resolving core dumps caused by incorrect resizing in sparse datacell by @inabao in #719
• fix non-memory io's incorrect result on large dataset by @LHT129 in #742
• fix segment fault and recall degradation issues in compressed HGraph by @nedchu in #784
• add spdlog to the deps of mockimpl by @wxyucs in #775
• use specified commit of thread_pool instead of master branch by @wxyucs in #792
• fix fht ci issue by @ShawnShawnYou in #823
• fix eval build failed but not report by @ShawnShawnYou in #799
• add spdlog into the dependencies of storage in cmake by @wxyucs in #844

New Features

• add elias-fano encoder by @nedchu in #536
• add clone index implement by @LHT129 in #585
• add sparse quantization (no simd optimization) by @inabao in #517
• add export model & hgraph implement by @LHT129 in #599
• add pq parameters by @LHT129 in #609
• add train interface for index by @LHT129 in #608
• Support Index Clone by @LHT129 in #605
• Support Export Model by @LHT129 in #633
• introduce ProductQuantizer by @LHT129 in #626
• Support Index Train by @LHT129 in #634
• support rabitq search with sq4-query by @ShawnShawnYou in #580
• implement pq for hgraph by @LHT129 in #646
• add pqfs simd operators by @LHT129 in #658
• support sparse vector in HGraph by @inabao in #604
• add pq implement for ivf by @LHT129 in #662
• Support basic optimizer by @ShawnShawnYou in #523
• introduce pqfastscan quantizer by @LHT129 in #691
• support merge on ivf index by @LHT129 in #694
• support building graph by Odescent in HGraph by @inabao in #630
• support ivfpq fastscan by @LHT129 in #702
• add computable operators for bitset by @LHT129 in #721
• add new interface for attr search by @LHT129 in #729
• add merge support for ivfpqfs by @LHT129 in #745
• add compressed graph support in HGraph by @nedchu in #747
• Search to add a separate allocator by @Carrot-77 in #716
• support mark-delete in Hgraph by @inabao in #783
• Support gno-imi partition strategy by @dasurax in #612
• add attribute inverted interface by @LHT129 in #759
• Add gramma expression define and expression visitor by @yulijunzj in #773
• introduce fht into vsag and add test for rotator and rq_quantizer_test by @jingyueob in #800
• support merge in HGraph by @inabao in #806
• add executor for ivf attr filter by @LHT129 in #801
• refactor vector transform to apply on other index by @LHT129 in #812
• support index serialization self-description by @wxyucs in #743

Improvements

• [skip ci]update README.md by @wxyucs in #546
• speed up hgraph concurrent add by @LHT129 in #566
• optimize some simd operators by @LHT129 in #560
• add ci test for release branch by @LHT129 in #575
• opt some prefetch by @LHT129 in #556
• some typo fix by @LHT129 in #568
• store common attribute on inner index interface by @LHT129 in #533
• reduce memory allocation when accessing neighbors in hnsw by @inabao in #578
• remove max_level_ in HGraph by @inabao in #567
• remove the dependency of bit by @inabao in #581
• speed up concurrent add on hgraph by @LHT129 in #583
• hgraph support extra info filter by @skylhd in #572
• add new version compatibility (0.14.x) by @LHT129 in #590
• speed up build asan/tsan by @LHT129 in #586
• remove some duplicate test by @LHT129 in #571
• fix get dist fail after delete by @ShawnShawnYou in #598
• replace io in DiskANN with asynchronous io by @inabao in #513
• add reorder for ivf by @LHT129 in #531
• [skip ci] add benchmark for cohere and openai dataset by @inabao in #601
• use hgraph route for ivf by @LHT129 in #508
• support vsag libraries distribution automatically on release by @wxyucs in #597
• improve coverage rate by @LHT129 in #614
• implement of IVF's export model by @LHT129 in #618
• enable hgraph to use an externally provided thread pool by @inabao in #625
• add curl to vsag develop dependencies by @wxyucs in #617
• use allocator to allocate memory for mutex by @inabao in #631
• speed up calculate batch distance by ids by @LHT129 in #632
• update ci dockerfile by @wxyucs in #643
• [skip ci]update binary distribution script by @wxyucs in #639
• speed up async io test on hgraph by @LHT129 in #640
• avoid auto cancel on main branch ci test by @LHT129 in #635
• support sending eval_performance results to influxdb by @wxyucs in #613
• remove force in memory for disk io type hgraph by @LHT129 in #652
• enable build_thread_count on hgraph by @LHT129 in #648
• add more examples by @LHT129 in #651
• optimize ivf index by @LHT129 in #636
• add compressed graph datacell with Elias-Fano Encoder by @nedchu in #596
• calc random orthogonal matrix lazily by @shadowao in #656
• support concurrent add on bruteforce by @LHT129 in #674
• speed up build by @LHT129 in #677
• refactor graph parameter by @inabao in #683
• use VsagException instead std errors by @LHT129 in #641
• add batch4 simd for fp32 compute by @LHT129 in #692
• use DistanceHeap replace MaxHeap by @LHT129 in #680
• add default value for optimizer by @ShawnShawnYou in #700
• enables the upper bound of ef_search, max_degree and ef_construction by @inabao in #668
• Implement of batch4 simd compute on HGraph by @LHT129 in #697
• reduce alloc buffer on merge by @LHT129 in #708
• use thread_pool to speedup kmeans by @LHT129 in #707
• update perf-90 by @inabao in #726
• speedup compile on CI test by @LHT129 in #703
• remove mtime change on ci test by @LHT129 in #736
• add simd operator to cal residual by @LHT129 in #733
• add fast index create for inner index by @LHT129 in #737
• add option for hgraph to build by base quantization by @LHT129 in #735
• make the member access of Resource private by @inabao in #728
• speedup kmeans by use hgraph for large k by @LHT129 in #750
• introduce fast bitset for attribute index by @LHT129 in #753
• add logger for kmeans train by @LHT129 in #757
• add bit operator simd implement by @LHT129 in #756
• adapt update with mark delete by @ShawnShawnYou in #762
• Support arthimetic operator for fp32 simd by @LHT129 in #761
• support build ivf index with residual by @inabao in #754
• support estimate memory in hnsw by @inabao in #709
• reduce test log level to avoid too much debug logs by @LHT129 in #771
• speed up fast_bitset by use simd operators by @LHT129 in #764
• compile openblas dynamic arch by @shadowao in #769
• support remove in graph datacell by @inabao in #695
• print memory usage detail in eval_performance for HGraph by @nedchu in #770
• optimize header files including by @wxyucs in #787
• add attribute query interface by @LHT129 in #766
• improve coverage rate by @LHT129 in #788
• enable id map function for ivf merge by @LHT129 in #776
• Try to use ccache on CI by @LHT129 in #793
• add batch4 compute for pq by @LHT129 in #794
• speedup functest on index(ivf, hgraph) by @LHT129 in #797
• add antlr4 into cache by @wxyucs in #798
• add write list filter by @LHT129 in #795
• refactored attribute to remove c++17 dependency in the interface by @wxyucs in #803
• add restrict keyword for some simd operators by @LHT129 in #809
• auto compatibility on CI by @LHT129 in #810
• release disk usage for CI asan build by @LHT129 in #819
• distinguish between memory usage during index build and search by @inabao in #822
• use antlr4's static library instead of dynamic library by @wxyucs in #830
• update script for compatibility by @LHT129 in #826
• rabitq sq4 quer performance optimization by @shadowao in #816
• bitset speedup by @LHT129 in #829
• support for HGraph using a static schema by @inabao in #815

Other Changes

• [main] fix interface KnnSearch with iter not adapted extra info by @lyxiong0 in #574
• [skip ci]update README.md and DEVELOPMENT.md by @wxyucs in #570
• [skip ci]update output path in perf yaml by @wxyucs in #704
• fix several bugs for performance by @ShawnShawnYou in #730
• [skip ci]add a link to RabitQ-Library in README by @wxyucs in #786
• [skip ci] update Hgraph doc by @LHT129 in #791
• Support neon instructions acceleration on ARM by @Danbaiwq in #765
• update test files for fht by @jingyueob in #825
• unify the distance for uniform quantizer by @inabao in #827

New Contributors

• @antfin-oss made their first contribution in #647
• @dasurax made their first contribution in #612
• @yulijunzj made their first contribution in #773
• @Danbaiwq made their first contribution in #765
• @jingyueob made their first contribution in #800

Full Changelog: v0.14.0...v0.15.0