v0.3.2 Speed up index creation by more than 60x
We discovered two thing making index creation unnecessarily long:
- Instead of using KMeans++ to initialize, just do random initialization
- Turn of BLAS on macOS because it turned out to be super slow
On macbook air, index creation goes from 25min on sift1m to 24s.
On Ubuntu, it's roughly a 6x speedup.
What's Changed
- Remove duckdb extension based on c++ codebase. by @eddyxu in #530
- Reader honor batch size by @eddyxu in #532
- Rust based duckdb extension by @eddyxu in #529
- pretty-print: add pretty print by @AsadullahFarooqi in #469
- Add unit tests to verify the read of struct by @eddyxu in #537
- [DuckDB] Support Struct, and a separate crate for re-usable components to build duckdb extensions by @eddyxu in #534
- [Rust] Fix main branch CI failure and bump arrow version by @eddyxu in #538
- metrics for vector index by @changhiskhan in #535
- [DuckDB] Add list support. by @eddyxu in #540
- Update README for vector index copy by @jaichopra in #542
- Support writing pandas dataframes directly by @changhiskhan in #543
- Improve IVF_PQ index creation performance by @eddyxu in #545
- update metrics by @changhiskhan in #547
- Fix duckdb CI failure by @eddyxu in #546
- Remove Mac accelerate code by @eddyxu in #548
- update docs by @changhiskhan in #549
- Set intel mac release version to 10.15 by @eddyxu in #551
- Test cli in Github Action by @eddyxu in #550
- Changhiskhan/nb docs by @changhiskhan in #552
- minor fix for GHA and github by @changhiskhan in #553
New Contributors
- @AsadullahFarooqi made their first contribution in #469
Full Changelog: v0.3.1...v0.3.2