v0.4.2 Polars, GCS, and distributed lances
A warm welcome to @hzhang86 as Lance's newest contributor. Thanks for adding TPCH benchmarks for Lance to establish a baseline. This is really helpful for us to focus performance optimization roadmap.
This release is packed with valuable features:
- Direct polars scan without needing to pull everything into memory is added.
- We expose FileFragment's to allow distributed processing engines like Spark to access parts of a Lance dataset easily
- Last but not least, we've added support for reading Lance data directly from GS buckets
What's Changed
- [Rust] FileReader read range API by @eddyxu in #752
- Support direct polars scan by @changhiskhan in #755
- [Rust] Persist graph using lance file format. by @eddyxu in #756
- Refactor PQ and OPQ training function to make it usable widely by @eddyxu in #758
- Matrix::centroids method by @eddyxu in #759
- [Python] Set minimal version of Polars for python tests by @eddyxu in #765
- [Rust] Refactor RecordBatchStream trait by @eddyxu in #766
- [Rust] Expose DataFragment as pubilc dataset api. by @eddyxu in #769
- Revert "[Python] Set minimal version of Polars for python tests (#765)" by @gsilvestrin in #770
- add python script to compare lance performance vs parquet TPCH by @hzhang86 in #749
- Expose index metadata by @changhiskhan in #768
- Google Cloud Storage support. by @gsilvestrin in #773
- [Python] Expose DataFragment via dataset by @eddyxu in #774
- Get S3 credentials from_env by @changhiskhan in #775
- Fix duckdb build by @eddyxu in #776
- [Rust] A arrow kernel to compute hash value of the array. by @eddyxu in #777
New Contributors
Full Changelog: v0.4.1...v0.4.2