v0.15.0: Experimental FTS, packed struct encoding

What's Changed

Breaking Changes 🛠

• feat: upgrade DataFusion, Arrow, PyO3, ObjectStore by @wjones127 in #2594

New Features 🎉

New feature breakdown

• feat: add fast search to to_table method by @chebbyChefNEQ in #2591
• feat: loosen pyarrow pin by @wjones127 in #2597
• feat: add FSST string compression by @broccoliSpicy in #2470
• feat: integrate inverted index into lance index APIs by @BubbleCal in #2577
  • You can now create full text search indices and pass text search queries into the full_text_query argument
• feat: add projection to take and take_rows by @westonpace in #2623
• feat: add storage_options to _BaseLanceDatasink, LanceDatasink, LanceCommitter by @BitPhinix in #2619
• feat: add bitpack encoding for LanceV2 by @albertlockett in #2333
• feat: add a packed struct encoding to lance by @raunaks13 in #2593
• feat: support deletion with stable row id and scalar indices by @wjones127 in #2609

Bug Fixes 🐛

• fix: avoid making empty range requests as these will err on S3 by @westonpace in #2592
• fix: upgrade AWS SDK by @wjones127 in #2616
• fix: specify behavior version when creating dynamodb config by @westonpace in #2624
• fix: pin numpy version to avoid numpy >= 2.0 until pyarrow supports it by @westonpace in #2637

Performance Improvements 🚀

• perf: improve inverted index performance by @BubbleCal in #2574
• perf: improve v2 scan performance by @westonpace in #2604
• perf: allow fragment scan for nearest query if there is a prefilter by @jiachengdb in #2631
• perf: speed up fsst decompression by @broccoliSpicy in #2626
• perf: skip documents with WAND by @BubbleCal in #2632

Other Changes

• refactor: move planner.rs, logical_expr.rs, and sql.rs from lance to lance_datafusion by @westonpace in #2622
• refactor: remove custom ProjectionExec by @wjones127 in #2608

New Contributors

• @BitPhinix made their first contribution in #2619

Full Changelog: v0.14.1...v0.15.0