v0.3.16 Filte pushdown improvements

Welcome @wangfenjin to lance contributors. Thanks for submitting a bug fix for the Lance DuckDB extensions 🔥

This release contains 2 workarounds for arrow limitations:

1. Lance datasets now support <field> LIKE '%' and <field> IN (<values>) filters to be passed in as string. Generic SQL syntax supported by datafusion is now accepted. This is a break from standard pyarrow Dataset behavior which only accepts arrow compute Expression, which is not present in rust and also does not support introspection in python for developers to build custom adapter.

2. When concatenating arrow dictionary arrays, the dict values are duplicated. There is currently no concrete plans to change this behavior in Arrow. Instead, we fix that at write time in Lance.

What's Changed

• Changed encoders to handle multiple Arrays by @gsilvestrin in #681
• Train kmeans iteratively by @eddyxu in #688
• Changed writers to handle multiple Arrays by @gsilvestrin in #691
• Streaming PQ by @eddyxu in #689
• [Bug] PQ training generates empty centroids by @eddyxu in #693
• Allow append mode even if dataset doesn't already exist by @ananis25 in #690
• Support "LIKE" and "IN" in filters by @eddyxu in #696
• fix typo by @wangfenjin in #697
• Improve indexing performance by @eddyxu in #699
• Compute PQ distortion. by @eddyxu in #695
• Bugfix for BinaryEncoder positions by @gsilvestrin in #698

New Contributors

• @wangfenjin made their first contribution in #697

Full Changelog: v0.3.15...v0.3.16