v0.6.4

[linhr]

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• fix: stale expected_partitions in PythonDataSourceWriteCommitExec causes intermittent write failure by @nis12ram in fix: stale expected_partitions in PythonDataSourceWriteCommitExec causes intermittent write failure #1934
• feat: follow Spark column metadata convention in Arrow by @codex in feat: follow Spark column metadata convention in Arrow #1938
• refactor: geo and variant Arrow extension types by @codex in refactor: geo and variant Arrow extension types #1958
• perf: cache compiled regex patterns in spark datetime format conversion by @everettroeth in perf: cache compiled regex patterns in spark datetime format conversion #1925
• fix: array intersect function by @shehabgamin in fix: array intersect function #1961
• fix: spark randn and randstr parity by @shehabgamin in fix: spark randn and randstr parity #1968
• fix: classify timestamp parse failures as datetime exceptions by @Copilot in fix: classify timestamp parse failures as datetime exceptions #1970
• fix: lowercase startswith/endswith display name from PySpark Column API by @pomykalakyle in fix: lowercase startswith/endswith display name from PySpark Column API #1977
• fix: parse Spark variant paths with Chumsky by @CoderHariswar in fix: parse Spark variant paths with Chumsky #1964
• fix: tolerate extra deletion vector fields in delta checkpoints & accept json style manifest by @lonless9 in fix: tolerate extra deletion vector fields in delta checkpoints & accept json style manifest #1966
• fix: render Arrow Time as Spark time(N) in data_type_to_simple_string by @pomykalakyle in fix: render Arrow Time as Spark time(N) in data_type_to_simple_string #1978
• fix: render full 2-arg display name for single-arg assert_true by @pomykalakyle in fix: render full 2-arg display name for single-arg assert_true #1979
• deps(cargo): bump the all group with 8 updates by @dependabot[bot] in deps(cargo): bump the all group with 8 updates #1985
• deps(npm): bump the all group with 6 updates by @dependabot[bot] in deps(npm): bump the all group with 6 updates #1984
• feat: register timestampdiff Spark function by @pomykalakyle in feat: register timestampdiff Spark function #1982
• feat: align coalesce with pyspark by @twsl in feat: align coalesce with pyspark #1856
• feat: implement to_csv Spark function by @anarefolio in feat: implement to_csv Spark function #1919
• feat: support GroupedMap/CoGroupedMap iterator UDFs for Spark 4.1.1 by @Copilot in feat: support GroupedMap/CoGroupedMap iterator UDFs for Spark 4.1.1 #1763
• fix: preserve delta schema order during scans by @lonless9 in fix: preserve delta schema order during scans #1992
• feat: safe guards for join reorder optimizer by @lonless9 in feat: safe guards for join reorder optimizer #1954
• refactor: generate Hive Metastore thrift client at build time by @codex in refactor: generate Hive Metastore thrift client at build time #1990
• refactor: introduce listing table source by @codex in refactor: introduce listing table source #1877
• refactor: move listing schema inference into ReadFormat by @codex in refactor: move listing schema inference into ReadFormat #2003
• fix: improve physical execution for coalesce by @twsl in fix: improve physical execution for coalesce #1999
• refactor: simplify schema and compression inference for listing files by @codex in refactor: simplify schema and compression inference for listing files #2009
• feat: improve to_csv function by @lonless9 in feat: improve to_csv function #2012
• feat: support hll & theta sketch functions by @lonless9 in feat: support hll & theta sketch functions #1971
• feat: support Variant Shredding for Delta Lake by @lonless9 in feat: support Variant Shredding for Delta Lake #2015
• feat: variant shredding write support for Delta Lake by @lonless9 in feat: variant shredding write support for Delta Lake #2020
• feat: support variant shredding for Iceberg by @lonless9 in feat: support variant shredding for Iceberg #2021
• deps(npm): bump the all group with 8 updates by @dependabot[bot] in deps(npm): bump the all group with 8 updates #2022
• deps(cargo): bump the all group with 7 updates by @dependabot[bot] in deps(cargo): bump the all group with 7 updates #2023
• fix: render implicit idx=1 in regexp_extract_all display name by @pomykalakyle in fix: render implicit idx=1 in regexp_extract_all display name #1983
• fix: support nested struct field reference for arrays by @linhr in fix: support nested struct field reference for arrays #2029
• feat: enable ANSI mode for pmod by @davidlghellin in feat: enable ANSI mode for pmod #2027
• fix: ignore format for timestamp data type in to_date function by @nis12ram in fix: ignore format for timestamp data type in to_date function #2025
• fix: improve complex type handling in Arrow/Pandas conversion by @lonless9 in fix: improve complex type handling in Arrow/Pandas conversion #1950
• feat: support grouping_id aggregate function by @lonless9 in feat: support grouping_id aggregate function #2017
• feat: support identity columns for Delta Lake by @lonless9 in feat: support identity columns for Delta Lake #2007
• feat: support check constraints for Delta Lake by @lonless9 in feat: support check constraints for Delta Lake #1993
• feat: add try_parse_json function by @davidlghellin in feat: add try_parse_json function #1636
• feat: support default column values for Delta Lake by @lonless9 in feat: support default column values for Delta Lake #2004
• fix: normalize negative zero in parse_json variant by @davidlghellin in fix: normalize negative zero in parse_json variant #2033
• test: round percentile output and tag plan-snapshot scenarios sail-only by @davidlghellin in test: round percentile output and tag plan-snapshot scenarios sail-only #2034
• fix: repartition race conditions by @linhr in fix: repartition race conditions #2037
• chore: prepare v0.6.4 by @linhr in chore: prepare v0.6.4 #2038

New Contributors

• @nis12ram made their first contribution in fix: stale expected_partitions in PythonDataSourceWriteCommitExec causes intermittent write failure #1934

Full Changelog: v0.6.3...v0.6.4

---

This discussion was created from the release v0.6.4.