v0.6.4

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• fix: stale expected_partitions in PythonDataSourceWriteCommitExec causes intermittent write failure by @nis12ram in #1934
• feat: follow Spark column metadata convention in Arrow by @codex in #1938
• refactor: geo and variant Arrow extension types by @codex in #1958
• perf: cache compiled regex patterns in spark datetime format conversion by @everettroeth in #1925
• fix: array intersect function by @shehabgamin in #1961
• fix: spark randn and randstr parity by @shehabgamin in #1968
• fix: classify timestamp parse failures as datetime exceptions by @Copilot in #1970
• fix: lowercase startswith/endswith display name from PySpark Column API by @pomykalakyle in #1977
• fix: parse Spark variant paths with Chumsky by @CoderHariswar in #1964
• fix: tolerate extra deletion vector fields in delta checkpoints & accept json style manifest by @lonless9 in #1966
• fix: render Arrow Time as Spark time(N) in data_type_to_simple_string by @pomykalakyle in #1978
• fix: render full 2-arg display name for single-arg assert_true by @pomykalakyle in #1979
• deps(cargo): bump the all group with 8 updates by @dependabot[bot] in #1985
• deps(npm): bump the all group with 6 updates by @dependabot[bot] in #1984
• feat: register timestampdiff Spark function by @pomykalakyle in #1982
• feat: align coalesce with pyspark by @twsl in #1856
• feat: implement to_csv Spark function by @anarefolio in #1919
• feat: support GroupedMap/CoGroupedMap iterator UDFs for Spark 4.1.1 by @Copilot in #1763
• fix: preserve delta schema order during scans by @lonless9 in #1992
• feat: safe guards for join reorder optimizer by @lonless9 in #1954
• refactor: generate Hive Metastore thrift client at build time by @codex in #1990
• refactor: introduce listing table source by @codex in #1877
• refactor: move listing schema inference into ReadFormat by @codex in #2003
• fix: improve physical execution for coalesce by @twsl in #1999
• refactor: simplify schema and compression inference for listing files by @codex in #2009
• feat: improve to_csv function by @lonless9 in #2012
• feat: support hll & theta sketch functions by @lonless9 in #1971
• feat: support Variant Shredding for Delta Lake by @lonless9 in #2015
• feat: variant shredding write support for Delta Lake by @lonless9 in #2020
• feat: support variant shredding for Iceberg by @lonless9 in #2021
• deps(npm): bump the all group with 8 updates by @dependabot[bot] in #2022
• deps(cargo): bump the all group with 7 updates by @dependabot[bot] in #2023
• fix: render implicit idx=1 in regexp_extract_all display name by @pomykalakyle in #1983
• fix: support nested struct field reference for arrays by @linhr in #2029
• feat: enable ANSI mode for pmod by @davidlghellin in #2027
• fix: ignore format for timestamp data type in to_date function by @nis12ram in #2025
• fix: improve complex type handling in Arrow/Pandas conversion by @lonless9 in #1950
• feat: support grouping_id aggregate function by @lonless9 in #2017
• feat: support identity columns for Delta Lake by @lonless9 in #2007
• feat: support check constraints for Delta Lake by @lonless9 in #1993
• feat: add try_parse_json function by @davidlghellin in #1636
• feat: support default column values for Delta Lake by @lonless9 in #2004
• fix: normalize negative zero in parse_json variant by @davidlghellin in #2033
• test: round percentile output and tag plan-snapshot scenarios sail-only by @davidlghellin in #2034
• fix: repartition race conditions by @linhr in #2037
• chore: prepare v0.6.4 by @linhr in #2038

New Contributors

• @nis12ram made their first contribution in #1934

Full Changelog: v0.6.3...v0.6.4