v0.6.3

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• perf: add SparkYear UDF with preimage for date-bucket filter pushdown by @davidlghellin in #1845
• fix: truncate large gRPC error messages to prevent metadata size overflow by @pomykalakyle in #1859
• perf: implement preimage hook on floor for Parquet filter pushdown by @davidlghellin in #1843
• perf: implement simplify hook on ceil and floor for integer identity and idempotence by @davidlghellin in #1844
• docs: add dbt integration guide by @pomykalakyle in #1863
• feat: support Variant type for Delta Lake and Iceberg & Iceberg V3 specs by @lonless9 in #1836
• fix: assign unique iceberg field ids to nested columns by @Copilot in #1871
• fix: use correct timestamp datatype for to_timestamp functions by @Copilot in #1873
• refactor: consolidate options resolution and move listing table modules by @codex in #1869
• refactor: split listing format read and write traits by @codex in #1874
• fix: resolve arrays_zip parity with Spark behavior by @Copilot in #1866
• fix: handle from_json/from_csv display names and schema parsing by @Copilot in #1830
• fix: correct Delta Lake partition handling with schema evolution and metadata updates by @lonless9 in #1878
• feat: improve row-level metrics for Delta Lake by @lonless9 in #1870
• test: add snapshots for ClickBench query plans by @linhr in #1885
• feat: support type widening for Delta Lake by @lonless9 in #1854
• fix: implement try_encode_udwf / try_decode_udwf for SparkNtile in codec by @davidlghellin in #1884
• deps(cargo): bump the all group with 11 updates by @dependabot[bot] in #1889
• deps(npm): bump the all group with 7 updates by @dependabot[bot] in #1888
• fix: improve Spark string function parity by @Copilot in #1881
• fix: type casting for ClickBench data by @linhr in #1894
• test: fix catalog pytest by @lonless9 in #1812
• feat: add HMS table metadata parity by @santosh-d3vpl3x in #1865
• fix: sort transform parsing for Iceberg REST catalog by @Copilot in #1867
• feat: implement a provider-agnostic caching layer for catalog listings by @aminebizid in #1886
• fix: allow generated columns for datasource target & introduce snapshot/commit context by @lonless9 in #1855
• feat: implement ALTER TABLE SET/UNSET TBLPROPERTIES for Iceberg by @lonless9 in #1822
• feat: add variant_explode and variant_explode_outer functions by @twsl in #1829
• fix: concat() with empty array and timestamp args by @davidlghellin in #1802
• chore: update pyarrow version constraints in pyproject.toml by @Copilot in #1902
• feat: implement coalesce function with Spark-compatible coercion for mixed string and temporal arguments by @Copilot in #1898
• fix: support namespace aliases for database SQL by @Copilot in #1904
• feat: enhance coalesce function to support ANSI mode for string and date types by @lonless9 in #1906
• test: fix issue with query result display by @linhr in #1907
• refactor: unify try_to_number into to_number with safe mode flag by @davidlghellin in #1879
• chore: remove prost recursion limit by @linhr in #1835
• docs: remove partial notation from CREATE TABLE feature status by @Copilot in #1912
• feat: align abs UDF with Spark by @davidlghellin in #1769
• feat: implement json_tuple Spark function by @anarefolio in #1897
• fix: interpret input in local time zone for to_timestamp_ntz by @linhr in #1915
• feat: support noop write format by @CoderHariswar in #1918
• fix: Improve missing JDBC data source error by @alexy in #1928
• fix: fixing create() table for Nessie, adding test for it by @alexy in #1924
• feat: remove Kubernetes worker pods on session end by @linhr in #1933
• deps(cargo): bump the all group with 2 updates by @dependabot[bot] in #1931
• docs: update supported features for Delta and Iceberg by @lonless9 in #1926
• feat: from_json builder approach by @kentkr in #1913
• feat: time zone conversion by @linhr in #1935
• deps(npm): bump the all group with 8 updates by @dependabot[bot] in #1930
• fix: preserve managed catalog table type by @Copilot in #1905
• feat: improve Iceberg REST catalog support by @shehabgamin in #1920
• fix: correct monotonic_id partitioning by @lonless9 in #1936
• fix: withMetadata should override metadata instead of merging by @Copilot in #1816
• fix: minor catalog provider changes by @linhr in #1939
• fix: cast output record batch to match schema in map partitions UDF by @shehabgamin in #1949
• chore: improve test setup in multi-platform build by @linhr in #1951
• chore: prepare v0.6.3 by @linhr in #1940

New Contributors

• @codex made their first contribution in #1869
• @aminebizid made their first contribution in #1886
• @anarefolio made their first contribution in #1897
• @CoderHariswar made their first contribution in #1918
• @alexy made their first contribution in #1928

Full Changelog: v0.6.2...v0.6.3