v0.6.5

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• fix: rustup profile for Dockerfile by @linhr in #2042
• docs: update contact info by @linhr in #2044
• chore: upgrade Codecov action by @linhr in #2045
• docs: remove AI writing patterns from prose by @pomykalakyle in #1980
• feat: catalog managed Delta Lake table by @lonless9 in #1957
• feat: catalog managed Iceberg table by @lonless9 in #1929
• refactor: move data source options by @linhr in #2047
• deps(npm): bump the all group with 4 updates by @dependabot[bot] in #2048
• refactor: logical planning for data source writer by @linhr in #2050
• chore: upgrade to DataFusion 54.0.0 and Arrow 58.3.0 by @davidlghellin in #1956
• feat: initial support for named window clause in SQL by @ctydw123 in #2039
• feat: add window function by @radmirnovii in #2006
• fix: parity with various funcs by @shehabgamin in #2054
• chore: use #!/usr/bin/env bash shebang in shell scripts by @davidlghellin in #2062
• fix: skip Spark Connect tests for DF ops unavailable in Spark 3.5 by @shehabgamin in #2064
• chore: clean up downcast_ref usage by @linhr in #2065
• fix: minor issues after DataFusion 54 upgrade by @linhr in #2066
• feat: support various sink modes for file writes by @linhr in #2053
• fix: align file/catalog write existence checks with Spark by @shehabgamin in #2068
• feat: implement to_xml Spark function by @anarefolio in #2032
• feat: add window_time function by @radmirnovii in #2063
• refactor: logical planning for DELETE by @linhr in #2069
• fix: align bin with Spark ANSI cast semantics by @davidlghellin in #2019
• refactor: logical planning for MERGE by @linhr in #2070
• deps(cargo): bump the all group with 12 updates by @dependabot[bot] in #2072
• deps(npm): bump the all group with 9 updates by @dependabot[bot] in #2071
• chore: move HMS codegen to a separate crate by @linhr in #2076
• fix: data source temporary views by @shehabgamin in #2059
• refactor: physical planner for row-level operations by @linhr in #2077
• fix: docker moby error by @twsl in #2080
• fix: align Spark parity for to_number format parsing by @davidlghellin in #2074
• perf: single-buffer concat_ws kernel by @davidlghellin in #2040
• fix: support array indexing and bracket paths in get_json_object by @davidlghellin in #2028
• feat: enable ANSI mode for negative by @davidlghellin in #2031
• perf: implement simplify hook on concat by @davidlghellin in #1891
• fix: don't fail a pending task while workers are still launching by @benzsevern in #2081
• fix: handle regexp_extract optional capture groups by @pomykalakyle in #2082
• fix: column default metadata handling for Hive Metastore catalog writes by @shehabgamin in #2058
• feat: introduce catalog execution context by @lonless9 in #2078
• chore: vendor arrow-pyarrow for PyO3 security fix by @linhr in #2088
• deps(npm): bump the all group with 3 updates by @dependabot[bot] in #2091
• fix: type casting issue for map_entries by @linhr in #2095
• feat: support filter array higher-order function with lambda by @davidlghellin in #2057
• chore: improve test selection logic by @linhr in #2102
• chore: move integration tests by @linhr in #2105
• fix: array_position by @shehabgamin in #2104
• chore: reorganize Python unit tests by @linhr in #2108
• fix: format sketch function default arguments by @lonless9 in #2030
• test: consolidate session management logic by @linhr in #2111
• test: clean up BDD step modules by @linhr in #2112
• test: rewrite test classes as module-level tests by @linhr in #2113
• refactor: simplify query planning and codec for lambda functions by @shehabgamin in #2106
• refactor: minor codec cleanup by @linhr in #2115
• fix: align lakehouse CREATE and REPLACE TABLE semantics by @lonless9 in #2097
• chore: use hashmap instead of array traversal for merge plan by @ctydw123 in #2110
• fix: align EXPLAIN EXTENDED, and COST output with Spark by @nis12ram in #2114
• feat: support transform array higher-order function with lambda by @davidlghellin in #2103
• feat: support exists and forall array higher-order function with lambda by @davidlghellin in #2109
• chore: update docs and tests for parity by @linhr in #2124
• fix: support Windows local paths for Iceberg by @lonless9 in #2119
• feat: support Parquet content-defined chunking options by @linhr in #2125
• fix: support timestampadd by @pomykalakyle in #2086
• feat: lambda aggregate by @shehabgamin in #2123
• chore: upgrade action versions by @linhr in #2126
• feat: support array_sort array higher-order function with comparator lambda by @davidlghellin in #2121
• fix: validate aggregate/reduce lambda arity by @davidlghellin in #2127
• fix: case insensitive function match by @shehabgamin in #2131
• refactor: use datafusion-spark UDF for xxhash64 by @davidlghellin in #2130
• test: skip Spark 3 DataFrame doctests unsupported by Spark Connect by @shehabgamin in #2133
• deps(cargo): bump the all group across 1 directory with 6 updates by @dependabot[bot] in #2132
• refactor: reorganize Delta Lake module layout by @lonless9 in #2122
• feat: add pivot (rewrite to aggregate with per-value FILTER) by @davidlghellin in #2120
• fix: make TableUpdate/ViewUpdate discriminated enums in Iceberg REST catalog by @alexy in #2134
• refactor: pivot by @shehabgamin in #2138
• feat: improve catalog providers by @linhr in #2141
• feat: misc parity by @shehabgamin in #2142
• fix: delta lake legacy protocol check blocking writes to delta-spark-created column-mapping tables by @zemin-piao in #2148
• chore: update OpenTelemetry dependencies by @linhr in #2149
• feat: improve schema_of_json function by @davidlghellin in #1225
• feat: unify to_timestamp/try_to_timestamp, honor ANSI, drop legacy UDF by @davidlghellin in #2100
• fix: restore SPARK_REMOTE="local" JVM validation under PySpark 4.1 by @davidlghellin in #2153
• feat: add try_to_time, unify to_time under SparkTime by @davidlghellin in #2094
• feat: enable ANSI mode for percentile_disc by @davidlghellin in #1962
• chore: prepare v0.6.5 by @linhr in #2156

New Contributors

• @ctydw123 made their first contribution in #2039
• @radmirnovii made their first contribution in #2006
• @benzsevern made their first contribution in #2081

Full Changelog: v0.6.4...v0.6.5