v0.6.5

[linhr]

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• fix: rustup profile for Dockerfile by @linhr in fix: rustup profile for Dockerfile #2042
• docs: update contact info by @linhr in docs: update contact info #2044
• chore: upgrade Codecov action by @linhr in chore: upgrade Codecov action #2045
• docs: remove AI writing patterns from prose by @pomykalakyle in docs: remove AI writing patterns from prose #1980
• feat: catalog managed Delta Lake table by @lonless9 in feat: catalog managed Delta Lake table #1957
• feat: catalog managed Iceberg table by @lonless9 in feat: catalog managed Iceberg table #1929
• refactor: move data source options by @linhr in refactor: move data source options #2047
• deps(npm): bump the all group with 4 updates by @dependabot[bot] in deps(npm): bump the all group with 4 updates #2048
• refactor: logical planning for data source writer by @linhr in refactor: logical planning for data source writer #2050
• chore: upgrade to DataFusion 54.0.0 and Arrow 58.3.0 by @davidlghellin in chore: upgrade to DataFusion 54.0.0 and Arrow 58.3.0 #1956
• feat: initial support for named window clause in SQL by @ctydw123 in feat: initial support for named window clause in SQL #2039
• feat: add window function by @radmirnovii in feat: add window function #2006
• fix: parity with various funcs by @shehabgamin in fix: parity with various funcs #2054
• chore: use #!/usr/bin/env bash shebang in shell scripts by @davidlghellin in chore: use #!/usr/bin/env bash shebang in shell scripts #2062
• fix: skip Spark Connect tests for DF ops unavailable in Spark 3.5 by @shehabgamin in fix: skip Spark Connect tests for DF ops unavailable in Spark 3.5 #2064
• chore: clean up downcast_ref usage by @linhr in chore: clean up downcast_ref usage #2065
• fix: minor issues after DataFusion 54 upgrade by @linhr in fix: minor issues after DataFusion 54 upgrade #2066
• feat: support various sink modes for file writes by @linhr in feat: support various sink modes for file writes #2053
• fix: align file/catalog write existence checks with Spark by @shehabgamin in fix: align file/catalog write existence checks with Spark #2068
• feat: implement to_xml Spark function by @anarefolio in feat: implement to_xml Spark function #2032
• feat: add window_time function by @radmirnovii in feat: add window_time function #2063
• refactor: logical planning for DELETE by @linhr in refactor: logical planning for DELETE #2069
• fix: align bin with Spark ANSI cast semantics by @davidlghellin in fix: align bin with Spark ANSI cast semantics #2019
• refactor: logical planning for MERGE by @linhr in refactor: logical planning for MERGE #2070
• deps(cargo): bump the all group with 12 updates by @dependabot[bot] in deps(cargo): bump the all group with 12 updates #2072
• deps(npm): bump the all group with 9 updates by @dependabot[bot] in deps(npm): bump the all group with 9 updates #2071
• chore: move HMS codegen to a separate crate by @linhr in chore: move HMS codegen to a separate crate #2076
• fix: data source temporary views by @shehabgamin in fix: data source temporary views #2059
• refactor: physical planner for row-level operations by @linhr in refactor: physical planner for row-level operations #2077
• fix: docker moby error by @twsl in fix: docker moby error #2080
• fix: align Spark parity for to_number format parsing by @davidlghellin in fix: align Spark parity for to_number format parsing #2074
• perf: single-buffer concat_ws kernel by @davidlghellin in perf: single-buffer concat_ws kernel #2040
• fix: support array indexing and bracket paths in get_json_object by @davidlghellin in fix: support array indexing and bracket paths in get_json_object #2028
• feat: enable ANSI mode for negative by @davidlghellin in feat: enable ANSI mode for negative #2031
• perf: implement simplify hook on concat by @davidlghellin in perf: implement simplify hook on concat #1891
• fix: don't fail a pending task while workers are still launching by @benzsevern in fix: don't fail a pending task while workers are still launching #2081
• fix: handle regexp_extract optional capture groups by @pomykalakyle in fix: handle regexp_extract optional capture groups #2082
• fix: column default metadata handling for Hive Metastore catalog writes by @shehabgamin in fix: column default metadata handling for Hive Metastore catalog writes #2058
• feat: introduce catalog execution context by @lonless9 in feat: introduce catalog execution context #2078
• chore: vendor arrow-pyarrow for PyO3 security fix by @linhr in chore: vendor arrow-pyarrow for PyO3 security fix #2088
• deps(npm): bump the all group with 3 updates by @dependabot[bot] in deps(npm): bump the all group with 3 updates #2091
• fix: type casting issue for map_entries by @linhr in fix: type casting issue for map_entries #2095
• feat: support filter array higher-order function with lambda by @davidlghellin in feat: support filter array higher-order function with lambda #2057
• chore: improve test selection logic by @linhr in chore: improve test selection logic #2102
• chore: move integration tests by @linhr in chore: move integration tests #2105
• fix: array_position by @shehabgamin in fix: array_position #2104
• chore: reorganize Python unit tests by @linhr in chore: reorganize Python unit tests #2108
• fix: format sketch function default arguments by @lonless9 in fix: format sketch function default arguments #2030
• test: consolidate session management logic by @linhr in test: consolidate session management logic #2111
• test: clean up BDD step modules by @linhr in test: clean up BDD step modules #2112
• test: rewrite test classes as module-level tests by @linhr in test: rewrite test classes as module-level tests #2113
• refactor: simplify query planning and codec for lambda functions by @shehabgamin in refactor: simplify query planning and codec for lambda functions #2106
• refactor: minor codec cleanup by @linhr in refactor: minor codec cleanup #2115
• fix: align lakehouse CREATE and REPLACE TABLE semantics by @lonless9 in fix: align lakehouse CREATE and REPLACE TABLE semantics #2097
• chore: use hashmap instead of array traversal for merge plan by @ctydw123 in chore: use hashmap instead of array traversal for merge plan #2110
• fix: align EXPLAIN EXTENDED, and COST output with Spark by @nis12ram in fix: align EXPLAIN EXTENDED, and COST output with Spark #2114
• feat: support transform array higher-order function with lambda by @davidlghellin in feat: support transform array higher-order function with lambda #2103
• feat: support exists and forall array higher-order function with lambda by @davidlghellin in feat: support exists and forall array higher-order function with lambda #2109
• chore: update docs and tests for parity by @linhr in chore: update docs and tests for parity #2124
• fix: support Windows local paths for Iceberg by @lonless9 in fix: support Windows local paths for Iceberg #2119
• feat: support Parquet content-defined chunking options by @linhr in feat: support Parquet content-defined chunking options #2125
• fix: support timestampadd by @pomykalakyle in fix: support timestampadd #2086
• feat: lambda aggregate by @shehabgamin in feat: lambda aggregate #2123
• chore: upgrade action versions by @linhr in chore: upgrade action versions #2126
• feat: support array_sort array higher-order function with comparator lambda by @davidlghellin in feat: support array_sort array higher-order function with comparator lambda #2121
• fix: validate aggregate/reduce lambda arity by @davidlghellin in fix: validate aggregate/reduce lambda arity #2127
• fix: case insensitive function match by @shehabgamin in fix: case insensitive function match #2131
• refactor: use datafusion-spark UDF for xxhash64 by @davidlghellin in refactor: use datafusion-spark UDF for xxhash64 #2130
• test: skip Spark 3 DataFrame doctests unsupported by Spark Connect by @shehabgamin in test: skip Spark 3 DataFrame doctests unsupported by Spark Connect #2133
• deps(cargo): bump the all group across 1 directory with 6 updates by @dependabot[bot] in deps(cargo): bump the all group across 1 directory with 6 updates #2132
• refactor: reorganize Delta Lake module layout by @lonless9 in refactor: reorganize Delta Lake module layout #2122
• feat: add pivot (rewrite to aggregate with per-value FILTER) by @davidlghellin in feat: add pivot (rewrite to aggregate with per-value FILTER) #2120
• fix: make TableUpdate/ViewUpdate discriminated enums in Iceberg REST catalog by @alexy in fix: make TableUpdate/ViewUpdate discriminated enums in Iceberg REST catalog #2134
• refactor: pivot by @shehabgamin in refactor: pivot #2138
• feat: improve catalog providers by @linhr in feat: improve catalog providers #2141
• feat: misc parity by @shehabgamin in feat: misc parity #2142
• fix: delta lake legacy protocol check blocking writes to delta-spark-created column-mapping tables by @zemin-piao in fix: delta lake legacy protocol check blocking writes to delta-spark-created column-mapping tables #2148
• chore: update OpenTelemetry dependencies by @linhr in chore: update OpenTelemetry dependencies #2149
• feat: improve schema_of_json function by @davidlghellin in feat: improve schema_of_json function #1225
• feat: unify to_timestamp/try_to_timestamp, honor ANSI, drop legacy UDF by @davidlghellin in feat: unify to_timestamp/try_to_timestamp, honor ANSI, drop legacy UDF #2100
• fix: restore SPARK_REMOTE="local" JVM validation under PySpark 4.1 by @davidlghellin in fix: restore SPARK_REMOTE="local" JVM validation under PySpark 4.1 #2153
• feat: add try_to_time, unify to_time under SparkTime by @davidlghellin in feat: add try_to_time, unify to_time under SparkTime #2094
• feat: enable ANSI mode for percentile_disc by @davidlghellin in feat: enable ANSI mode for percentile_disc #1962
• chore: prepare v0.6.5 by @linhr in chore: prepare v0.6.5 #2156

New Contributors

• @ctydw123 made their first contribution in feat: initial support for named window clause in SQL #2039
• @radmirnovii made their first contribution in feat: add window function #2006
• @benzsevern made their first contribution in fix: don't fail a pending task while workers are still launching #2081

Full Changelog: v0.6.4...v0.6.5

---

This discussion was created from the release v0.6.5.

[keen85]

Thanks for the new release.

I'm very excited about the new lambda functions!

It seems there was a behavior change in terms of how Sail handles Windows paths:
In pysail==0.6.4, df.write.save(path="C:/foo/ts=2026-01-01 10:00:00") worked fine. Since Windows does not allow colons in filesystem paths, Sail implicitly changed the path to "C:/foo/ts=2026-01-01 10%3A00%3A00", where : is URL-encoded to %3A.

pysail==0.6.5 seems to be more strict here. df.write.save(path="C:/foo/ts=2026-01-01 10:00:00") fails.

Is this change intentional? I'm not sure how OSS Spark behaves in this regard on Windows and whether Sail is now just mimicking this behavior.

[linhr]

Thanks for the report! We didn't make this change intentionally. It could be possible that DataFusion or object_store added some validation for this case.

If I were to design the user experience, I feel rejecting the path makes sense, since this is a file system operation so failing fast on invalid characters is a clearer signal to the user. Implicit URL encoding (the previous behavior) actually sounds like a surprise to me in this case.

But definitely we would like to see how JVM Spark behaves in this case and we'll mirror the behavior whenever possible. I haven't got a chance to validate this, but would you be able to see how it goes in JVM Spark if you have a handy Windows setup?

[keen85]

I also don't have a fully working JVM Spark environment on Windows - setting that up is tedious, which is actually one of the reasons we turned to Sail in the first place 😂

However, my research concluded that JVM Spark does not perform any implicit URL encoding and will fail in this scenario.

Therefore, pysail 0.6.5 is likely more consistent with JVM Spark now.

We will find a way to work around this.