v0.7.0

[linhr]

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• feat: implement from_xml Spark function by @anarefolio in feat: implement from_xml Spark function #2098
• perf: implement simplify hook on concat_ws by @davidlghellin in perf: implement simplify hook on concat_ws #2157
• chore: migrate Rust edition to 2024 by @lonless9 in chore: migrate Rust edition to 2024 #2208
• fix: correct ceil/floor edge-case bugs and ANSI overflow behavior by @davidlghellin in fix: correct ceil/floor edge-case bugs and ANSI overflow behavior #1729
• deps(cargo): bump the all group with 5 updates by @dependabot[bot] in deps(cargo): bump the all group with 5 updates #2218
• deps(npm): bump the all group with 5 updates by @dependabot[bot] in deps(npm): bump the all group with 5 updates #2216
• feat: add OpenAPI client code generator by @linhr in feat: add OpenAPI client code generator #2215
• chore: clippy 1.97.0 warnings by @lonless9 in chore: clippy 1.97.0 warnings #2205
• refactor: bump Rust MSRV to 1.96.0 and adopt newer stable language features by @lonless9 in refactor: bump Rust MSRV to 1.96.0 and adopt newer stable language features #2224
• feat: migrate catalog OpenAPI clients by @linhr in feat: migrate catalog OpenAPI clients #2226
• fix: discriminator enum variants with common types in OpenAPI code generator by @linhr in fix: discriminator enum variants with common types in OpenAPI code generator #2227
• fix: support MERGE INTO with path-based target and DataFrame source references by @linhr with @Copilot in fix: support MERGE INTO with path-based target and DataFrame source references #1654
• fix: align to_xml serialization with Spark semantics by @davidlghellin in fix: align to_xml serialization with Spark semantics #2233
• fix: match Spark ordering semantics for agg and window functions by @shehabgamin in fix: match Spark ordering semantics for agg and window functions #2228
• feat: reject multiple TABLE arguments in UDTFs unless explicitly enabled by @davidlghellin in feat: reject multiple TABLE arguments in UDTFs unless explicitly enabled #2234
• feat: add hms support for spark datasource tables by @santosh-d3vpl3x in feat: add hms support for spark datasource tables #2230
• fix: count_min_sketch param types + optimize sketch aggregates by @davidlghellin in fix: count_min_sketch param types + optimize sketch aggregates #2190
• test: cover behaviour-governing arguments passed as a column by @davidlghellin in test: cover behaviour-governing arguments passed as a column #2241
• fix: parallelize Iceberg data file writes by @lonless9 in fix: parallelize Iceberg data file writes #2229
• fix: measure BINARY in bytes for length, octet_length and bit_length by @davidlghellin in fix: measure BINARY in bytes for length, octet_length and bit_length #2238
• docs: refresh site favicon by @everettroeth in docs: refresh site favicon #2249
• deps(npm): bump the all group with 8 updates by @dependabot[bot] in deps(npm): bump the all group with 8 updates #2245
• deps(cargo): bump the all group with 8 updates by @dependabot[bot] in deps(cargo): bump the all group with 8 updates #2247
• perf: optimize queries with linear sums by @lonless9 in perf: optimize queries with linear sums #2244
• feat: push down projections through ExplicitRepartitionExec by @nis12ram in feat: push down projections through ExplicitRepartitionExec #2183
• fix: resolve next_day day names per row and stop trimming them by @davidlghellin in fix: resolve next_day day names per row and stop trimming them #2254
• test: fix Delta location for Windows by @linhr in test: fix Delta location for Windows #2256
• test: skip Spark 3 Writer V2 provider test by @pomykalakyle in test: skip Spark 3 Writer V2 provider test #2258
• fix: handle nullable checkpoint fields and JSON replay schemas for Delta Lake by @lonless9 in fix: handle nullable checkpoint fields and JSON replay schemas for Delta Lake #2263
• fix: honor Spark's lambda short-circuit in exists and forall under ANSI by @davidlghellin in fix: honor Spark's lambda short-circuit in exists and forall under ANSI #2260
• feat: share driver server among sessions by @linhr in feat: share driver server among sessions #2253
• fix: improve shared driver server by @linhr in fix: improve shared driver server #2269
• perf: use datafusion_common's fast HashMap/HashSet on hot paths by @davidlghellin in perf: use datafusion_common's fast HashMap/HashSet on hot paths #2273
• perf: build arrays_zip result with a single take instead of per-row concat by @davidlghellin in perf: build arrays_zip result with a single take instead of per-row concat #2272
• feat: support the DataFrame inputFiles operation by @fangchenli in feat: support the DataFrame inputFiles operation #2239
• deps(npm): bump the all group with 11 updates by @dependabot[bot] in deps(npm): bump the all group with 11 updates #2276
• deps(cargo): bump the all group with 16 updates by @dependabot[bot] in deps(cargo): bump the all group with 16 updates #2278
• feat: support explicit Glue catalog IDs by @pomykalakyle in feat: support explicit Glue catalog IDs #2285
• test: do not mention specific Spark version by @linhr in test: do not mention specific Spark version #2286
• fix: improve path glob filtering by @rohankumardubey in fix: improve path glob filtering #2284
• test: configure stack size for function gold tests by @linhr in test: configure stack size for function gold tests #2290
• feat: full java datetime support by @twsl in feat: full java datetime support #2099
• chore: support Spark 4.2 by @linhr in chore: support Spark 4.2 #2289
• fix: delta nested column mapping fix by @zemin-piao in fix: delta nested column mapping fix #2266
• chore: bump quinn-proto from 0.11.14 to 0.11.16 in the cargo group across 1 directory by @dependabot[bot] in chore: bump quinn-proto from 0.11.14 to 0.11.16 in the cargo group across 1 directory #2295
• feat: versioned Spark config by @linhr in feat: versioned Spark config #2296
• docs: add guide for Spark version update by @linhr in docs: add guide for Spark version update #2302
• chore: update Spark patch versions by @linhr in chore: update Spark patch versions #2307
• test: migrate function doctests to feature files and consolidate with Scenario Outlines by @davidlghellin in test: migrate function doctests to feature files and consolidate with Scenario Outlines #2299
• feat: support Aliyun OSS object storage by @weixiuli in feat: support Aliyun OSS object storage #2283
• chore: bump datafusion to 54.1.0 by @lonless9 in chore: bump datafusion to 54.1.0 #2293
• feat: add basic checkpoint support by @lonless9 in feat: add basic checkpoint support #2270
• feat: blocking shuffle via object storage by @linhr in feat: blocking shuffle via object storage #2232
• fix: honor absolute Iceberg write data paths by @lonless9 in fix: honor absolute Iceberg write data paths #2274
• feat: heartbeat for command execution by @linhr in feat: heartbeat for command execution #2323
• deps(npm): bump eslint from 10.7.0 to 10.8.0 in the all group by @dependabot[bot] in deps(npm): bump eslint from 10.7.0 to 10.8.0 in the all group #2317
• deps(cargo): bump the all group with 2 updates by @dependabot[bot] in deps(cargo): bump the all group with 2 updates #2319
• refactor: move distributed EXPLAIN output to CODEGEN mode by @lonless9 in refactor: move distributed EXPLAIN output to CODEGEN mode #2316
• fix: improve Spark datetime compatibility by @lonless9 in fix: improve Spark datetime compatibility #2322
• fix: allow Iceberg REST create-and-write with access session hints by @lonless9 in fix: allow Iceberg REST create-and-write with access session hints #2292
• fix: stop driver server properly by @linhr in fix: stop driver server properly #2326
• refactor: consolidate schema type merging logic by @rohankumardubey in refactor: consolidate schema type merging logic #2312
• fix: always report job runner history by @linhr in fix: always report job runner history #2327
• refactor: specify components for actor creation by @linhr in refactor: specify components for actor creation #2329
• feat!: remove the extension option for Parquet data source by @linhr in feat!: remove the extension option for Parquet data source #2330
• fix: preserve DDL parsing errors during schema fallback by @shehabgamin in fix: preserve DDL parsing errors during schema fallback #2332
• fix: validate CSV expression function options like Spark by @davidlghellin in fix: validate CSV expression function options like Spark #2255
• perf: optimize regexp_replace for anchored capture extraction by @lonless9 in perf: optimize regexp_replace for anchored capture extraction #2303
• fix: require cascade for non-empty memory databases by @ShiroKSH in fix: require cascade for non-empty memory databases #2336
• feat!: update OpenTelemetry settings by @linhr in feat!: update OpenTelemetry settings #2340
• feat: support a bearer token file for the Iceberg REST catalog by @Robin-Everaars in feat: support a bearer token file for the Iceberg REST catalog #2288
• feat: adjust checkpoint and shuffle storage paths by @linhr in feat: adjust checkpoint and shuffle storage paths #2341
• fix: cast view-type inputs in Arrow-optimized Python UDFs by @shehabgamin in fix: cast view-type inputs in Arrow-optimized Python UDFs #2334
• fix: harden Delta column mapping for schema evolution by @lonless9 in fix: harden Delta column mapping for schema evolution #2324
• test: use local path for Delta mergeSchema write by @lonless9 in test: use local path for Delta mergeSchema write #2344
• docs: update user guides and development notes by @linhr in docs: update user guides and development notes #2346
• chore: rename default shuffle backend by @linhr in chore: rename default shuffle backend #2347
• chore: prepare v0.7.0 by @linhr in chore: prepare v0.7.0 #2342

New Contributors

• @fangchenli made their first contribution in feat: support the DataFrame inputFiles operation #2239
• @rohankumardubey made their first contribution in fix: improve path glob filtering #2284
• @weixiuli made their first contribution in feat: support Aliyun OSS object storage #2283
• @ShiroKSH made their first contribution in fix: require cascade for non-empty memory databases #2336
• @Robin-Everaars made their first contribution in feat: support a bearer token file for the Iceberg REST catalog #2288

Full Changelog: v0.6.6...v0.7.0

---

This discussion was created from the release v0.7.0.

[keen85]

@lonless9, can you provide some information on how to use the new checkpoint mechanism?
From unit tests it seems that env var SAIL_EXECUTION__CHECKPOINT__PATH needs to be set.
I tried that but I still encounter pyspark.errors.exceptions.connect.AnalysisException: checkpoint object-store path is not configured; set execution.checkpoint.path.

Is this one of the settings that needs to be set prior to creating the session?

[lonless9]

I think so, given that this is a server configuration. And both blocking shuffle and checkpoint should worth some documentation. cc @linhr

[linhr]

Yeah all Sail configuration needs to be provided before creating the Spark Connect server.