v0.7.0

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• feat: implement from_xml Spark function by @anarefolio in #2098
• perf: implement simplify hook on concat_ws by @davidlghellin in #2157
• chore: migrate Rust edition to 2024 by @lonless9 in #2208
• fix: correct ceil/floor edge-case bugs and ANSI overflow behavior by @davidlghellin in #1729
• deps(cargo): bump the all group with 5 updates by @dependabot[bot] in #2218
• deps(npm): bump the all group with 5 updates by @dependabot[bot] in #2216
• feat: add OpenAPI client code generator by @linhr in #2215
• chore: clippy 1.97.0 warnings by @lonless9 in #2205
• refactor: bump Rust MSRV to 1.96.0 and adopt newer stable language features by @lonless9 in #2224
• feat: migrate catalog OpenAPI clients by @linhr in #2226
• fix: discriminator enum variants with common types in OpenAPI code generator by @linhr in #2227
• fix: support MERGE INTO with path-based target and DataFrame source references by @linhr with @Copilot in #1654
• fix: align to_xml serialization with Spark semantics by @davidlghellin in #2233
• fix: match Spark ordering semantics for agg and window functions by @shehabgamin in #2228
• feat: reject multiple TABLE arguments in UDTFs unless explicitly enabled by @davidlghellin in #2234
• feat: add hms support for spark datasource tables by @santosh-d3vpl3x in #2230
• fix: count_min_sketch param types + optimize sketch aggregates by @davidlghellin in #2190
• test: cover behaviour-governing arguments passed as a column by @davidlghellin in #2241
• fix: parallelize Iceberg data file writes by @lonless9 in #2229
• fix: measure BINARY in bytes for length, octet_length and bit_length by @davidlghellin in #2238
• docs: refresh site favicon by @everettroeth in #2249
• deps(npm): bump the all group with 8 updates by @dependabot[bot] in #2245
• deps(cargo): bump the all group with 8 updates by @dependabot[bot] in #2247
• perf: optimize queries with linear sums by @lonless9 in #2244
• feat: push down projections through ExplicitRepartitionExec by @nis12ram in #2183
• fix: resolve next_day day names per row and stop trimming them by @davidlghellin in #2254
• test: fix Delta location for Windows by @linhr in #2256
• test: skip Spark 3 Writer V2 provider test by @pomykalakyle in #2258
• fix: handle nullable checkpoint fields and JSON replay schemas for Delta Lake by @lonless9 in #2263
• fix: honor Spark's lambda short-circuit in exists and forall under ANSI by @davidlghellin in #2260
• feat: share driver server among sessions by @linhr in #2253
• fix: improve shared driver server by @linhr in #2269
• perf: use datafusion_common's fast HashMap/HashSet on hot paths by @davidlghellin in #2273
• perf: build arrays_zip result with a single take instead of per-row concat by @davidlghellin in #2272
• feat: support the DataFrame inputFiles operation by @fangchenli in #2239
• deps(npm): bump the all group with 11 updates by @dependabot[bot] in #2276
• deps(cargo): bump the all group with 16 updates by @dependabot[bot] in #2278
• feat: support explicit Glue catalog IDs by @pomykalakyle in #2285
• test: do not mention specific Spark version by @linhr in #2286
• fix: improve path glob filtering by @rohankumardubey in #2284
• test: configure stack size for function gold tests by @linhr in #2290
• feat: full java datetime support by @twsl in #2099
• chore: support Spark 4.2 by @linhr in #2289
• fix: delta nested column mapping fix by @zemin-piao in #2266
• chore: bump quinn-proto from 0.11.14 to 0.11.16 in the cargo group across 1 directory by @dependabot[bot] in #2295
• feat: versioned Spark config by @linhr in #2296
• docs: add guide for Spark version update by @linhr in #2302
• chore: update Spark patch versions by @linhr in #2307
• test: migrate function doctests to feature files and consolidate with Scenario Outlines by @davidlghellin in #2299
• feat: support Aliyun OSS object storage by @weixiuli in #2283
• chore: bump datafusion to 54.1.0 by @lonless9 in #2293
• feat: add basic checkpoint support by @lonless9 in #2270
• feat: blocking shuffle via object storage by @linhr in #2232
• fix: honor absolute Iceberg write data paths by @lonless9 in #2274
• feat: heartbeat for command execution by @linhr in #2323
• deps(npm): bump eslint from 10.7.0 to 10.8.0 in the all group by @dependabot[bot] in #2317
• deps(cargo): bump the all group with 2 updates by @dependabot[bot] in #2319
• refactor: move distributed EXPLAIN output to CODEGEN mode by @lonless9 in #2316
• fix: improve Spark datetime compatibility by @lonless9 in #2322
• fix: allow Iceberg REST create-and-write with access session hints by @lonless9 in #2292
• fix: stop driver server properly by @linhr in #2326
• refactor: consolidate schema type merging logic by @rohankumardubey in #2312
• fix: always report job runner history by @linhr in #2327
• refactor: specify components for actor creation by @linhr in #2329
• feat!: remove the extension option for Parquet data source by @linhr in #2330
• fix: preserve DDL parsing errors during schema fallback by @shehabgamin in #2332
• fix: validate CSV expression function options like Spark by @davidlghellin in #2255
• perf: optimize regexp_replace for anchored capture extraction by @lonless9 in #2303
• fix: require cascade for non-empty memory databases by @ShiroKSH in #2336
• feat!: update OpenTelemetry settings by @linhr in #2340
• feat: support a bearer token file for the Iceberg REST catalog by @Robin-Everaars in #2288
• feat: adjust checkpoint and shuffle storage paths by @linhr in #2341
• fix: cast view-type inputs in Arrow-optimized Python UDFs by @shehabgamin in #2334
• fix: harden Delta column mapping for schema evolution by @lonless9 in #2324
• test: use local path for Delta mergeSchema write by @lonless9 in #2344
• docs: update user guides and development notes by @linhr in #2346
• chore: rename default shuffle backend by @linhr in #2347
• chore: prepare v0.7.0 by @linhr in #2342

New Contributors

• @fangchenli made their first contribution in #2239
• @rohankumardubey made their first contribution in #2284
• @weixiuli made their first contribution in #2283
• @ShiroKSH made their first contribution in #2336
• @Robin-Everaars made their first contribution in #2288

Full Changelog: v0.6.6...v0.7.0