v0.7.1

Overview

You can find the release summary in the documentation.

The PySail Python package is available on PyPI.

What's Changed

• test: improve function test selection by @linhr in #2352
• fix: retry Iceberg REST bootstrap authentication by @Robin-Everaars in #2348
• fix: validate explicit Parquet read schemas by @rohankumardubey in #2351
• test: initialize the global state when reusing an existing server by @davidlghellin in #2353
• refactor: move server and telemetry code by @linhr in #2355
• feat: support basic merge operation for Iceberg by @lonless9 in #2222
• refactor: improve child actor management by @linhr in #2359
• feat: list Spark 4.2 built-in functions by @lonless9 in #2325
• docs: add LangChain integration guide by @pomykalakyle in #2354
• deps(cargo): bump the all group with 6 updates by @dependabot[bot] in #2366
• deps(npm): bump vue-tsc from 3.3.8 to 3.3.9 in the all group by @dependabot[bot] in #2363
• fix: make Python data source options case-insensitive by @ErikBPF in #2371
• chore: cargo update by @lonless9 in #2365
• fix: resolve Delta MERGE constraints by visible names by @alexy in #2374
• feat: add basic Celeborn lifecycle manager by @linhr in #2373
• fix: harden Iceberg MERGE semantics by @lonless9 in #2362
• feat: implement vector inner product by @jaideeppyne in #2360
• fix: propagate swallowed cascade errors in Iceberg drop_database by @Robin-Everaars in #2297
• fix: idempotent worker shutdown cleanup by @weixiuli in #2392
• feat: add basic Celeborn shuffle client by @linhr in #2378
• refactor: rename actor message and move actor options by @linhr in #2397
• test: align function tests with Spark values by @davidlghellin in #2375
• refactor: generalize task stream management by @linhr in #2398
• refactor: add task runner actor by @linhr in #2399
• fix: improve Delta protocol and checkpoint compatibility by @lonless9 in #2231
• feat: add Celeborn shuffle backend by @linhr in #2403
• fix: refine Celeborn shuffle backend implementation by @linhr in #2414
• feat: improve Celeborn transport and support connection reuse by @linhr in #2415
• fix: support three-argument dateadd by @rohankumardubey in #2429
• deps(npm): bump the all group with 6 updates by @dependabot[bot] in #2425
• deps(cargo): bump the all group with 4 updates by @dependabot[bot] in #2427
• fix: honor ANSI mode for invalid to_date strings by @lonless9 in #2405
• fix: align empty structs and to_json key ordering with Spark by @lonless9 in #2408
• fix: coerce string sum across aggregate and window paths by @lonless9 in #2410
• feat: support Celeborn revive logic by @linhr in #2423
• chore: always show gold data report by @linhr in #2431
• fix: shuffle output mode for scalar subquery by @linhr in #2432
• test: port HMS provider integration tests to Python by @lonless9 in #2416
• fix: align USING and NATURAL join output-key ownership with Spark by @lonless9 in #2411
• fix: lay the correctness groundwork for the DataFusion upgrade by @lonless9 in #2401
• fix: replace malformed UTF-8 in CSV reads by @lonless9 in #2412
• fix: align non-ANSI STRING-to-INT casts with Spark by @lonless9 in #2406
• fix: match Spark string and day-time interval addition semantics by @lonless9 in #2407
• fix: align CASE, IF, and COALESCE string/temporal coercion with Spark by @lonless9 in #2409
• fix: pin Iceberg writer metadata during planning by @lonless9 in #2402
• chore: update TPC-H benchmark results by @linhr in #2439
• chore: update README with the latest benchmark results by @linhr in #2440
• docs: add notes for benchmark by @linhr in #2441
• fix: preserve lambda bindings across projection pushdown by @lonless9 in #2413
• test: port Iceberg REST catalog tests to Python by @lonless9 in #2434
• test: add general proxy for Celeborn tests by @linhr in #2442
• fix: support Celeborn soft split properly by @linhr in #2443
• fix: support Python data source filter pushdown on PySpark 4.0 by @maxvwa in #2437
• test: fix flaky Celeborn partition split test by @lonless9 in #2447
• fix: correct SQL sequence syntax descriptor cardinality by @lonless9 in #2372
• perf: compile column regex patterns once per batch, not per row by @radmirnovii in #2448
• deps(cargo): bump the all group with 6 updates by @dependabot[bot] in #2454
• deps(npm): bump the all group with 5 updates by @dependabot[bot] in #2452
• feat: expose some optimizer config options by @linhr in #2451
• chore: clippy 1.98.0 warnings by @lonless9 in #2460
• fix: enable metadata-only COUNT(1) for Delta tables by @lonless9 in #2461
• test: port Glue tests to Python by @lonless9 in #2445
• test: fix Iceberg row-level tests on Windows by @lonless9 in #2474
• fix: return signed row counts from delta commit plans by @lonless9 in #2475
• fix: correct DV scans and log replay planning for Delta Lake by @lonless9 in #2450
• feat: improve Celeborn integration by @linhr in #2465
• fix: improve sequence, sampling, and convert_tz parity by @shehabgamin in #2430
• feat: event processor for system tables by @linhr in #2477
• refactor: declare output nullability for scalar functions Spark always marks nullable by @davidlghellin in #2436
• test: port Unity Catalog tests to Python by @lonless9 in #2466
• fix: supervise cluster task panics by @rohankumardubey in #2480
• fix: preserve null rows in Delta DELETE by @lonless9 in #2321
• fix: disabled file cache configuration by @shehabgamin in #2481
• fix: enable partition pruning for Delta MERGE target scans by @lonless9 in #2478
• fix: allow multipart parts to upload concurrently by @shehabgamin in #2482
• test: remove flaky object store test by @linhr in #2485
• perf: decode only the projected columns when reading string-typed CSV by @shehabgamin in #2484
• feat: optimize grouped and exact aggregates from metadata for Delta Lake by @lonless9 in #2467
• feat: update Celeborn endpoint config by @linhr in #2486
• docs: add shuffle and checkpoint guides by @linhr in #2491
• fix: stabilize Iceberg row-level operations with storage shuffle by @lonless9 in #2490
• chore: prepare v0.7.1 by @linhr in #2487

New Contributors

• @ErikBPF made their first contribution in #2371
• @jaideeppyne made their first contribution in #2360
• @maxvwa made their first contribution in #2437

Full Changelog: v0.7.0...v0.7.1