55.0.0

Apache DataFusion Ballista 55.0.0 Changelog

This release consists of 322 commits from 25 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

• fix(scheduler): persist terminal job state before cache eviction #2037 (phillipleblanc)
• fix: spill sort shuffle writer when the memory pool rejects a reservation grow #2061 (andygrove)
• fix: enable the executor client pool by default #2069 (andygrove)
• fix: map a task's partition through UnionExec when restricting file scans #2070 (andygrove)
• fix: keep EmptyExec partition count intact across stage serialization [AQE] #2062 (andygrove)
• fix: remove the static planner SortMergeJoinExec broadcast conversion #2072 (andygrove)
• fix: propagate sort shuffle memory_limit_per_task_bytes to executors #2090 (andygrove)
• fix: executor hanging on ctrl+c #2107 (sandugood)
• fix: Various fixes for AQE join selection #2124 (andygrove)
• fix: Enable more TPC-DS queries #2064 (andygrove)
• fix(executor): keep heartbeating while all task slots are busy #2159 (andygrove)
• fix: unskip five TPC-DS queries by rewriting generator column names #2183 (andygrove)
• fix: stop every task running every union branch at low partition counts #2186 (andygrove)
• fix(scheduler): execute null-aware anti joins in one task #2188 (phillipleblanc)
• fix(scheduler): keep GROUP BY-less aggregates when propagating empty stages #2194 (andygrove)
• fix(scheduler): harden PropagateEmptyExecRule against grouping sets, embedded projections, and dropped exchange partitioning #2200 (andygrove)
• fix(cli): sort jobs by stage completion ratio #1947 (QuakeWang)
• fix(scheduler): preserve stages on transition errors #2215 (QuakeWang)
• fix(scheduler): fail running jobs when all executors are lost #2212 (akshaychitneni)
• fix(executor): bind the gRPC listener before registering with the scheduler #2225 (andygrove)
• fix: update mTLS example for poll_loop arguments #2229 (phillipleblanc)
• fix(scheduler): route launch-failure executor removal through ExecutorLost #2240 (andygrove)
• fix: classify wrapped FetchFailed as fetch partition error #2202 (villebro)
• fix(core): fix main build errors #2254 (avantgardnerio)
• fix: pin datafusion versions to 54.0.0 for python subproject #2277 (milenkovicm)
• fix(scheduler): key AQE shuffle output by job_id #2278 (villebro)
• fix(scheduler): default no-executors grace to executor timeout (>= heartbeat) #2279 (akshaychitneni)
• fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 #2289 (avantgardnerio)
• fix(ci): apply cargo fmt to chaos-executor #2304 (andygrove)
• fix(scheduler): label every operator in the DOT graph #2305 (andygrove)
• fix(core): report the real partitioning from ShuffleWriterExec #2307 (andygrove)
• fix(scheduler): drop an executor's cached gRPC client when it is removed #2310 (Dandandan)
• fix: repair AQE plan rewrites, remove a redundant shuffle, and enable adaptive query planning by default #2315 (Dandandan)
• fix(tui): use the shared API wire types instead of local copies #2258 (andygrove)
• fix(chaos): harden k8s backend — supervise port-forward, guard fixture dir #2346 (akshaychitneni)
• fix(docker): base runtime images on ubuntu:26.04 #2380 (avantgardnerio)
• fix(scheduler): fix test job-completion polling #2354 (NoahKusaba)
• fix(ci): pin tinyvec in the MSRV dependency checker #2421 (andygrove)
• fix(executor): wake graceful shutdown after tasks drain #2393 (1fanwang)
• fix(scheduler): repair test broken by a stale-base merge #2439 (NoahKusaba)
• fix: derive stage input/output rows from the plan's leaves and root instead of summing every operator #2435 (Dandandan)
• fix(ci): pull the MinIO test image from quay.io #2447 (comphead)
• fix(ci): resolve datafusion.proto from datafusion-proto-models #2390 (AnuragRaut08)
• fix(ci): upload both cluster logs from the container benchmark jobs #2460 (amitvijapur)
• fix(ci): run the S3 integration tests against RustFS instead of MinIO #2488 (NoahKusaba)
• fix(executor): forward arrow-ipc-optimizations to ballista-core #2373 (stag7824)
• fix(scheduler): refund the vcores of running tasks when a job is aborted #2477 (amitvijapur)
• fix(scheduler): fail jobs whose tasks cannot be prepared #2492 (andygrove)
• fix(scheduler): declare distributed EXPLAIN output columns not-null #2494 (andygrove)
• fix(core): refuse queries that mix information_schema with other tables #2493 (andygrove)
• fix(release): prepare the 55 release process #2527 (andygrove)
• fix(docker): publish images from release tag pushes #2526 (andygrove)

Performance related:

• perf(kll): batch-oriented ingest KLL #2216 (avantgardnerio)
• perf(core): zero-copy block receive on the remote fetch path #2312 (Dandandan)
• perf(core): push a consumer's row limit into RangeShuffleReaderExec #2334 (Dandandan)
• perf(scheduler): push filters once, not on every replan #2344 (Dandandan)
• perf(scheduler): only give a broadcast build side its own stage when the probe is partitioned #2342 (Dandandan)
• perf(scheduler): read an unfiltered small scan inline instead of staging it #2338 (Dandandan)
• perf(core): write one sort-shuffle file per task instead of one per input #2316 (Dandandan)
• perf(core): reuse shuffle partition-index buffers across batches #2359 (NoahKusaba)
• perf(core): strength reduce hash partition modulo #2363 (NoahKusaba)
• perf(core): skip IPC validation on the sort-shuffle read paths #2365 (NoahKusaba)
• perf(scheduler): stop r...

54.1.0

Apache DataFusion Ballista 54.1.0 Changelog

This release consists of 10 commits from 3 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

• fix: [branch-54] disable the SortMergeJoinExec broadcast conversion by default #2130 (andygrove)
• fix: [branch-54] map a task's partition through UnionExec when restricting file scans (backport of #2070) #2131 (andygrove)
• fix: [branch-54] keep EmptyExec partition count intact across stage serialization (backport of #2062) #2132 (andygrove)
• fix: [branch-54] send the shutdown notification before dropping the notifier (backport of #2107) #2237 (sandugood)
• fix: [branch-54] bind the gRPC listener before registering with the scheduler (backport of #2225) #2236 (andygrove)
• fix: [branch-54] keep GROUP BY-less aggregates when propagating empty stages (backport of #2194) #2235 (andygrove)
• fix: [branch-54] spill sort shuffle writer when the memory pool rejects a reservation grow (backport of #2061) #2238 (andygrove)
• fix: [branch-54] enable the executor client pool by default (backport of #2069) #2234 (andygrove)
• fix: [branch-54] see through Shared wrapper when classifying task IO errors (backport of #2119) #2239 (goingforstudying-ctrl)

Other:

• chore: [branch-54] upgrade DataFusion to 54.1.0 and bump Ballista version to 54.1.0 #2127 (andygrove)

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

     8	Andy Grove
     1	sandugood
     1	goingforstudying-ctrl

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.

54.0.0

Apache DataFusion Ballista 54.0.0 Changelog

This release consists of 171 commits from 11 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

• fix(python): clear latent lint debt and gate wheel jobs on lint/tests #1749 (andygrove)
• fix(python): exclude mimalloc from Python wheel to avoid macOS segfault #1769 (andygrove)
• fix: handle task execution errors and panics in executor #1793 (milenkovicm)
• fix: downgrade time version to fix build issue #1862 (milenkovicm)
• fix: use DataFrame.name instead of hardcoded string in metaclass method wrapping #1866 (djanand)
• fix: honor client datafusion.* session config overrides in scheduler planning #1902 (andygrove)
• fix: Adaptive query planning fails with remote object stores: "No suitable object store found" #1962 (milenkovicm)
• fix: satisfy clippy 1.97 useless_borrows_in_formatting #1978 (andygrove)

Performance related:

• perf: avoid redundant TaskStatus clones in scheduler update_task_status #1983 (andygrove)
• perf: prune irrelevant partitions task #1911 (coderfender)

Implemented enhancements:

• feat: task_manager delegats physical plan creation to execution graph #1726 (milenkovicm)
• feat: ShuffleReader shows upstream_stage #1767 (milenkovicm)
• feat: runtime stage's metrics rendering #1778 (sandugood)
• feat: add /api/job/{job_id}/config endpoint to retrieve session configuration #1792 (killzoner)
• feat(aqe): delay join decision & introduce broadcast join in AQE #1752 (milenkovicm)
• feat: Add web rendering of the TUI application #1742 (martin-g)
• feat(aqe): chaos monkey testing for ballista execution #1802 (milenkovicm)
• feat(aqe): empty join handling #1723 (sandugood)
• feat(tui): Show shimmer animation for "Queued" and "Running" jobs #1808 (martin-g)
• feat(cli): use local time in tracing subscriber logging for ballista-cli #1824 (jiengup)
• feat(scheduler): Use command line options for CORS origins/methods instead of env vars #1818 (martin-g)
• feat(tui): Add configuration popup for jobs #1825 (jiengup)
• feat(TUI): enable various plan rendering formats #1828 (sandugood)
• feat(tui): Add theming support #1850 (martin-g)
• feat: NewType pattern for job_id #1852 (JarroVGIT)
• feat(tui): Show job failed status below the Jobs table #1871 (martin-g)
• feat: broadcast small build sides via non-zero CollectLeft thresholds #1900 (andygrove)
• feat: broadcast small build side of SortMergeJoinExec in the static planner #1904 (andygrove)
• feat: Allow submitting a pre-built physical plan to the scheduler #1924 (andygrove)
• feat: cancel running stages and tasks on job failure or cancellation #1903 (killzoner)
• feat: support running TPC-H benchmarks against S3-compatible object storage #1942 (andygrove)
• feat: bound shuffle fetch with a reduce-side in-flight governor #1951 (andygrove)
• feat: improve session state creation for AQE planner #1963 (milenkovicm)
• feat: delete intermediate shuffle files on job success #1982 (andygrove)
• feat: Share read-side runtime state across a session's tasks on the executor #1995 (andygrove)

Documentation updates:

• docs(release): document PyPI publish process for python client #1610 (andygrove)
• docs(release): cover TestPyPI verification and fix stale wheel-download notes #1760 (andygrove)
• refactor: make use of async_trait::async_trait consistent across project #1848 (JarroVGIT)
• chore(docs): Update docs website to bring inline with Datafusion style #1857 (JarroVGIT)
• Improve changelog generator and split CHANGELOG.md into per-version files #1768 (andygrove)
• docs: add versioned upgrade guide (54.0.0) #1939 (andygrove)
• docs: document TPC-H plan-stability golden files and regeneration #1979 (andygrove)
• docs: update companies using ballista #1992 (coderfender)
• docs: link AQE limitations to tracking issues and add AQE contributor guide section #1990 (andygrove)

Other:

• chore(deps): bump idna from 3.11 to 3.15 in /python #1737 (dependabot[bot])
• chore(ci): Execute the CI workflows only when this is a push to main or a PR against any branch #1738 (martin-g)
• minor: move aqe executions plans to a directory #1744 (milenkovicm)
• ci: pin all third-party actions to ASF-allowlisted SHAs #1748 (andygrove)
• ci: pin Windows wheel builder to windows-2022 #1751 (andygrove)
• chore: bump python crate to 53.0.0 and use path deps for ballista crates #1750 (andygrove)
• ci: produce real aarch64 wheel and upload sdist artifact #1753 (andygrove)
• ci: only run Python Release Build on RC tags and matching PRs #1754 (andygrove)
• ci: gate TPC-H workflow on rust-only path allow list #1755 (andygrove)
• ci: gate Docker workflow on rust-only path allow list #1756 (andygrove)
• ci: gate Rust workflow on workspace path allow list and rename to Ballista Rust #1757 (andygrove)
• ci: use maturin sdist subcommand to avoid building a wheel #1759 (andygrove)
• ci: gate CodeQL workflow on .github path changes #1758 (andygrove)
• minor: Ballista showplan helper script #1761 (milenkovicm)
• chore(deps): bump requests from 2.32.5 to 2.33.0 in /python #1781 (dependabot[bot])
• chore(ci): Execute the "Dependencies" workflow on push to main and for PRs #1782 (martin-g)
• chore(deps): bump astral-tokio-tar from 0.6.1 to 0.6.2 #1794 (dependabot[bot])
• chore(deps): bump pygments from 2.19.2 to 2.20.0 in /python #1800 (dependabot[bot])
• minor: code cleanup and log level changes [#1805](https://github.com/apache/datafusion-balli...

53.0.0

Apache DataFusion Ballista 53.0.0 Changelog

This release consists of 174 commits from 16 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

• fix: remove unwrap from executor and improve task error handling #1540 (eachsaj)
• fix: handle None task slot in update_task_info after executor lost #1523 (milenkovicm)
• fix: [Python] serialize optimized logical plan to fix subquery support #1586 (andygrove)
• fix: allow S3 access without explicit credentials #1584 (andygrove)
• fix: route collect/show/to_pandas through Ballista cluster #1585 (andygrove)
• fix: propagate session config from Python client to Ballista cluster #1592 (andygrove)
• fix(python): ignore ballista-namespaced cluster_config keys locally #1613 (andygrove)
• fix: df.write_ fix as it was broken after update #1625 (milenkovicm)
• fix(rest): remove unwrap and return 404 if executor does not exist #1628 (milenkovicm)
• fix(metrics): avoid stage metrics inflation by tracking partition snapshots #1652 (danielhumanmod)
• fix: compilation issue after merge #1658 (milenkovicm)
• fix: rest api calculates stage running time correctly #1675 (milenkovicm)
• fix: REST API does not show running jobs #1703 (gittihub-jpg)
• fix: no executor warning, correct prometheus feature name in TUI #1698 (killzoner)

Implemented enhancements:

• [REST]: Cancelling a completed/failed job should return "false" #1494 (martin-g)
• feat: Add plain "status" field to the JobResponse #1497 (martin-g)
• feat: Expose Logical and Physical plan details in the REST API #1498 (milenkovicm)
• feat: (remote) shuffle reader cleanup #1503 (milenkovicm)
• feat: enable scheduler rest api by default #1506 (milenkovicm)
• feat: [REST] Return the job's start and end times #1511 (martin-g)
• feat: remove hash policy task distribution as not used #1526 (eachsaj)
• feat: remove full path from partition locations #1527 (milenkovicm)
• feat: add config to ExecutorEngine::create_query_stage_exec #1542 (milenkovicm)
• feat: Improve REST API adding task related info to job stages #1543 (milenkovicm)
• feat: Ballista Text User Interface #1436 (martin-g)
• feat: jupyter notebook support #1513 (sandugood)
• feat: ExecutionEngine::create_query_stage_exec accepts partition_id #1556 (milenkovicm)
• feat: make ballista client retry policy configurable #1577 (eachsaj)
• feat: Executor system & process metrics reporting #1547 (sandugood)
• feat: Scheduler config update #1597 (sandugood)
• feat: Support EXPLAIN ANALYZE in Ballista #1567 (danielhumanmod)
• feat: Add standalone shuffle writer benchmark that shuffles real Parquet input #1600 (andygrove)
• feat: defer sort-shuffle materialization with interleave_record_batch #1598 (andygrove)
• feat(executor): bound executor memory via --memory-pool-size #1624 (andygrove)
• feat(bench): show TPC-H query timings in seconds and add total time #1641 (andygrove)
• feat(aqe): Support sort-based shuffle writer in AQE #1640 (danielhumanmod)
• feat: rest api supports plan tree rendering #1650 (sandugood)
• feat: Cache ballista clients on executor #1578 (milenkovicm)
• feat(aqe): Lazy stage evaluation in AQE #1649 (milenkovicm)
• feat: move shuffle writer disk I/O off tokio worker threads #1537 (hcrosse)
• feat(scheduler): broadcast-style hash join for small-side joins #1647 (andygrove)
• feat: default to sort-merge join #1651 (andygrove)
• feat: TUI shows running job information #1717 (milenkovicm)
• feat(aqe): CoalescePartitionsRule — shuffle-partition coalescing on resolved stats #1684 (metegenez)
• feat: TUI make task popup scrollable #1725 (milenkovicm)
• feat(tui): Use separate areas for the table and its associated scrollbar #1729 (martin-g)

Documentation updates:

• docs: document hash-based and sort-based shuffle implementations #1595 (andygrove)
• docs: improve Python documentation structure #1579 (andygrove)
• docs: add Python client build-from-source section to contributor guide #1620 (andygrove)
• chore: remove NYC Taxi benchmark #1644 (andygrove)
• docs: document experimental Adaptive Query Execution in tuning guide #1645 (andygrove)
• build(bench): rework docker-compose TPC-H stack #1646 (andygrove)
• docs: add Ballista TUI documentation #1593 (goingforstudying-ctrl)
• [TUI] Show executor's details in a popup #1670 (martin-g)
• [TUI] Add a config setting for rendering job stage's plan as a tree #1704 (martin-g)
• [TUI] Add support for horizontal scrolling to the job/stage plan popups #1711 (martin-g)
• [TUI] Add screenshots of the TUI application in README/cli.md #1714 (martin-g)

Other:

• CI: Add CodeQL workflow for GitHub Actions security scanning #1484 (kevinjqliu)
• ci: Harden labeler workflow, remove unnecessary checkout from pull_request_target job #1487 (kevinjqliu)
• chore(deps): bump tokio from 1.49.0 to 1.50.0 #1490 (dependabot[bot])
• chore(deps): bump actions/setup-node from 6.2.0 to 6.3.0 #1489 (dependabot[bot])
• ci: add take and stale #1488 (kevinjqliu)
• chore(deps): bump uuid from 1.21.0 to 1.22.0 #1492 (dependabot[bot])
• chore(deps): bump github/codeql-action from 4.32.5 to 4.32.6 #1491 (dependabot[bot])
• chore(deps): bump libc from 0.2.182 to 0.2.183 #1495 (dependabot[bot])
• chore(deps): bump graphviz-rust from 0.9.6 to 0.9.7 #1496 (dependabot[bot])
• chore(deps): bump quinn-proto from 0.11.13 to 0.11.14 [...

52.0.0

Apache DataFusion Ballista 52.0.0 Changelog

Performance related:

• perf: optimize shuffle writer with buffered I/O and fix file size bug #1386 (andygrove)

Implemented enhancements:

• feat: add config option for skipping arrow ipc read validation #1374 (killzoner)
• feat: improve tpch benchmark CLI #1391 (andygrove)
• feat: Add sort-based shuffle implementation #1389 (andygrove)
• feat: New ballista python interface #1338 (milenkovicm)
• feat: Add batch coalescing ability to shuffle reader exec #1380 (danielhumanmod)
• feat: Add arrow flight proxy to scheduler #1351 (sebbegg)
• feat: Creating SubstraitSchedulerClient and standalone Substrait examples #1376 (mattcuento)
• feat: Cluster RPC customisations to support TLS and custom headers #1400 (phillipleblanc)
• feat: add -c config override flag to tpch benchmark #1435 (andygrove)
• feat: Extract execution_graph to a trait #1361 (milenkovicm)
• feat: Add spark-compat mode to integrate datafusion-spark features au… #1416 (mattcuento)
• feat: add Dataframe.cache() factory (no planner handling) #1420 (killzoner)
• feat: Adaptive query execution (AQE) planner fundamentals #1372 (milenkovicm)
• feat: Make push scheduling policy default as it has lower latency #1461 (milenkovicm)
• feat: job scheduling with push based job status updates #1478 (milenkovicm)

Fixed bugs:

• fix: compile issue after unsuccessful merge #1402 (milenkovicm)
• fix: prost build keda and TLS RPC example #1429 (killzoner)
• fix: remove scheduler_config_spec.toml as it is unused #1462 (milenkovicm)
• fix: Don't use maxrows as a "fetched rows" but calculate it from the batches #1480 (martin-g)

Documentation updates:

• docs: fix outdated content in documentation #1385 (andygrove)
• docs: use tpchgen-rs for TPC-H data generation #1390 (andygrove)
• docs: add Jupyter notebook support documentation #1399 (andygrove)
• chore: Document ballista features in README.md #1418 (mattcuento)

Merged pull requests:

• feat: add config option for skipping arrow ipc read validation #1374 (killzoner)
• docs: fix outdated content in documentation #1385 (andygrove)
• restrict python CI to python directory #1383 (Huy1Ng)
• perf: optimize shuffle writer with buffered I/O and fix file size bug #1386 (andygrove)
• docs: use tpchgen-rs for TPC-H data generation #1390 (andygrove)
• feat: improve tpch benchmark CLI #1391 (andygrove)
• doc: Add Ballista extensions example to the docs. #1382 (LouisBurke)
• feat: Add sort-based shuffle implementation #1389 (andygrove)
• feat: New ballista python interface #1338 (milenkovicm)
• doc: add more details for protobuf extension #1393 (LouisBurke)
• feat: Add batch coalescing ability to shuffle reader exec #1380 (danielhumanmod)
• docs: add Jupyter notebook support documentation #1399 (andygrove)
• feat: Add arrow flight proxy to scheduler #1351 (sebbegg)
• chore: update datafusion to 52 #1394 (killzoner)
• feat: Creating SubstraitSchedulerClient and standalone Substrait examples #1376 (mattcuento)
• fix: compile issue after unsuccessful merge #1402 (milenkovicm)
• feat: Cluster RPC customisations to support TLS and custom headers #1400 (phillipleblanc)
• chore: Document ballista features in README.md #1418 (mattcuento)
• fix: prost build keda and TLS RPC example #1429 (killzoner)
• Improve sort-based shuffle: single spill file per partition and batch coalescing #1431 (andygrove)
• feat: add -c config override flag to tpch benchmark #1435 (andygrove)
• feat: Extract execution_graph to a trait #1361 (milenkovicm)
• chore: add confirmation before tarball is released #1445 (milenkovicm)
• minor: add test to cover IPC arrow file read #1450 (milenkovicm)
• feat: Add spark-compat mode to integrate datafusion-spark features au… #1416 (mattcuento)
• feat: add Dataframe.cache() factory (no planner handling) #1420 (killzoner)
• fix: remove scheduler_config_spec.toml as it is unused #1462 (milenkovicm)
• feat: Adaptive query execution (AQE) planner fundamentals #1372 (milenkovicm)
• feat: Make push scheduling policy default as it has lower latency #1461 (milenkovicm)
• minor: improve log statements #1482 (milenkovicm)
• chore: update datafusion to 52.2 and other deps to latest #1483 (milenkovicm)
• fix: Don't use maxrows as a "fetched rows" but calculate it from the batches #1480 (martin-g)
• feat: job scheduling with push based job status updates #1478 (milenkovicm)

51.0.0

Apache DataFusion Ballista 51.0.0 Changelog

Implemented enhancements:

• feat: Support distributed plan in EXPLAIN command #1309 (danielhumanmod)
• feat: update rust edition to 2024 #1355 (killzoner)
• feat: capture more metrics in distributed_query #1353 (PhVHoang)
• feat: Bump docker rust to rust:1.92-trixie #1365 (mattcuento)
• feat: Scheduler supports substrait logical plan and remove deprecated sql support #1360 (mattcuento)

Merged pull requests:

• minor: minor changes to release script/docs #1342 (andygrove)
• feat: Support distributed plan in EXPLAIN command #1309 (danielhumanmod)
• chore: update datafusion to 51.0 #1345 (danielhumanmod)
• doc: Add a note that datafusion may need to be downgraded after installing it #1348 (martin-g)
• minor: Make DisplayAs consistent and more readable for ShuffleExec #1347 (milenkovicm)
• minor: remove unnecessary clone functions #1352 (mmooyyii)
• feat: update rust edition to 2024 #1355 (killzoner)
• chore(doc): Clean up deployment docs. #1354 (LouisBurke)
• feat: capture more metrics in distributed_query #1353 (PhVHoang)
• doc: Fix plan translation example to use correct aggregation and column #1362 (mattcuento)
• chore: update ballista version to 51.0.0 (from 50.0.0) #1363 (milenkovicm)
• feat: Bump docker rust to rust:1.92-trixie #1365 (mattcuento)
• chore: Add missing public API documentation/comments #1364 (killzoner)
• feat: Scheduler supports substrait logical plan and remove deprecated sql support #1360 (mattcuento)

50.0.0

Apache DataFusion Ballista 50.0.0 Changelog

Implemented enhancements:

• feat: make gRPC timeout configurations user-configurable #1337 (CuteChuanChuan)

Fixed bugs:

• fix: update REST API route syntax for axum 0.8 compatibility #1330 (tomsanbear)
• fix: Executor does not panic if using unwritable work dir #1332 (mach-kernel)
• fix: failing documentation #1339 (milenkovicm)

Merged pull requests:

• infra: macos-13 is deprecated #1324 (kevinjqliu)
• chore: update to datafusion v50 #1320 (milenkovicm)
• chore: Update datafusion to 50.2 #1326 (milenkovicm)
• document private items and update docs CI #1327 (killzoner)
• fix: update REST API route syntax for axum 0.8 compatibility #1330 (tomsanbear)
• add msrvcheck #1328 (killzoner)
• fix: Executor does not panic if using unwritable work dir #1332 (mach-kernel)
• chore: Pinning versions of external actions. #1334 (samueleresca)
• chore: update python deps to 49 #1335 (milenkovicm)
• chore: update datafusion to 50.3 #1336 (milenkovicm)
• fix: failing documentation #1339 (milenkovicm)
• feat: make gRPC timeout configurations user-configurable #1337 (CuteChuanChuan)
• minor: change log level #1340 (milenkovicm)

49.0.0

Apache DataFusion Ballista 49.0.0 Changelog

Full Changelog

Implemented enhancements:

• feat: Feat force shuffle reader to read all files using arrow flight client #1310 (milenkovicm)
• feat: implement job data cleanup in pull-staged strategy #1219 #1314 (KR-bluejay)

Fixed bugs:

• fix: fail job in case of serde error (pull-mode) #1297 (milenkovicm)
• fix: Ensure stage-level sort requirements are enforced in distributed planning #1306 (metegenez)
• fix: UnresolvedShuffleExec should support with_new_children #1300 (milenkovicm)
• fix: ShuffleReader should return statistics #1302 (milenkovicm)
• fix: Disable CollectLeft join as it is broken in ballista #1301 (milenkovicm)
• fix: Issue with JoinSelection and NestedLoopJoin::swap_inputs when stage is resolved #1307 (milenkovicm)

Merged pull requests:

• chore: update datafusion to 49 #1285 (milenkovicm)
• chore: Improve GitHub actions/python workflows #1289 (Huy1Ng)
• chore: update datafusion to 49.0.2 #1298 (milenkovicm)
• minor: make shuffle exec display consistent #1299 (milenkovicm)
• fix: fail job in case of serde error (pull-mode) #1297 (milenkovicm)
• fix: Ensure stage-level sort requirements are enforced in distributed planning #1306 (metegenez)
• fix: UnresolvedShuffleExec should support with_new_children #1300 (milenkovicm)
• fix: ShuffleReader should return statistics #1302 (milenkovicm)
• fix: Disable CollectLeft join as it is broken in ballista #1301 (milenkovicm)
• chore: notice and cargo deps cleanup #1295 (milenkovicm)
• chore(deps): bump tracing-subscriber from 0.3.19 to 0.3.20 #1303 (dependabot[bot])
• chore(deps): bump tracing-subscriber from 0.3.19 to 0.3.20 in /python #1308 (dependabot[bot])
• refactor: expand ClusterEventSender visibility to public #1312 (Th824)
• fix: Issue with JoinSelection and NestedLoopJoin::swap_inputs when stage is resolved #1307 (milenkovicm)
• minor: enable json write test #1311 (milenkovicm)
• feat: Feat force shuffle reader to read all files using arrow flight client #1310 (milenkovicm)
• feat: implement job data cleanup in pull-staged strategy #1219 #1314 (KR-bluejay)
• feat: Improve Remote Shuffle Read Speed and Resource Utilisation #1318 (milenkovicm)
• fix: Issue with JoinSelection and CrossJoinExec when stages have been resoled #1322 (ZihuanLing)

48.0.0

Apache DataFusion Ballista 48.0.0 Changelog

Full Changelog

Fixed bugs:

• fix: devcontainer protoc:1 feature url #1278 (Almaz-KG)
• fix: change code to support rust 2024 #1283 (milenkovicm)
• fix: remove configure_me #1282 (milenkovicm)

Documentation updates:

• chore: update datafusion to 48 #1270 (milenkovicm)
• docs: Apply method chaining in example #1276 (0ne-stone)

Merged pull requests:

• chore: update datafusion to 48 #1270 (milenkovicm)
• docs: Apply method chaining in example #1276 (0ne-stone)
• fix: devcontainer protoc:1 feature url #1278 (Almaz-KG)
• improve rust workflows without cache #1275 (Huy1Ng)
• fix: change code to support rust 2024 #1283 (milenkovicm)
• fix: remove configure_me #1282 (milenkovicm)
• chore: update python module to latest ballista release (v.47) #1279 (milenkovicm)
• chore: update datafusion to 48.0.1 #1284 (milenkovicm)

47.0.0

Apache DataFusion Ballista 47.0.0 Changelog

Full Changelog

Implemented enhancements:

• feat: expose submit and cancel job methods as public in scheduler #1260 (milenkovicm)
• feat: split scheduler gprc creation #1261 (milenkovicm)
• feat: expose cluster state notifications #1263 (milenkovicm)
• feat: remove ClusterStorageConfig as it is redundant #1265 (milenkovicm)
• feat: disable task stage plan binary cache #1266 (milenkovicm)
• feat: ClusterState does not cache session contexts #1226 (milenkovicm)

Fixed bugs:

• fix: clippy issue after rust update to 1.87 #1262 (milenkovicm)
• fix: fix tests failing on windows #1273 (Huy1Ng)

Merged pull requests:

• fix: clippy issue after rust update to 1.87 #1262 (milenkovicm)
• feat: expose submit and cancel job methods as public in scheduler #1260 (milenkovicm)
• feat: split scheduler gprc creation #1261 (milenkovicm)
• feat: expose cluster state notifications #1263 (milenkovicm)
• minor: release docker on when release has been tagged #1264 (milenkovicm)
• feat: remove ClusterStorageConfig as it is redundant #1265 (milenkovicm)
• feat: disable task stage plan binary cache #1266 (milenkovicm)
• feat: ClusterState does not cache session contexts #1226 (milenkovicm)
• chore(deps): update to datafusion 47.0.0 #1250 (milenkovicm)