From dc32930b660a812b9aa4b366833106cd51664681 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Sun, 9 Aug 2026 08:42:56 -0600 Subject: [PATCH] docs: add Ballista 54.1.0 release announcement blog post --- .../2026-08-09-datafusion-ballista-54.1.0.md | 96 +++++++++++++++++++ 1 file changed, 96 insertions(+) create mode 100644 content/blog/2026-08-09-datafusion-ballista-54.1.0.md diff --git a/content/blog/2026-08-09-datafusion-ballista-54.1.0.md b/content/blog/2026-08-09-datafusion-ballista-54.1.0.md new file mode 100644 index 00000000..e6fec162 --- /dev/null +++ b/content/blog/2026-08-09-datafusion-ballista-54.1.0.md @@ -0,0 +1,96 @@ +--- +layout: post +title: Apache DataFusion Ballista 54.1.0 Released +date: 2026-08-09 +author: pmc +categories: [release] +--- + + +[TOC] + +We are pleased to announce version [54.1.0] of [Apache DataFusion Ballista]. Ballista is a distributed query +execution engine that enhances [Apache DataFusion] by enabling parallel execution of workloads across multiple +nodes. + +[54.1.0]: https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md +[Apache DataFusion Ballista]: https://datafusion.apache.org/ballista/ +[Apache DataFusion]: https://datafusion.apache.org + +This is a patch release, consisting of 10 commits from 3 contributors. It follows [54.0.0] and contains no +new features and no breaking changes — only an upgrade to DataFusion 54.1.0 and a set of bug fixes +backported from `main`. Users running 54.0.0 are encouraged to upgrade. + +[54.0.0]: /blog/2026/07/12/datafusion-ballista-54.0.0/ + +## Upgrade to DataFusion 54.1.0 + +Ballista 54.1.0 is built on DataFusion 54.1.0, picking up the fixes in that patch release. No API changes +were required, so the [54.0.0 upgrade guide] still applies for anyone moving from an earlier release. + +[54.0.0 upgrade guide]: https://datafusion.apache.org/ballista/upgrading/54.0.0.html + +## Cluster stability fixes + +Three fixes address problems that only appear in a real multi-node deployment: + +- **Executor client pool enabled by default.** The executor's `client_ttl` defaulted to `0`, which disabled + the gRPC client pool entirely. Without a pool, the shuffle-read path opened and discarded a TCP connection + per fetch, so a single shuffle-heavy query could exhaust the host's ephemeral ports and fail with + `AddrNotAvailable` — as would every query after it, until those sockets left `TIME_WAIT`. A single-executor + cluster never showed this, because shuffle reads are served from local files. `client_ttl` now defaults to + 30 seconds, so the pool is on unless explicitly disabled. +- **Executor startup race.** The executor spawned its gRPC server and registered with the scheduler + immediately afterwards, but the socket was not bound yet when the scheduler dialed back. Losing that race + produced `ECONNREFUSED`, a failed registration, and an executor that died at startup. The listener is now + bound before registration can run. +- **Executor shutdown.** The shutdown notification is now sent before the notifier is dropped, so a graceful + shutdown is actually observed by the components waiting on it. + +## Task retry and shuffle memory + +- **Retryable IO errors on a join's build side.** A retryable IO error raised on a join's shared build side + arrives at the failed-task classifier wrapped in `DataFusionError::Shared`, so a match on the outermost + variant never saw the underlying `IoError` and the task was marked non-retryable. Classification now + operates on the root error, regardless of wrapping. +- **Sort shuffle writer respects memory pool pressure.** The sort shuffle writer discarded the result of + `try_grow` on its memory reservation, so a rejected grow produced no spill: the batch stayed buffered while + the shared `MemoryPool` had not accounted for it, and other consumers saw memory as free that was not. A + rejected grow is now treated as a spill trigger alongside the existing per-task budget. + +## Planning correctness + +- **`GROUP BY`-less aggregates are preserved** when the scheduler propagates empty stages, so a query whose + input produces no rows still returns the correct single-row aggregate result rather than nothing. +- **`EmptyExec` partition counts survive stage serialization**, fixing plans whose partitioning changed as a + stage was sent to an executor. +- **Task partitions are mapped through `UnionExec`** when restricting file scans, so each task scans the + files it was assigned. +- **`SortMergeJoinExec` broadcast conversion is now disabled by default.** The static-planner optimization + that broadcasts the small build side of a sort-merge join, added in 54.0.0, is opt-in in this release while + outstanding issues are resolved. + +## Thank You + +Thanks to Andy Grove, Alexander Domenti, and goingforstudying-ctrl for the commits in this release, and to +everyone who contributed by filing issues, reviewing PRs, and providing feedback. Full details are in the +[changelog]. + +[changelog]: https://github.com/apache/datafusion-ballista/blob/54.1.0/docs/source/changelog/54.1.0.md