Ballista 53.0.0 blog post by andygrove · Pull Request #188 · apache/datafusion-site

andygrove · 2026-05-24T14:26:51Z

Summary

Staged version: https://datafusion.staged.apache.org/blog/2026/05/24/datafusion-ballista-53.0.0/

Adds release blog post for Apache DataFusion Ballista 53.0.0
Covers changes since 43.0.0 (production deployment, shuffle subsystem, REST API, Python interface, Spark/Substrait)
Covers 53.0.0-specific highlights (TUI, experimental AQE, EXPLAIN ANALYZE, sort-merge join default, sort-based shuffle default)
Covers active work and roadmap based on open issues and EPICs
Notes that Python wheels in 53.0.0 still report 52.0.0 and that 53-line wheels will land with 53.1.0 on PyPI

Tracking issue: apache/datafusion-ballista#1762

Test plan

Render locally with make and inspect formatting / image references
Confirm link to the previous Ballista 43.0.0 post resolves

milenkovicm · 2026-05-24T15:26:52Z

+- Per-executor system and process metrics are reported, and Prometheus metrics integration is available
+  behind a feature flag.
+
+### A new Python interface


Perhaps we should mention work with datafusion python team to improve this integration?

milenkovicm

Few minor comments, good overview.
I'm very happy with this release

milenkovicm · 2026-05-24T21:37:23Z

+[Apache DataFusion]: https://datafusion.apache.org
+
+The last Ballista blog post covered [43.0.0], released in January 2025. In the year and a bit since, the
+project has quietly shipped a release for every DataFusion release: 44, 45, 46, 47, 48, 49, 50, 51, 52, and


Do we need to mention every release by number ? Each of them is dear to my heart but it might be a bit excessive 😀

milenkovicm · 2026-05-24T21:42:55Z

+Spark-compatible SQL semantics on a Ballista cluster.
+
+The scheduler also has a Substrait surface: `SubstraitSchedulerClient` accepts Substrait logical plans, and
+the deprecated SQL-string submission path has been removed. This is an important step toward decoupling


SQL string submission was very small functionality to mention here, maybe we do not need it

milenkovicm · 2026-05-24T21:49:39Z

+Plan rendering, including a graph view, is available directly from the TUI:
+
+<img
+src="/blog/images/datafusion-ballista-53.0.0/tui-job-plan-graph-popup.png"


I believe image with text plan might be better than graph current one

Can you point me to an appropriate image to use?

Keeping the graph image in this revision — happy to add a text-plan screenshot in a follow-up if you have one you'd like to use.

we have

but its very busy.

We can create two

Swapped in the two text-plan screenshots you posted (logical + physical) in place of the graph image. Thanks for grabbing them.

milenkovicm · 2026-05-24T21:53:30Z

+## Thank You
+
+This release is the result of work from many contributors over the past 16 months. Thanks especially to
+Marko Milenković, Martin Grigorov, Daniel Tu, Alexander Domenti, Mete Genez, Saj, Harrison Crosse, and


Mere Genez => Metehan Yildirim
Saj => Sajeevan Achuthan

milenkovicm · 2026-05-24T21:56:22Z

+
+This release is the result of work from many contributors over the past 16 months. Thanks especially to
+Marko Milenković, Martin Grigorov, Daniel Tu, Alexander Domenti, Mete Genez, Saj, Harrison Crosse, and
+many others whose contributions are visible in the [changelog]. Thanks also to the broader DataFusion


You've forgotten to put your name Andy

Claude doesn't value my contributions to this release, apparently!

kevinjqliu

LGTM!

maybe mention the new logo! it looks great

kevinjqliu · 2026-05-25T04:25:17Z

+
+- **S3 object store support** has been added to both the executor and scheduler binaries, including
+  credentials derived from the standard AWS environment, instance metadata, and explicit configuration.
+- **Docker images** for the scheduler and executor are now published on each release, making Docker Compose


maybe link to https://datafusion.apache.org/ballista/user-guide/deployment/docker.html

kevinjqliu · 2026-05-25T04:25:42Z

+
+### REST API and observability
+
+The scheduler's REST API has grown from a small status surface to the primary control plane for inspecting


maybe link https://datafusion.apache.org/ballista/user-guide/scheduler.html#rest-api

kevinjqliu · 2026-05-25T04:28:38Z

+The release process has also been extended so that future Ballista releases will publish Python wheels to
+[PyPI] as `ballista`. Note that the Python bindings included in **53.0.0 still report version 52.0.0**
+because the version bump landed shortly after the 53.0.0 release candidate was tagged. Wheels matching
+the 53 line will be published with **53.1.0**, which is expected to follow shortly.


is this still the case? the testpypi package reports 53.0.0

uv run --python 3.10 --with "ballista==53.0.0" \ --index-url https://test.pypi.org/simple/ \ --extra-index-url https://pypi.org/simple/ \ --index-strategy unsafe-best-match \ python -u -c "import ballista; print(ballista.__version__)"

Yeah, this was fixed in 53.0.0-rc1-testpypi3 but was incorrect in 53.0.0-rc1 .. maybe I should just skip this note

martin-g · 2026-05-25T13:39:35Z

+alt="Ballista TUI plan graph popup"
+/>
+
+A web rendering of the TUI is in development.


We are very optimistic here! 😄
The web version of the TUI is functional but it is very ugly! Ratzilla produces a ton of empty spans (<span> </span>) to render empty "pixel"s

While writing the above I realized that the WebTUI could look much better if it was rendered in a <canvas> HTML element and I went to suggest this to the Ratzilla project but obviously someone else already realized this before me! 😄

Now I just need to make a different banner for the WebTUI build!

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>

andygrove · 2026-05-25T14:10:31Z

LGTM!

maybe mention the new logo! it looks great

We already announced the new logo in the previous blog post: https://datafusion.apache.org/blog/output/2025/02/02/datafusion-ballista-43.0.0/

- Tighten release-numbers paragraph - Link Docker images and REST API mentions to the user guide - Acknowledge collaboration with the datafusion-python team - Drop wheel-version 52.0.0 caveat (wheels now report 53.0.0) - Remove mention of the deprecated SQL-string submission path - Replace TUI graph image with logical and physical text-plan screenshots - Fix contributor names and add Andy Grove to the Thank You section

andygrove · 2026-05-25T14:15:08Z

Thanks for the reviews @milenkovicm @kevinjqliu @martin-g. I will go ahead and merge. Feel free to create follow on PRs if you spot any issues once this is live.

andygrove · 2026-05-25T14:15:45Z

@milenkovicm I need an approval before I can merge - could you oblige?

milenkovicm · 2026-05-25T14:16:44Z

Thanks a lot @andygrove

andygrove · 2026-05-25T14:22:22Z

Post is live at https://datafusion.apache.org/blog/output/2026/05/24/datafusion-ballista-53.0.0/

Ballista 53.0.0 blog post

9c66a80

andygrove marked this pull request as draft May 24, 2026 14:27

milenkovicm reviewed May 24, 2026

View reviewed changes

kevinjqliu reviewed May 25, 2026

View reviewed changes

martin-g reviewed May 25, 2026

View reviewed changes

Update content/blog/2026-05-24-datafusion-ballista-53.0.0.md

fe3f543

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>

andygrove marked this pull request as ready for review May 25, 2026 14:09

milenkovicm approved these changes May 25, 2026

View reviewed changes

andygrove merged commit c3f829f into main May 25, 2026
4 checks passed

andygrove deleted the site/ballista-53 branch May 25, 2026 14:19


		### REST API and observability

		The scheduler's REST API has grown from a small status surface to the primary control plane for inspecting

Conversation

andygrove commented May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

milenkovicm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andygrove commented May 25, 2026

Uh oh!

andygrove commented May 25, 2026

Uh oh!

andygrove commented May 25, 2026

Uh oh!

milenkovicm commented May 25, 2026

Uh oh!

Uh oh!

andygrove commented May 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andygrove commented May 24, 2026 •

edited

Loading