Release v0.24.0: new compute engine

[Edwardvaneechoud]

Changes since v0.23.3.

A lot changed in this release, and ideally you won't notice too of it.

Flowfile has a new compute engine underneath. Every node now runs on a compute backend, through a versioned protocol between the server that plans your flow and the worker that does the work. If everything went right, your flows open, run and preview exactly as they did before. That was the goal: change the foundation without moving the furniture.

Why

Until now, a flow could only use the machine that ran Flowfile. That's fine on a laptop, but a team or a large dataset outgrows one machine quickly. From this release on, the editor and the workers no longer have to live on the same machine, and that opens up a lot:

• Use multiple machines to run your flows. Add several workers and choose which flow, or even which node, runs on which. A heavy join can go to a big machine while the rest of the flow stays on the default worker.
• Compute close to your data. A worker in your own cloud account reads, transforms and writes your object storage directly. Only the preview travels back to Flowfile, not your data.
• Polars Cloud. The protocol is designed so native Polars Cloud support can be added later, giving you horizontal scaling when your data asks for it.

What changed underneath

• Core plans, the worker computes (Feature/implement details new compute setup #852, Feature/default execution to new backend #861). Core now sends each node's plan to a worker and gets back a reference to the result plus a preview. Core never opens a file the worker wrote, which is what lets the worker run on another machine with its own storage. This is the default on the desktop app, Docker and the pip install.

• Add remote workers (Feature/implement details new compute setup #852, Compute/release gate #866). Add a worker in Settings → Execution → Backends with its URL and token, then click Handshake. Flowfile checks that the worker speaks the same protocol and runs the same Polars build before it sends any work. The Remote workers guide walks through the setup.

• Choose where flows and nodes run (Feature/implement details new compute setup #852). Flow Settings → Execution location replaces the old Local/Remote switch. Existing flows open as Auto, which behaves the way they always did. A node can also pick its own Compute backend in its General Settings, and if it can't run where you put it, the canvas tells you why before the run starts rather than halfway through.

• Sturdier scheduled and CLI runs (Feature/implement details new compute setup #852, Compute/release gate #866). Scheduled runs and flowfile run flow now run each node in its own child process. A node that runs out of memory fails on its own instead of taking the whole run down, and Cancel actually stops the node that's running. There's also a new --backend <id> option to pick where a CLI run computes. More in Where flows run.

• Credentials stay out of plans (Feature/implement details new compute setup #852). A cloud read now carries only a reference to its stored connection. The credentials are looked up where the plan runs, so they never travel inside the plan itself.

Other improvements

• Unpivot by data type: pick all numeric, float, string or date columns in the editor, or use on=ff.numeric() from Python (fix: add unpivot data-type selectors and enhance unpivot functionality #864).
• Empty catalogs and schemas can now be edited and deleted (Feature/implement details new compute setup #852).
• Cancelling a run no longer leaves nodes stuck as running (Fix cancellation state on canvas after run ends #811).
• Previews show again for flows in Performance mode (Enhance execution logic to support preview mode in performance flows #868).
• Notebook: no more phantom changes on REST, database and Kafka sources, and resetting a node clears its old error (fix: enforce source columns in notebook by caching fields and updating select input logic #865, Clear errors on node reset and ensure proper seeding after failed reads in notebook kernel #867).
• Flow as an API runs even when its API Response node was never configured (Fix/set default for api response #863).

Upgrading

Most setups need nothing. Check these:

• Docker: scheduled runs moved to the core container. They now compute there and use its memory. Set FLOWFILE_COMPUTE_BACKEND=worker in core's environment to keep them on the worker container.
• Core and workers need the same Polars build. Desktop and Docker use Polars' 64-bit runtime. For a pip-installed core with a separate worker, install polars-runtime-64 at the same Polars version. If they don't match, the handshake refuses the worker and says why.
• Worker pool on by default, plus a time limit. The warm pool (4 processes) now runs on every platform: faster node starts, a bit more memory at idle. Scheduled and CLI runs stop a node after one hour. Turn these off with FLOWFILE_WORKER_POOL_SIZE=0 and FLOWFILE_WORKER_TASK_TIMEOUT=0.
• Database migrations 036–038 run on startup. A SQLite catalog is backed up first.

Falling back to the old execution model

The previous execution model is still in this release. If a flow behaves differently after upgrading, set FLOWFILE_COMPUTE_PROTOCOL=0 in the environment Flowfile starts in (Docker: your .env). Core then uses the old worker routes again.

If you run into something like this, please open an issue so I can fix it. The same goes for anything else that doesn't work as expected.

Security

• python-jose is replaced by PyJWT.
• fsspec, s3fs, gcsfs, filelock, gitpython and aiohttp are updated.
• Kernel images move to 0.6.4 with a patched pillow (Fix backend dependency vulnerabilities #853, Drop vulnerable pillow 12.3.0 from the kernel lock #854).

New contributor 🎉

A big thanks to @LynxLeuck for their first contribution! #811 fixes how the canvas shows a cancelled run, and comes with tests for cancelling while switching flows, refreshing mid-cancel and a Save As during a run.

Full Changelog: v0.23.3...v0.24.0

---

This discussion was created from the release Release v0.24.0: new compute engine.