v1.23 — Explore to flow, drill-through, and pushdown engines in open core

Explore → flow

Create node (above any chart in Explore) turns what you built into a processor in the open flow, saved and ready to wire in. Choose what the node returns:

• The data, with the chart's filters (the default): every column, so the flow does its own shaping
• The chart's result: the aggregated query itself

The engine follows the source:

• Node outputs and persisted tables become a DuckDB processor over its input df
• Live databases offer pandas (reads the result into SFS through get_engine()) or SQL ⚡ (runs inside the database; only the result comes back)

Drill-through

Go from a chart over cached data to the detail rows behind it, in a live table.

• Set it up once per source: pick a connection and a detail table, and map the key fields. Same-named text and date columns are suggested.
• Drill lists the chart's points; pick one to see up to 1,000 matching rows, filtered by that point's values and the chart's filters.
• Fields without a mapping are listed as not applied, never guessed.
• Explore these rows opens the detail query as a live SQL source.

The Explore node opens the Explore view

The Explore node now keeps its input from each run and opens it in the Explore view inside SFS: click Open in Explore on the node or in its panel. Existing flows work without changes; the old Report name setting is ignored.

PyGWalker has been removed from the report viewer. Explore reports published by older versions still appear there, as a notice and a 1,000-row preview. The Report node (Streamlit dashboards) is unchanged.

SQL and Ibis pushdown and the advanced scheduler are open core

These were labelled enterprise but always shipped in this repository. They are now open core and labelled free in the app and the docs, and they stay here:

• SQL ⚡ and Ibis ⚡ pushdown engines
• Advanced scheduling: retries with exponential backoff, Slack and email failure alerts, calendar rules (weekdays only, skip dates)

Future enterprise features (authentication and SSO, RBAC, scaling across machines, support) will be new capabilities, shipped as a separately licensed plugin on top of this core. See EDITIONS.md.

Install note: the pip extra for the pushdown engines is now pushdown:

pip install -e ".[all-connectors,pushdown]"

The old name enterprise still works, so existing SFS_EXTRAS settings keep building.

Fixes

• Processor output preview: clicking a processor's output after a run returned a server error (since v1.10). The preview also loaded the entire frame to show 500 rows; it now reads only the rows it shows.
• Explore node before its first run: now says "Run the flow, then open it again" instead of an internal error.

Moved

The repository now lives in the Sidewinder Labs organisation: github.com/sidewinder-labs/sidewinder-flow-studio. Old links redirect. To update an existing clone:

git remote set-url origin https://github.com/sidewinder-labs/sidewinder-flow-studio.git

v1.20 — Explore: drag-and-drop charts over your flows and your databases

Explore

A new Explore view (top-bar button) replaces the old PyGWalker-based Explore node as the way to look at your data. Drag fields onto a chart, add filters, bin, drill into dates, and each chart is compiled to SQL and run where the data lives. Only the aggregated result reaches the browser, so a 30-million-row table answers in well under a second.

What you can explore

• Node outputs: any cached frame from a flow run, grouped by project and flow
• Persisted tables: anything checkpointed with Persist output
• Live databases: open a connection to browse its schemas and tables, or write a SQL query and explore its result, without building a flow first

Built for real databases

• Charts are translated to each database's own SQL (PostgreSQL, ClickHouse, MySQL, SQL Server, Snowflake, Trino, …)
• Date filters are sent as plain ranges on the column, so indexes and partition pruning still apply
• Every chart has a row cap (50,000 by default) and a query timeout (30 s), enforced by the database itself where it supports one
• A SQL source must be a single SELECT; anything else is refused before it reaches the database
• Show SQL lists every query with its timings
• Views can be bookmarked; the URL carries the source

Tip: use read-only credentials for connections you explore.

New settings: EXPLORE_ROW_CAP, EXPLORE_TIMEOUT_S.

The Explore node still works for now and will open the new view in a later release.

Fixes

• DuckDB 2.0: DuckDB nodes failed on fresh images because DuckDB 2.0 renamed its Arrow API. SFS now works with DuckDB 1.x and 2.x.
• ClickHouse fast_write: loads over HTTP wrote empty rows. Data is now sent as Arrow in 100k-row chunks, and a missing table is created automatically (MergeTree, nullable columns). Load from a Polars or DuckDB node to keep integer types.
• Local AI (Ollama) in Docker: connection failures now explain the fix (host.docker.internal, OLLAMA_HOST=0.0.0.0). Slow local models get a longer timeout, and long generated code is no longer cut off.
• AI-generated code: steps that compare dates now convert them first, instead of silently returning no rows.
• Processor panel: on 1080p screens the code editor covered the Save button. The panel now scrolls.

Upgrading

Rebuild your images: docker compose build --no-cache && docker compose up -d. No migrations; existing projects and connections work as before.

v1.18 — Docker images ship connector drivers by default

Connector drivers in the Docker images

A fresh clone used to build a backend image with no warehouse drivers, so connection tests failed only in Docker. Both images now install the drivers by default.

• SFS_EXTRAS build argument picks which pip extras go into the image. It defaults to all-connectors,enterprise and is passed through from both compose files. Set it in .env for a smaller image (SFS_EXTRAS=trino,enterprise) or leave it empty for core engines only (SFS_EXTRAS=).
• Driver summary at build time: the build log prints one line listing which drivers installed, so a missing driver shows up before you run anything.
• Single-container image (Dockerfile.single) now does a regular install instead of an editable pip install -e ., matching the backend image.

SQL Server (mssql) still needs OS-level ODBC libraries, so it shows as missing in the stock image.

Examples you can run

The engine hints, AI prompt placeholders and CAPABILITIES.md examples now use the free employees sample database, so every example works against data anyone can load.

See docs/INSTALLATION.md → "Choosing drivers for the Docker image".

v1.17 — Out-of-core engines, parallel branches, robust fast_write

Covers v1.13–v1.17.

Larger-than-RAM data on one machine

• DuckDB scans the cached Arrow file instead of loading it into memory: it reads only the columns a query uses and spills to disk above its memory limit (SFS_DUCKDB_MEMORY_LIMIT, or per node).
• DuckDB as a source: a DuckDB node right after a trigger can read files directly — read_csv_auto(...), read_parquet(...).
• Polars streams its result straight to the cache with sink_ipc, so outputs needn't fit in memory either.

Parallel branches

Set SFS_PARALLEL_BRANCHES=1 to run independent branches concurrently. Nodes are grouped into dependency layers and each layer runs together. Off by default.

fast_write

• Accepts pandas, Polars (eager or lazy) and Arrow tables
• Creates missing tables from the frame's schema; recreate=True rebuilds after column changes
• PostgreSQL COPY streams in chunks — tested at 33.7M rows

Reliability

• A connection whose password can't be decrypted (SECRET_KEY changed) no longer blocks every flow — only nodes using it fail, with a clear message
• Run-start failures now appear as a banner instead of silently doing nothing
• Docker image includes greenlet, required by SQLAlchemy's async engine

v1.12 — Isolated plugin runtime, Router node, engine parameters

Plugins can now run in their own virtual environment, and flows can branch on their data.

Isolated plugin runtime

A plugin can execute its work in a separate interpreter with its own pinned dependencies, exchanging Apache Arrow files. The SFS process never imports the library, so packages with aggressive version pins (PyCaret and friends) can't disturb the ETL environment.

export SFS_PLUGIN_VENVS_DIR=./plugin_venvs
python -m venv $SFS_PLUGIN_VENVS_DIR/sklearn
$SFS_PLUGIN_VENVS_DIR/sklearn/bin/pip install pyarrow pandas scikit-learn

plugins/sfs-plugin-sklearn now ships both variants — scikit-learn (in-process) and scikit-learn (isolated) — sharing one runner, as a reference for your own plugins.

Router node

Conditional branching. Each branch has a label and a Python expression evaluated against df; the first true one wins and the others are skipped. Branches appear as separate connection points on the node, so you wire them visually — and a node that several branches feed into still runs.

len(df) == 0
df.amount.sum() > 1_000_000
df.status.eq('error').any()

Engine parameters

Engines can declare form fields — string, number, boolean, select, column (a dropdown of the upstream frame's real columns), connection — rendered above the code editor and delivered to the engine with defaults applied. Optional and additive: engines that declare none are unchanged, and the code cell always remains.

Documentation

Added deployment guidance that was previously implicit: node code runs with the backend's privileges and is not sandboxed, node boundaries materialise full frames, execution is sequential, and scheduled retries re-run the whole flow (so make writes idempotent). EDITIONS.md now states explicitly that functionality already published here will not be retroactively removed.