Release v0.12.6
This release lets the catalog store table data in object storage (S3, GCS, or Azure) instead of only the local filesystem, reworks the designer into a tabbed, draggable panel layout with an open-flows view on the home screen, and fixes the AI planner looping when it should add a join node.
Object storage for the catalog (#555)
Catalog tables can now live in cloud object storage, with the metadata database staying local.
- Pick storage per catalog in the UI: creating a top-level catalog now has an Object storage (optional) section — enter a URI (e.g.
s3://bucket/catalog) and choose an existing cloud-storage connection for credentials. Leave it blank to keep storing locally. - S3, GCS, and Azure are supported (
s3:///s3a://,gs:///gcs://,az:///abfs:///abfss:///adl://) — anything backed by aCloudStorageConnection. - Schemas and tables inherit their backend from the parent catalog; you set it once at the catalog level.
- Set a global default for new catalogs with
FLOWFILE_CATALOG_STORAGE_URIplusFLOWFILE_CATALOG_STORAGE_CONNECTION(the name of an existing cloud connection); the per-catalog UI/API setting overrides it. Unset ⇒ local Delta directories (today's behavior). - New catalog UI: a storage badge in the catalog tree showing where a table lives (provider + location) and a namespace detail panel showing the full storage configuration; table version history is cached for snappier reopens.
- Reads, writes, merges, and optimize/vacuum all work against both local and cloud backends; the worker decrypts the owner-encrypted connection itself, so the
$ffsec$secret format and worker/scheduler are unchanged. - Forward-only by design: the config only steers brand-new tables; existing tables keep writing to their own stored location, so switching the config later never misroutes or corrupts a table's path.
Tabbed, dynamic designer screens (#557)
The designer's separate floating panels were consolidated into a flexible tabbed-drawer layout.
- Unified right drawer with Settings / Results / Code tabs, and a unified bottom dock with Data / Logs tabs — replacing the previous independent floating windows (the AI assistant stays its own drawer).
- Draggable, resizable panels that scale relative to the canvas, with new tabs auto-focusing (e.g. the Code tab on Ctrl+G).
- Data preview now shows a "Select a step to observe the data" placeholder when no node is selected, and clicking a node in the Results panel opens its data in the bottom dock.
- Open flows on the home screen: a new section lists currently-loaded flows with a Running badge and an unsaved-changes indicator; click one to jump straight back into it.
- Kafka sync creation moved into the Kafka Connection Manager (a dedicated Kafka Syncs card), and the catalog header actions were collapsed into a single New dropdown.
Fixes & improvements
- AI planner no longer loops when combining sources (#556): asking the agent to join/merge two sources used to get it stuck retrying an invalid
connect; it now recognizes that combining data means adding a join or union node and pivots to that automatically. Backed by stricter connection validation (rejecting wiring into a source or a non-existent target with clear reasons) and updated planner prompts.
Upgrade notes
- Database: migration 028 runs automatically on startup, adding the nullable
storage_uriandstorage_connection_namecolumns tocatalog_namespaces. Additive and forward-only — existing catalogs read backNULL(local filesystem), so no manual action or backfill is needed. - Catalog object storage is opt-in. Existing catalogs and tables stay local; only catalogs you explicitly point at object storage (in the UI, or via
FLOWFILE_CATALOG_STORAGE_URI+FLOWFILE_CATALOG_STORAGE_CONNECTIONfor new catalogs) use the cloud backend.
Full changelog: v0.12.5...v0.12.6