Skip to content

Release v0.12.6

Choose a tag to compare

@Edwardvaneechoud Edwardvaneechoud released this 30 Jun 20:54
ab765ba

This release lets the catalog store table data in object storage (S3, GCS, or Azure) instead of only the local filesystem, reworks the designer into a tabbed, draggable panel layout with an open-flows view on the home screen, and fixes the AI planner looping when it should add a join node.

Object storage for the catalog (#555)

Catalog tables can now live in cloud object storage, with the metadata database staying local.

  • Pick storage per catalog in the UI: creating a top-level catalog now has an Object storage (optional) section — enter a URI (e.g. s3://bucket/catalog) and choose an existing cloud-storage connection for credentials. Leave it blank to keep storing locally.
  • S3, GCS, and Azure are supported (s3:///s3a://, gs:///gcs://, az:///abfs:///abfss:///adl://) — anything backed by a CloudStorageConnection.
  • Schemas and tables inherit their backend from the parent catalog; you set it once at the catalog level.
  • Set a global default for new catalogs with FLOWFILE_CATALOG_STORAGE_URI plus FLOWFILE_CATALOG_STORAGE_CONNECTION (the name of an existing cloud connection); the per-catalog UI/API setting overrides it. Unset ⇒ local Delta directories (today's behavior).
  • New catalog UI: a storage badge in the catalog tree showing where a table lives (provider + location) and a namespace detail panel showing the full storage configuration; table version history is cached for snappier reopens.
  • Reads, writes, merges, and optimize/vacuum all work against both local and cloud backends; the worker decrypts the owner-encrypted connection itself, so the $ffsec$ secret format and worker/scheduler are unchanged.
  • Forward-only by design: the config only steers brand-new tables; existing tables keep writing to their own stored location, so switching the config later never misroutes or corrupts a table's path.

Tabbed, dynamic designer screens (#557)

The designer's separate floating panels were consolidated into a flexible tabbed-drawer layout.

  • Unified right drawer with Settings / Results / Code tabs, and a unified bottom dock with Data / Logs tabs — replacing the previous independent floating windows (the AI assistant stays its own drawer).
  • Draggable, resizable panels that scale relative to the canvas, with new tabs auto-focusing (e.g. the Code tab on Ctrl+G).
  • Data preview now shows a "Select a step to observe the data" placeholder when no node is selected, and clicking a node in the Results panel opens its data in the bottom dock.
  • Open flows on the home screen: a new section lists currently-loaded flows with a Running badge and an unsaved-changes indicator; click one to jump straight back into it.
  • Kafka sync creation moved into the Kafka Connection Manager (a dedicated Kafka Syncs card), and the catalog header actions were collapsed into a single New dropdown.

Fixes & improvements

  • AI planner no longer loops when combining sources (#556): asking the agent to join/merge two sources used to get it stuck retrying an invalid connect; it now recognizes that combining data means adding a join or union node and pivots to that automatically. Backed by stricter connection validation (rejecting wiring into a source or a non-existent target with clear reasons) and updated planner prompts.

Upgrade notes

  • Database: migration 028 runs automatically on startup, adding the nullable storage_uri and storage_connection_name columns to catalog_namespaces. Additive and forward-only — existing catalogs read back NULL (local filesystem), so no manual action or backfill is needed.
  • Catalog object storage is opt-in. Existing catalogs and tables stay local; only catalogs you explicitly point at object storage (in the UI, or via FLOWFILE_CATALOG_STORAGE_URI + FLOWFILE_CATALOG_STORAGE_CONNECTION for new catalogs) use the cloud backend.

Full changelog: v0.12.5...v0.12.6