Skip to content

v1.0.0-rc.3

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 27 Oct 02:47
· 172 commits to main since this release
18c4ee7

The third release candidate of Midstream: a streaming platform on Kaspr that materializes Kafka topics into Apache Iceberg tables on Google Cloud Storage, with a console and an Iceberg REST catalog.

What changed since 1.0.0-rc.2: domains

One console now serves any number of Midstream deployments — domains, one per cluster or team — each keeping its own accounts, datasets, tables, bucket and settings (handbook 20).

  • One console, one URL. Every domain's pages under /d/<domain>, a domain switcher, and the fleet, datasets and search across every domain at once. A release of the chart is a domain (domain.id, domain.name) and can run the shared console with its registry of every domain (console.domains[]), the console alone (controlPlane.enabled: false), or a domain alone (console.enabled: false). Old per-domain hostnames redirect to /d/<domain>/ (route.redirectFrom).
  • Sign-in through Microsoft Entra ID, GitHub, Google or any OpenID Connect provider, into every domain at once. The console runs the flow; each domain checks the provider's answer itself and makes the account under its own first-sign-in policy (open or invited) and allowed email domains. Password sign-in stays, one domain at a time, and a domain can turn it off for everyone but its administrators.
  • Access tokens every domain honours, signed with a federation key the domains share; revoking one reaches every domain.
  • SQL across domains: tables are <domain>.<dataset>.<table> (native: <domain>_iceberg.<dataset>.<table>), and a statement can join tables of several domains, each checked against what the person may do in its own domain.
  • One Iceberg REST catalog over every domain at the console's /catalog, with namespaces <domain> and <domain>.<dataset>; each domain's own at /d/<domain>/catalog.

Validated on a lab cluster with two domains, through the API and in a browser, with fault injection and a performance comparison against 1.0.0-rc.2 on a 111-million-record topic: the same 102,033,103 rows committed in 88.9 minutes against 89.4, and console requests within about 1.5 ms of rc.2's (handbook 20 §20.15).

Install

Artifact Where
Helm chart oci://ghcr.io/vmirz/charts/midstream, version 1.0.0-rc.3
Console image ghcr.io/vmirz/midstream-console:1.0.0-rc.3
Python package midstream-platform 1.0.0rc3 on PyPI

The installation guide, docs/install.md, takes an empty namespace to a first materialization, and its §7 adds more domains and sign-in through a provider.

Requires Kubernetes with kaspr-operator 0.19.7 or newer and Kaspr 0.11.30 or newer, a Kafka cluster, and a GCS bucket. Upgrading from rc.2 is a clean install: the console's settings, the session cookie, SQL names and the access-token format all changed. Delete the materializations, uninstall, and install each domain and the console anew — what changed for whom is in handbook 16 §16.14.