Repository navigation
v1.0.0-rc.3
Pre-releaseThe third release candidate of Midstream: a streaming platform on Kaspr that materializes Kafka topics into Apache Iceberg tables on Google Cloud Storage, with a console and an Iceberg REST catalog.
What changed since 1.0.0-rc.2: domains
One console now serves any number of Midstream deployments — domains, one per cluster or team — each keeping its own accounts, datasets, tables, bucket and settings (handbook 20).
- One console, one URL. Every domain's pages under
/d/<domain>, a domain switcher, and the fleet, datasets and search across every domain at once. A release of the chart is a domain (domain.id,domain.name) and can run the shared console with its registry of every domain (console.domains[]), the console alone (controlPlane.enabled: false), or a domain alone (console.enabled: false). Old per-domain hostnames redirect to/d/<domain>/(route.redirectFrom). - Sign-in through Microsoft Entra ID, GitHub, Google or any OpenID Connect provider, into every domain at once. The console runs the flow; each domain checks the provider's answer itself and makes the account under its own first-sign-in policy (
openorinvited) and allowed email domains. Password sign-in stays, one domain at a time, and a domain can turn it off for everyone but its administrators. - Access tokens every domain honours, signed with a federation key the domains share; revoking one reaches every domain.
- SQL across domains: tables are
<domain>.<dataset>.<table>(native:<domain>_iceberg.<dataset>.<table>), and a statement can join tables of several domains, each checked against what the person may do in its own domain. - One Iceberg REST catalog over every domain at the console's
/catalog, with namespaces<domain>and<domain>.<dataset>; each domain's own at/d/<domain>/catalog.
Validated on a lab cluster with two domains, through the API and in a browser, with fault injection and a performance comparison against 1.0.0-rc.2 on a 111-million-record topic: the same 102,033,103 rows committed in 88.9 minutes against 89.4, and console requests within about 1.5 ms of rc.2's (handbook 20 §20.15).
Install
| Artifact | Where |
|---|---|
| Helm chart | oci://ghcr.io/vmirz/charts/midstream, version 1.0.0-rc.3 |
| Console image | ghcr.io/vmirz/midstream-console:1.0.0-rc.3 |
| Python package | midstream-platform 1.0.0rc3 on PyPI |
The installation guide, docs/install.md, takes an empty namespace to a first materialization, and its §7 adds more domains and sign-in through a provider.
Requires Kubernetes with kaspr-operator 0.19.7 or newer and Kaspr 0.11.30 or newer, a Kafka cluster, and a GCS bucket. Upgrading from rc.2 is a clean install: the console's settings, the session cookie, SQL names and the access-token format all changed. Delete the materializations, uninstall, and install each domain and the console anew — what changed for whom is in handbook 16 §16.14.