Repository navigation
Releases
v1.3.2
What's Changed
- README: what Midstream is, in screenshots and short lists by @vmirz in #33
- Console: counts that agree, a lag floor that needs the records read, grouped numbers by @vmirz in #34
- Release 1.3.2: definitions bundles by @vmirz in #36
- Deliver every Kaspr app's components in a definitions bundle by @vmirz in #35
Full Changelog: v1.3.1...v1.3.2
v1.3.1
What's Changed
- Keep the console's DuckDB within its pod, and end runs it lost by @vmirz in #31
- Release 1.3.1: the console's memory by @vmirz in #32
Full Changelog: v1.3.0...v1.3.1
v1.3.0
What's Changed
- Console: every screen in Current, and Settings says workers by @vmirz in #26
- Invitations: invite people by email, across domains, before they sign in by @vmirz in #27
- Table page: each tab keeps its place until the page is left by @vmirz in #28
- Console: Explore, explorations over any table, dataset and domain by @vmirz in #29
- Release 1.3.0: Explore and invitations by @vmirz in #30
Full Changelog: v1.2.0...v1.3.0
v1.2.0
What's Changed
- Console: the Current design language by @vmirz in #19
- Console: a page for every table, with the waterline by @vmirz in #20
- Console: a steadier Data view and the Connect tab in Current by @vmirz in #21
- Keep the console to its design language: a skill, the rule, and a check by @vmirz in #22
- Console: a plan before edit and delete, datasets shared like documents, searchable Settings by @vmirz in #23
- Console: veiled masks, a row beside the grid, small questions run as asked, longer charts by @vmirz in #24
- Release 1.2.0: the console in Current by @vmirz in #25
Full Changelog: v1.1.0...v1.2.0
v1.1.0
What's Changed
- Storage backends: the lake on GCS, Amazon S3 or Azure Blob Storage by @vmirz in #17
- Release 1.1.0: storage backends by @vmirz in #18
Full Changelog: v1.0.0-rc.5...v1.1.0
v1.0.0-rc.5
What's Changed
- Refuse source topics Kafka does not list; document the Kafka user's ACLs by @vmirz in #11
- Describe test-cluster runs in general terms by @vmirz in #12
- Publish a console image of every pushed commit; re-apply a runtime taken back to an earlier spec by @vmirz in #13
- Data protection: mask sensitive fields before they reach the lake by @vmirz in #14
- Reprotect: mask the rows already written by @vmirz in #15
- Release 1.0.0-rc.5: data protection by @vmirz in #16
Full Changelog: v1.0.0-rc.4...v1.0.0-rc.5
v1.0.0-rc.4
What's Changed
- Kafka over TLS: kaspr.tls for every KasprApp and Midstream's own clients by @vmirz in #9
- Release 1.0.0-rc.4: Kafka over TLS by @vmirz in #10
Full Changelog: v1.0.0-rc.3...v1.0.0-rc.4
v1.0.0-rc.3
The third release candidate of Midstream: a streaming platform on Kaspr that materializes Kafka topics into Apache Iceberg tables on Google Cloud Storage, with a console and an Iceberg REST catalog.
What changed since 1.0.0-rc.2: domains
One console now serves any number of Midstream deployments — domains, one per cluster or team — each keeping its own accounts, datasets, tables, bucket and settings (handbook 20).
- One console, one URL. Every domain's pages under
/d/<domain>, a domain switcher, and the fleet, datasets and search across every domain at once. A release of the chart is a domain (domain.id,domain.name) and can run the shared console with its registry of every domain (console.domains[]), the console alone (controlPlane.enabled: false), or a domain alone (console.enabled: false). Old per-domain hostnames redirect to/d/<domain>/(route.redirectFrom). - Sign-in through Microsoft Entra ID, GitHub, Google or any OpenID Connect provider, into every domain at once. The console runs the flow; each domain checks the provider's answer itself and makes the account under its own first-sign-in policy (
openorinvited) and allowed email domains. Password sign-in stays, one domain at a time, and a domain can turn it off for everyone but its administrators. - Access tokens every domain honours, signed with a federation key the domains share; revoking one reaches every domain.
- SQL across domains: tables are
<domain>.<dataset>.<table>(native:<domain>_iceberg.<dataset>.<table>), and a statement can join tables of several domains, each checked against what the person may do in its own domain. - One Iceberg REST catalog over every domain at the console's
/catalog, with namespaces<domain>and<domain>.<dataset>; each domain's own at/d/<domain>/catalog.
Validated on a lab cluster with two domains, through the API and in a browser, with fault injection and a performance comparison against 1.0.0-rc.2 on a 111-million-record topic: the same 102,033,103 rows committed in 88.9 minutes against 89.4, and console requests within about 1.5 ms of rc.2's (handbook 20 §20.15).
Install
| Artifact | Where |
|---|---|
| Helm chart | oci://ghcr.io/vmirz/charts/midstream, version 1.0.0-rc.3 |
| Console image | ghcr.io/vmirz/midstream-console:1.0.0-rc.3 |
| Python package | midstream-platform 1.0.0rc3 on PyPI |
The installation guide, docs/install.md, takes an empty namespace to a first materialization, and its §7 adds more domains and sign-in through a provider.
Requires Kubernetes with kaspr-operator 0.19.7 or newer and Kaspr 0.11.30 or newer, a Kafka cluster, and a GCS bucket. Upgrading from rc.2 is a clean install: the console's settings, the session cookie, SQL names and the access-token format all changed. Delete the materializations, uninstall, and install each domain and the console anew — what changed for whom is in handbook 16 §16.14.
v1.0.0-rc.2
The second release candidate of Midstream: a streaming platform on Kaspr that materializes Kafka topics into Apache Iceberg tables on Google Cloud Storage, with a console and an Iceberg REST catalog.
What changed since 1.0.0-rc.1
- The chart installs the release it belongs to.
package.sourcedefaults toindex: every app and runtime installsmidstream-platformat the chart's appVersion from PyPI.package.source: gitremains the development source. kaspr.config: the Kaspr settings an installation shares, applied to every app and runtime under the chart's own keys.kaspr.config.topicReplicationFactor: 1is what a cluster with fewer than three brokers needs (Kaspr's default is 3).- The install guide shows the published chart, a Kafka listener without credentials (
kaspr.authentication: {type: plain}), and the init container's real name,install-packages. - Documentation reads Midstream throughout; the source of truth is
docs/source-of-truth.md, and the handbook records the first installation from the published artifacts, verified on a cluster from scratch (docs/handbook/13-testing-and-development.md§13.8i).
Install
| Artifact | Where |
|---|---|
| Helm chart | oci://ghcr.io/vmirz/charts/midstream, version 1.0.0-rc.2 |
| Console image | ghcr.io/vmirz/midstream-console:1.0.0-rc.2 |
| Python package | midstream-platform 1.0.0rc2 on PyPI |
The installation guide, docs/install.md, takes an empty namespace to a first materialization: Secrets, values, helm install, first sign-in.
Requires Kubernetes with kaspr-operator 0.19.7 or newer and Kaspr 0.11.30 or newer, a Kafka cluster, and a GCS bucket. Upgrading from rc.1: helm upgrade with the new version; an installation that set K_TOPIC_REPLICATION_FACTOR through extraEnv can move it to kaspr.config.topicReplicationFactor.
v1.0.0-rc.1
The first release candidate of Midstream: a streaming platform on Kaspr that materializes Kafka topics into Apache Iceberg tables on Google Cloud Storage, with a console and an Iceberg REST catalog.
Install
| Artifact | Where |
|---|---|
| Helm chart | oci://ghcr.io/vmirz/charts/midstream, version 1.0.0-rc.1 |
| Console image | ghcr.io/vmirz/midstream-console:1.0.0-rc.1 |
| Python package | midstream-platform 1.0.0rc1 on PyPI |
The installation guide, docs/install.md, takes an empty namespace to a first materialization: Secrets, values, helm install, first sign-in.
Requires Kubernetes with kaspr-operator 0.19.7 or newer and Kaspr 0.11.30 or newer, a Kafka cluster, and a GCS bucket.
Assets
The wheel and sdist attached here are the files published to PyPI, and the chart archive is the one published to GHCR.