Skip to content

Releases

v1.3.2

Choose a tag to compare

@github-actions github-actions released this 05 Nov 18:28
20f89b0

What's Changed

  • README: what Midstream is, in screenshots and short lists by @vmirz in #33
  • Console: counts that agree, a lag floor that needs the records read, grouped numbers by @vmirz in #34
  • Release 1.3.2: definitions bundles by @vmirz in #36
  • Deliver every Kaspr app's components in a definitions bundle by @vmirz in #35

Full Changelog: v1.3.1...v1.3.2

v1.3.1

Choose a tag to compare

@github-actions github-actions released this 03 Nov 13:46
f4f57e0

What's Changed

  • Keep the console's DuckDB within its pod, and end runs it lost by @vmirz in #31
  • Release 1.3.1: the console's memory by @vmirz in #32

Full Changelog: v1.3.0...v1.3.1

v1.3.0

Choose a tag to compare

@github-actions github-actions released this 03 Nov 03:19
cf9733a

What's Changed

  • Console: every screen in Current, and Settings says workers by @vmirz in #26
  • Invitations: invite people by email, across domains, before they sign in by @vmirz in #27
  • Table page: each tab keeps its place until the page is left by @vmirz in #28
  • Console: Explore, explorations over any table, dataset and domain by @vmirz in #29
  • Release 1.3.0: Explore and invitations by @vmirz in #30

Full Changelog: v1.2.0...v1.3.0

v1.2.0

Choose a tag to compare

@github-actions github-actions released this 01 Nov 22:39
36a83c4

What's Changed

  • Console: the Current design language by @vmirz in #19
  • Console: a page for every table, with the waterline by @vmirz in #20
  • Console: a steadier Data view and the Connect tab in Current by @vmirz in #21
  • Keep the console to its design language: a skill, the rule, and a check by @vmirz in #22
  • Console: a plan before edit and delete, datasets shared like documents, searchable Settings by @vmirz in #23
  • Console: veiled masks, a row beside the grid, small questions run as asked, longer charts by @vmirz in #24
  • Release 1.2.0: the console in Current by @vmirz in #25

Full Changelog: v1.1.0...v1.2.0

v1.1.0

Choose a tag to compare

@github-actions github-actions released this 31 Oct 15:04
43f6ab8

What's Changed

  • Storage backends: the lake on GCS, Amazon S3 or Azure Blob Storage by @vmirz in #17
  • Release 1.1.0: storage backends by @vmirz in #18

Full Changelog: v1.0.0-rc.5...v1.1.0

v1.0.0-rc.5

v1.0.0-rc.5 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 30 Oct 20:35
63c15e8

What's Changed

  • Refuse source topics Kafka does not list; document the Kafka user's ACLs by @vmirz in #11
  • Describe test-cluster runs in general terms by @vmirz in #12
  • Publish a console image of every pushed commit; re-apply a runtime taken back to an earlier spec by @vmirz in #13
  • Data protection: mask sensitive fields before they reach the lake by @vmirz in #14
  • Reprotect: mask the rows already written by @vmirz in #15
  • Release 1.0.0-rc.5: data protection by @vmirz in #16

Full Changelog: v1.0.0-rc.4...v1.0.0-rc.5

v1.0.0-rc.4

v1.0.0-rc.4 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 27 Oct 23:17
9174a37

What's Changed

  • Kafka over TLS: kaspr.tls for every KasprApp and Midstream's own clients by @vmirz in #9
  • Release 1.0.0-rc.4: Kafka over TLS by @vmirz in #10

Full Changelog: v1.0.0-rc.3...v1.0.0-rc.4

v1.0.0-rc.3

v1.0.0-rc.3 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 27 Oct 02:47
18c4ee7

The third release candidate of Midstream: a streaming platform on Kaspr that materializes Kafka topics into Apache Iceberg tables on Google Cloud Storage, with a console and an Iceberg REST catalog.

What changed since 1.0.0-rc.2: domains

One console now serves any number of Midstream deployments — domains, one per cluster or team — each keeping its own accounts, datasets, tables, bucket and settings (handbook 20).

  • One console, one URL. Every domain's pages under /d/<domain>, a domain switcher, and the fleet, datasets and search across every domain at once. A release of the chart is a domain (domain.id, domain.name) and can run the shared console with its registry of every domain (console.domains[]), the console alone (controlPlane.enabled: false), or a domain alone (console.enabled: false). Old per-domain hostnames redirect to /d/<domain>/ (route.redirectFrom).
  • Sign-in through Microsoft Entra ID, GitHub, Google or any OpenID Connect provider, into every domain at once. The console runs the flow; each domain checks the provider's answer itself and makes the account under its own first-sign-in policy (open or invited) and allowed email domains. Password sign-in stays, one domain at a time, and a domain can turn it off for everyone but its administrators.
  • Access tokens every domain honours, signed with a federation key the domains share; revoking one reaches every domain.
  • SQL across domains: tables are <domain>.<dataset>.<table> (native: <domain>_iceberg.<dataset>.<table>), and a statement can join tables of several domains, each checked against what the person may do in its own domain.
  • One Iceberg REST catalog over every domain at the console's /catalog, with namespaces <domain> and <domain>.<dataset>; each domain's own at /d/<domain>/catalog.

Validated on a lab cluster with two domains, through the API and in a browser, with fault injection and a performance comparison against 1.0.0-rc.2 on a 111-million-record topic: the same 102,033,103 rows committed in 88.9 minutes against 89.4, and console requests within about 1.5 ms of rc.2's (handbook 20 §20.15).

Install

Artifact Where
Helm chart oci://ghcr.io/vmirz/charts/midstream, version 1.0.0-rc.3
Console image ghcr.io/vmirz/midstream-console:1.0.0-rc.3
Python package midstream-platform 1.0.0rc3 on PyPI

The installation guide, docs/install.md, takes an empty namespace to a first materialization, and its §7 adds more domains and sign-in through a provider.

Requires Kubernetes with kaspr-operator 0.19.7 or newer and Kaspr 0.11.30 or newer, a Kafka cluster, and a GCS bucket. Upgrading from rc.2 is a clean install: the console's settings, the session cookie, SQL names and the access-token format all changed. Delete the materializations, uninstall, and install each domain and the console anew — what changed for whom is in handbook 16 §16.14.

v1.0.0-rc.2

v1.0.0-rc.2 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 26 Oct 05:00
bc557cb

The second release candidate of Midstream: a streaming platform on Kaspr that materializes Kafka topics into Apache Iceberg tables on Google Cloud Storage, with a console and an Iceberg REST catalog.

What changed since 1.0.0-rc.1

  • The chart installs the release it belongs to. package.source defaults to index: every app and runtime installs midstream-platform at the chart's appVersion from PyPI. package.source: git remains the development source.
  • kaspr.config: the Kaspr settings an installation shares, applied to every app and runtime under the chart's own keys. kaspr.config.topicReplicationFactor: 1 is what a cluster with fewer than three brokers needs (Kaspr's default is 3).
  • The install guide shows the published chart, a Kafka listener without credentials (kaspr.authentication: {type: plain}), and the init container's real name, install-packages.
  • Documentation reads Midstream throughout; the source of truth is docs/source-of-truth.md, and the handbook records the first installation from the published artifacts, verified on a cluster from scratch (docs/handbook/13-testing-and-development.md §13.8i).

Install

Artifact Where
Helm chart oci://ghcr.io/vmirz/charts/midstream, version 1.0.0-rc.2
Console image ghcr.io/vmirz/midstream-console:1.0.0-rc.2
Python package midstream-platform 1.0.0rc2 on PyPI

The installation guide, docs/install.md, takes an empty namespace to a first materialization: Secrets, values, helm install, first sign-in.

Requires Kubernetes with kaspr-operator 0.19.7 or newer and Kaspr 0.11.30 or newer, a Kafka cluster, and a GCS bucket. Upgrading from rc.1: helm upgrade with the new version; an installation that set K_TOPIC_REPLICATION_FACTOR through extraEnv can move it to kaspr.config.topicReplicationFactor.

v1.0.0-rc.1

v1.0.0-rc.1 Pre-release
Pre-release

Choose a tag to compare

@vmirz vmirz released this 26 Oct 03:52
d107e53

The first release candidate of Midstream: a streaming platform on Kaspr that materializes Kafka topics into Apache Iceberg tables on Google Cloud Storage, with a console and an Iceberg REST catalog.

Install

Artifact Where
Helm chart oci://ghcr.io/vmirz/charts/midstream, version 1.0.0-rc.1
Console image ghcr.io/vmirz/midstream-console:1.0.0-rc.1
Python package midstream-platform 1.0.0rc1 on PyPI

The installation guide, docs/install.md, takes an empty namespace to a first materialization: Secrets, values, helm install, first sign-in.

Requires Kubernetes with kaspr-operator 0.19.7 or newer and Kaspr 0.11.30 or newer, a Kafka cluster, and a GCS bucket.

Assets

The wheel and sdist attached here are the files published to PyPI, and the chart archive is the one published to GHCR.