Skip to content

Release v0.3.0

Choose a tag to compare

@homer6 homer6 released this 17 Oct 23:51
· 253 commits to main since this release
d7551c7

v0.3.0

Released: 2026-10-17 · Branch: dev · Previous: v0.2.0 (2026-10-12)

v0.2.0 was the engine a hosted workspace runs on. v0.3.0 is its part of secure
networking (orama-bridge#186):
a workspace Zetta now keeps the workspace's network history, gives each app connection its
own scoped credential, and joins Zetta nodes in separate private networks through
zetta-tunnel peer gateways. The mutual-TLS and framing code the cluster uses is now a
library that zetta-tunnel links too.

Seven pull requests since v0.2.0: four features (#1284, #1285, #1286, #1287), a test
fixture fix (#1293), the release procedure (#1278), and the merge of main back into
dev after v0.2.0 (#1279).
The generated OpenAPI has 176 /v1 paths (was 172), and the test sources hold 2,992
test cases (was 2,977).


Behaviour changes

There is no behaviour change that a v0.2.0 caller has to adapt to. Two things are new at
startup and in the permission vocabulary:

  • A node declares four network record collections when it starts, and refuses to start
    if it cannot (zetta: cannot initialize network record collections: ...). The
    declarations are idempotent and byte-identical on every node.
  • Two permission atoms are added, network:read and network:write. The canonical
    vocabulary has 34 atoms (was 32). No built-in role is changed.

The cluster's inter-node protocol is unchanged: the handshake order, wire bytes, frame
bound and plaintext mode are the same as in v0.2.0 (#1284). A cluster with nodes of both
versions was not tested.


Added

Workspace network records — #1281

The records people inspect about a workspace's network live in that workspace's Zetta;
Lake's Postgres keeps only the desired state the operator enforces.

Record type Collection Retention
connection network_connections 365 days
grant network_grants 365 days
tunnel_session network_tunnel_sessions 90 days
activity network_activity 30 days
  • write_network_record (REST and MCP) takes the producer's record as it is: Lake's
    connection and grant lifecycle snapshots, and zetta-tunnel's relay session, refusal and
    egress, peer, ingress and mesh decision records. It waits for the owner's fsync and for
    every reachable live peer to install the write, and needs network:write.
  • list_network_records reads a type newest first, or by one of its indexes (event,
    workspace, ids, role, decision, rule), with a page cursor. It needs network:read.
  • A tunnel session's start and end share one key, so the end replaces the live row with
    its final byte totals. Other records are keyed by the SHA-256 of their canonical JSON,
    so an identical retry is idempotent.
  • The JSON format is zetta.NetworkRecord version 1, published as
    schemas/network_record.schema.json. Unknown fields and malformed identifiers are
    refused before storage.
  • Neither permission grants the generic record API.

See docs/network/network-records.md.

Scoped app credentials — #1280

An app reaching a workspace Zetta no longer inherits workspace-wide access.

  • issue_app_credential turns a connection's credential_scope (read or write;
    a database, tables, topic prefixes) into a connection-owned user and least-privilege
    role, and returns a bearer credential once. Only its salted hash is stored. A second
    live credential for the same connection is a conflict. It needs tokens:manage.
  • revoke_app_credential revokes every token of that connection and removes its role
    assignment. Repeating it answers revoked: 0, so an operator can reconcile absence.
  • read grants SQL reads on only the named database or tables, and topic reads and
    receives on only the named prefixes. write adds SQL writes and publish, update and
    delete-event rights on the same resources. Neither grants DDL, user or token
    administration, topic creation or deletion, or network records.
  • Every use goes through the existing registry checks and audit trail; revocation fails
    authentication immediately.

See docs/network/scoped-app-access.md.

Zetta peer links through peer gateways — #1282

Zetta nodes in separate private networks form one cluster through zetta-tunnel peer
gateways. A node points --peer at its local import listener and runs --dial-out-only,
so it advertises no private address that could collide with another cluster's pod range.
Zetta's own mutual TLS 1.3 stays end to end: the gateways copy bytes and never hold the
join token.

PeerGateway.OverlappingPrivateNodesReplicateThroughOpaqueGateways proves it with two
nodes on the same modelled private address and two opaque forwarders: one cluster forms,
and CRDT writes on either node converge. See
docs/network/peer-gateway-links.md.

The transport library — #1283

zetta::transport (zetta_transport) is the cluster's mutual-TLS and framing code as a
library of its own:

  • PEM credential loading, and checks before a connection starts: the key matches the
    leaf, the leaf verifies against the CA bundle now, and no system trust anchor is added;
  • a Corosio context that is TLS 1.3 only and verifies the peer in both roles;
  • FramedStream, newline-framed reads and writes with a frame bound.

-DZETTA_TRANSPORT_ONLY=ON configures only the library and its direct dependencies (Capy,
Corosio, OpenSSL), static or shared, so zetta-tunnel can add_subdirectory it. Zetta's
cluster uses the same target. See
docs/cluster/transport-library.md.

The version, held by a test

project(VERSION) in CMakeLists.txt now says the release version (it said 0.0.1, a
known issue of v0.2.0). tests/test_release.cpp reads kVersion from
include/zetta/version.hpp and checks that it equals the CMake project version, the
committed schemas/openapi.json info.version, and that this page exists and is listed
in the index.


Fixed

A test crashed on a race its fixture created — #1292

RestClusterFixture.RunReportAnswersForADatasetlessRun ended in a segmentation fault or an
abort now and then. Under AddressSanitizer it was a heap-use-after-free: the churn run's
thread appended to the Run's log while the test thread serialized the Run. The fixture
never ticked its node, and a node that no loop has ticked runs owner work inline on the
calling thread. The fixture now ticks once after attaching its cluster, as both apps do
from their first loop iteration (#1293). Test-only: production nodes were not
affected.
Under ASan with 8 parallel copies it crashed once in 80 runs before, and
0 times in 200 after.

A documentation drift check skipped silently

CliDocs.EveryFlagMainOffersIsDocumented found src/main.cpp only through an environment
variable or the working directory, and under ctest the working directory is the build
tree, so it skipped. It now falls back to the source directory the tests were built from.


Documentation


Not included

  • zetta-tunnel itself: its executable, gateway authorization, gateway certificates and
    readiness live in zetta-tunnel. This
    release gives it the library, the record API and the peer contract.
  • Projecting the app credential to the app and reconciling connections: the operator
    and Lake (orama-bridge#186).
  • Moving Lake's users and tokens into workspace Zettas
    (#1172).
  • Clustered pub/sub (#1038).
  • No GPU build, real-checkpoint or external-engine test was run, and the Linux test suite
    was not run (see Verification).

Where it has run

Before this tag, the post-v0.2.0 tree has not run on a cluster. On 2026-10-17 every
Zetta on the development cluster (orama-production-cluster-01: the Lake bus
lake/lake-prod-zetta-zetta-0 and four workspace Zetta pods) runs
oramasearch/zetta:main-4015ef76, which is v0.2.0. The new capabilities were exercised
by the test suite only, in-process, with real Corosio TCP and mutual TLS between nodes.

The release image is oramasearch/zetta:v0.3.0 (the same digest as main-d7551c75,
sha256:0dad6a7fba1842a1406a3430394fd98a06d3241d9989a69470f6ad921a54328a). Rolling it to the cluster is a separate step, through the
operator's default Zetta image, and needs Steve's go-ahead.


Verification

Full test suite, run serially (ctest --test-dir build, no -j), macOS 14 with 16
hardware threads, Release builds, while other builds kept the host's load average between
5 and 30. Run 3 is the release commit; runs 1, 2 and the shared run are the same tree
without the fixture fix (#1293) and, for run 1, without the CliDocs fix.

Run Build Tests Passed Failed Skipped Disabled Time
1 static 2,888 2,870 1 17 1 880 s
2 static 2,888 2,870 2 16 1 775 s
shared -DBUILD_SHARED_LIBS=ON 2,888 2,871 1 16 1 861 s
3 (release) static 2,888 2,872 0 16 1 762 s

The failures in the earlier runs:

  • RestClusterFixture.RunReportAnswersForADatasetlessRun, a segmentation fault (run 2) and
    an abort (shared): the fixture race fixed in this release (#1292, #1293). After the fix
    it passed 200 of 200 (8 parallel copies, 25 repeats each) and run 3.
  • ClusterSecurity.MeshConvergesOverMutualTlsUnderSustainedLoad (run 1): documents did not
    converge within the deadline under load
    (#959,
    #1256). Passed 3 of 3 alone, and in
    the other runs.
  • ObjectRoutes.RangedReadsStayInsideTheirBudgets (run 2): a 64 KiB ranged GET p99 of
    7.19 ms against a 2 ms budget under load
    (#1256). Passed 3 of 3 alone, and in
    the other runs.

Skipped: 16 tests need a real checkpoint, a GPU, a generated corpus or an external
OramaCore engine that this host does not configure, and one benchmark is disabled. Run 1
also skipped CliDocs.EveryFlagMainOffersIsDocumented, fixed in this release (see Fixed).

The new version test: ReleaseVersion.MatchesTheCMakeProjectVersion,
ReleaseVersion.MatchesTheCommittedOpenApiDocument and ReleaseVersion.IsDocumented
pass.

A running node reports 0.3.0: a headless node built from this branch logs
zetta 0.3.0: node started, answers "version": "0.3.0" on GET /v1/server_info, and
serverInfo.version 0.3.0 to an MCP initialize. zetta --help prints
zetta 0.3.0 — a database node; there is no --version flag.

Linux: the release image is built from the tag on the cluster BuildKit (linux/amd64,
Ubuntu 24.04, Clang 19, zetta target only, no tests). It built from a detached checkout
of the tag, with the one compiler warning above, and is oramasearch/zetta:main-d7551c75,
also tagged oramasearch/zetta:v0.3.0, both naming sha256:0dad6a7fba1842a1406a3430394fd98a06d3241d9989a69470f6ad921a54328a. The
Docker Hub repository still answers 404 to an anonymous request. The Linux test suite was
not run: the native amd64 build box did not answer on 2026-10-17.

The build has the same two warnings as v0.2.0: a discarded io_result at
src/core/ml/forecast_execution.cpp:99, and a duplicate-library note from the linker.


Known issues

Security first. These are open in the tracker; none is new in this release.

  • #819, #793
    — some malformed SQL (a malformed table reference, REVOKE TOKEN, a scoped
    GRANT ON table(cols), SELECT current_user()) was reported to throw an uncaught
    ParseError that ends the node: a denial of service by anyone who can send SQL. On the
    macOS build of this release, SELECT * FROM system.*, REVOKE TOKEN abc,
    GRANT sql:read ON t(a,b) TO x and SELECT current_user() over /v1/sql each answer
    engine_error and the node stays up (checked 2026-10-17). #793 names a toolchain
    miscompile as the cause, and the Linux image (Clang 19) was not checked, so both stay
    open.
  • #760 — column masking can be
    bypassed with an aggregate (SELECT id, MAX(salary) ... GROUP BY id).
  • #786, #835
    — mutual-TLS gossip can stall because the read and write pumps share one OpenSSL
    stream, and a node can silently stop applying inbound sync. The transport moved into a
    library in this release without changing that behaviour.
  • #762 — authorization is off by
    default: a network-facing node without --require-auth accepts unauthenticated
    requests.
  • #763, #764
    — a joining node can create a second owner account, and replicated identity records are
    trusted with no proof of origin. Any node holding the join token is trusted.
  • #761, #765, #766, #767
    — role management is unbounded, a row policy with no condition allows all rows, any
    authenticated user can read the policy, user and role catalogue, and blob permissions
    match by text prefix rather than path.
  • #759, #830
    — still open in the tracker, although v0.2.0 checks every data-changing statement's
    right (#1229) and seals saved
    connector credentials (#1230).
    Re-check and close them.
  • #769, #770, #771, #669
    — lower-priority hardening: the admin token is not hashed, the join token in an
    environment variable is exposed widely, a peer's advertised addresses are not capped,
    and there is no certificate enrollment, pinning or revocation.
  • #1276 — run can fail with
    engine_error (key 'system.cpu.percent' not found), or end the process on
    system.throughput.per_second, when a system metric was not sampled. Seen within tens of
    milliseconds of node start. Retry the run.
  • Tests that fail under a loaded host and pass alone, two of which failed in this
    release's earlier runs (see Verification): #1288 (near-data featurize under
    ctest -j 6), #1256 and
    #959 (mesh and store latency
    benchmarks), #1248,
    #1124,
    #964,
    #963.
  • #999 — no reachable consent
    approval surface, so consent enforcement stays off by default.
  • Still open from v0.2.0: #1001
    (Snowflake is PAT/OAuth only), #977
    (a fresh node has no agent loaded), #1272,
    #1255,
    #1203,
    #1174, pub/sub is one node
    (#1038,
    #1037,
    #1036), and the source connectors are
    Postgres and Snowflake only (#1014).

Upgrading

  1. No migration. A v0.2.0 data directory starts as is; the node adds the four
    network record collections on first start.
  2. Grant network:write to zetta-tunnel and network:read to Lake's service role
    (each with mcp:call) before they write or read network records.
  3. Grant tokens:manage to whoever issues app credentials.
  4. Upgrade every node of a cluster together, as for v0.2.0: mixed versions were not
    tested.