Skip to content

Release v0.4.0

Choose a tag to compare

@homer6 homer6 released this 20 Oct 02:35
· 230 commits to main since this release
104e8a9

v0.4.0

Released: 2026-10-19 · Branch: dev · Previous: v0.3.0 (2026-10-17)

v0.3.0 was Zetta's part of secure networking. v0.4.0 is about datasets: they keep the
column order of the file they came from and report typed columns, LIMIT is pushed to the
partitions instead of cutting a full scan, text columns become categorical features, the
dataset contract is typed from the partition catalog, and a dataset can be loaded from a
stored object up to 1 GiB instead of 16 MiB inline.

Four pull requests since v0.3.0: three features (#1299, #1297, #1303) and the v0.3.0 image
digest in the release notes (#1295). The generated OpenAPI has 177 /v1 paths (was 176),
the capability registry has 168 capabilities (was 167), and the test sources hold 3,042 test
cases (was 2,992).


Behaviour changes

  • get_dataset answers columns as typed objects, [{name, type, nullable, null_fraction}],
    where it answered a list of names (#1299). list_datasets still answers names, so its callers
    are unchanged. A caller reading get_dataset's columns as strings must read .name. Lake's
    dataset page reads the new shape from oramasearch/lake#834 (lake
    v0.3.0).
  • Columns are listed in the source's order, not alphabetically, in get_dataset,
    list_datasets, system.datasets and SELECT *: the CSV header, the Parquet schema, the order a
    JSON or JSON Lines source first writes each key, a connector query's SELECT list, or a
    CREATE TABLE column list (#1299). A dataset loaded before this release keeps its
    alphabetical order
    and says column_order: "first_seen"; loading the file again gives the
    source order. Nothing restores the old order for new loads.
  • Text columns with 2 to 32 levels become one-hot categorical features of the M-1 GBDT
    (#1297). A model trained on the same data can now use columns it used to drop with
    not numeric; categorical columns are not supported yet, so its features and scores differ. The
    model stores the fitted levels, and score, explain and POST /v1/models/{name}/predict take
    the original text column: a row missing a categorical column, or giving it a number, is
    refused
    with the column named. An unseen level or a null encodes as all zeros. Models trained
    before this release are unchanged.
  • The dataset contract is typed from the partition catalog where every row is in a partition
    that recorded its schema (#1297). The decisions match the 256-row sample in every case but one:
    a column the catalog types numeric whose first 256 rows are all null now becomes a feature.
  • upload_dataset's refusal of a body over 16 MiB now names load_dataset_object (#1303).
  • A column the caller may not read answers type, nullable and null_fraction as null in
    get_dataset, and ORDER BY on it is refused, as WHERE and GROUP BY already were (#1299).

A v0.3.0 node and a v0.4.0 node answer the same SQL with the same rows: a v0.3.0 responder
refuses the new keep and order_by fields of sql.scan, and a v0.4.0 coordinator asks it again
without them (#1299). A whole mixed-version cluster was not tested; upgrade every node together.


Added

load_dataset_object: a dataset from a stored object — #1303

Loads a CSV or JSON Lines object from a bucket as a dataset, past upload_dataset's 16 MiB inline
cap, for Lake's chunked upload (#1302; oramasearch/lake#846). By
@thatjuan.

  • A Job using the add_dataset operation. The caller also needs objects:read on the bucket. It
    answers a run_id at once; poll it with get_run.
  • Refusals come before a run is created, in this order: invalid_input, unauthorized,
    not_found (/bucket), conflict (/table_name), engine_error.
  • The object is read in 4 MiB slices on a worker and handed to the owner in 10,000-row batches.
    The table is partition-only: rows live in partition bodies on their owners, and SELECT
    gathers across them. A failed load drops its table; remove_object runs after either outcome.
  • upload_dataset, add_dataset_file and load_dataset_object decode through one line decoder,
    so they refuse in the same words.
  • Limits: 1 GiB per object (kMaxDatasetObjectBytes), 16 MiB per line.
  • The source's column order is kept on this path too: the CSV header, or the order the JSON
    Lines rows first write each key (ObjectLoad.KeepsTheSourceColumnOrder, added when #1303 was
    rebased onto #1299).

Measured by the PR on a 67 MiB CSV (1,405,770 rows, one node, over REST): 41 s, 1.7 GB of node
memory added, a longest owner-loop stall of 2.7 s, against 126 s, 5.8 GB and 8.5 s when the rows
were also copied into the replicated document. See
docs/data/add-dataset.md.

LIMIT pushed to the partitions; OFFSET and FETCH — #1299

  • FETCH {FIRST|NEXT} [n] {ROW|ROWS} ONLY is LIMIT n; OFFSET m takes ROW/ROWS. TOP n is
    still refused.
  • Over a partition-only table, a non-aggregating SELECT with a LIMIT sends offset + limit and
    its ORDER BY keys to each owner, which returns at most that many rows per partition. The
    coordinator still sorts, skips and cuts, so the answer equals the full scan's.
  • Over a replicated table without ORDER BY, rows past offset + limit are not copied out. An
    aggregate is never cut.

See docs/data/query.md and
docs/data/scatter-gather-select.md.

Source column order and typed columns — #1299 (#1021)

  • The order is recorded at load time under a new CRDT key cols:<table-id>, kept in table.json,
    migrated by migrate_to_replication and removed by drop_table.
  • Every summary carries column_order: "source" | "first_seen".
  • type is the JSON family (integer, number, string, boolean, object, array, null,
    mixed, unknown). A partition-only table is profiled from the column_stats that
    write_partitioned now records; a partition written before has null_fraction null.

See docs/data/column-order.md.

Categorical features and catalog typing — #1297 (#1203)

  • A text column is a categorical feature when every value is text, it has 2 to 32 levels, at most
    half its values are distinct, and the values are not dates. Anything else stays excluded, and the
    run log says why.
  • Levels are fitted on training rows only (after the as_of cutoff). One-hot reads no label, so
    no target information leaks into a held-out fold.
  • feature_schema.categorical_encoding holds the fitted levels; feature_schema.inputs is the
    predict input contract by original column name.
  • resolve_dataset reads only the columns it cannot type from the catalog, projected with
    SELECT <columns>, and says in its first note where the types came from.

Measured by the PR on the 15,000-row demo table: resolve_dataset went from 86–88 ms to
34–39 ms; held-out AUC on the demo is unchanged at 0.731. On the getting-started churn data the
categories lift held-out AUC to 0.845, from 0.464 with the numeric column alone. See
docs/data/categorical-features.md and
docs/data/dataset-path.md.


Documentation

  • New: docs/data/column-order.md,
    docs/data/categorical-features.md.
  • Updated: add-dataset.md, object-store.md, query.md, scatter-gather-select.md,
    system-datasets.md, dataset-path.md, partition-manifest.md, partitioned-write-path.md,
    the API pages and the code map. schemas/openapi.json,
    schemas/partition_scan_request.schema.json and schemas/predict_request.schema.json are
    regenerated.
  • v0.3.0 records its image digest (#1295).

Not included

  • Other file formats from an object: Excel (#1307), Parquet one row group at a time (#1306),
    a JSON array (#1305), gzip and other delimiters (#1304).
  • LIMIT on the dataset-typing read: resolve_dataset still scans the table and cuts the
    typing sample to 256 rows in memory; appending LIMIT to that read is a one-line follow-up now
    that #1299 pushes it down.
  • Class imbalance warnings and weighting (#1298).
  • No GPU build, real-checkpoint or external-engine test was run, and the Linux test suite was not
    run (see Verification).

Where it has run

This release has not run on a cluster. v0.3.0 (main-d7551c75) was never rolled either: on
2026-10-19 every Zetta on the development cluster still runs v0.2.0. The new capabilities were
exercised by the test suite only, and #1303's 67 MiB load by its author on one local node.

The release image is oramasearch/zetta:v0.4.0, the same digest as main-104e8a98
(sha256:73e613748c1d6114a31fd8c462b55a09109de1b7c73323042429cb8a8509e1a9), built from a detached checkout of the tag on the cluster BuildKit (linux/amd64, Ubuntu
24.04, Clang 19, zetta target only, no tests) with the same compiler warning as on macOS. The Docker
Hub repository answers 404 to an anonymous request. Rolling it is a separate step,
through the operator's default Zetta image, and needs Steve's go-ahead.


Verification

Full test suite, run serially (ctest --test-dir build, no -j) on the release branch,
macOS 14 arm64, Homebrew LLVM Clang, static Release build, 806 s, load average 6 to 21 from other
work on the host:

2,938 tests · 2,921 passed · 0 failed · 16 skipped · 1 disabled

No test was rerun. Skipped: 16 tests need a real model checkpoint, a GPU, a generated corpus
or dataset, or an external OramaCore engine that this host does not configure; the pub/sub
throughput benchmark is disabled.

Before the release, the same tree without the version change ran the suite in parallel
(ctest -j, after #1303 was rebased): 2,930 of 2,938 passed, and the 8 that failed under that load
passed when rerun serially: RestServerFixture.UploadDatasetOverRestAcceptsALargeBody, three
ObjectRoutes latency budgets, ProjectGraph.ReplicatesToPeer, Testkit.DistributedCasesRunOnARealCluster
and two ClusterRouting membership tests (the loaded-host issues under Known issues).

The version test: ReleaseVersion.MatchesTheCMakeProjectVersion,
ReleaseVersion.MatchesTheCommittedOpenApiDocument and ReleaseVersion.IsDocumented pass.

A running node reports 0.4.0: zetta --help prints zetta 0.4.0 — a database node, and a
headless node built from this branch logs zetta 0.4.0: node started.

Build: the same two warnings as v0.3.0, a discarded io_result at
src/core/ml/forecast_execution.cpp:99 and a duplicate-library note from the linker.

Not run: the shared-library build, the Linux test suite (#1296), and any GPU,
real-checkpoint or external-engine test.


Known issues

Security first. These are open in the tracker.

  • #1300 (new, found reviewing #1299, older
    than it) — analyze_dataset describes a column the caller may not read (its null rate, kind and
    summary). #1299 closed the same gap in get_dataset.
  • #819, #793
    — some malformed SQL was reported to end the node. On the macOS build of v0.3.0 each case
    answered engine_error; the Linux image was not checked (#1296).
  • #760 — column masking can be bypassed with
    an aggregate.
  • #786, #835
    — mutual-TLS gossip can stall, and a node can silently stop applying inbound sync.
  • #762 — authorization is off by default.
  • #763, #764
    — a joining node can create a second owner account, and replicated identity records are trusted
    with no proof of origin.
  • #761, #765, #766, #767
    — role management is unbounded, a row policy with no condition allows all rows, any
    authenticated user can read the policy catalogue, and blob permissions match by text prefix.
  • #759, #830
    — still open although v0.2.0 addressed them; to be checked and closed (#1296).
  • #769, #770, #771, #669
    — lower-priority hardening (admin token, join token, advertised addresses, certificates).
  • #1301 (new) — a column that is boolean in
    some partitions and integer in others is integer for training but mixed in get_dataset;
    and a one-hot feature name (a=b=c) can collide with a column named a=b.
  • #1276 — run can fail when a system
    metric was not sampled, within milliseconds of node start. Retry the run.
  • Tests that fail under a loaded host and pass alone (see Verification):
    #1256,
    #1248,
    #1124,
    #1288,
    #959,
    #964,
    #963.
  • #1296 — v0.3.0 follow-ups: the Linux
    test suite and the two security re-checks above.
  • Still open from earlier releases: #999
    (consent enforcement stays off), #1001,
    #977,
    #1272,
    #1255,
    #1174, pub/sub is one node
    (#1038,
    #1037,
    #1036), and the source connectors are
    Postgres and Snowflake only (#1014).

Upgrading

  1. No migration. A v0.3.0 data directory starts as is. Datasets already loaded keep their
    alphabetical column order until they are loaded again.
  2. Read get_dataset's columns as objects (.name), and send categorical columns by their
    original name and text value to predict, score and explain for models trained on this
    release.
  3. Lake pairs with this release from lake v0.3.0: its dataset page reads the typed columns and
    asks for its first rows with LIMIT. Lake does not call load_dataset_object yet
    (oramasearch/lake#846).
  4. Upgrade every node of a cluster together: mixed versions were not tested.