Repository navigation
Release v0.4.0
v0.4.0
Released: 2026-10-19 · Branch: dev · Previous: v0.3.0 (2026-10-17)
v0.3.0 was Zetta's part of secure networking. v0.4.0 is about datasets: they keep the
column order of the file they came from and report typed columns, LIMIT is pushed to the
partitions instead of cutting a full scan, text columns become categorical features, the
dataset contract is typed from the partition catalog, and a dataset can be loaded from a
stored object up to 1 GiB instead of 16 MiB inline.
Four pull requests since v0.3.0: three features (#1299, #1297, #1303) and the v0.3.0 image
digest in the release notes (#1295). The generated OpenAPI has 177 /v1 paths (was 176),
the capability registry has 168 capabilities (was 167), and the test sources hold 3,042 test
cases (was 2,992).
Behaviour changes
get_datasetanswerscolumnsas typed objects,[{name, type, nullable, null_fraction}],
where it answered a list of names (#1299).list_datasetsstill answers names, so its callers
are unchanged. A caller readingget_dataset'scolumnsas strings must read.name. Lake's
dataset page reads the new shape from oramasearch/lake#834 (lake
v0.3.0).- Columns are listed in the source's order, not alphabetically, in
get_dataset,
list_datasets,system.datasetsandSELECT *: the CSV header, the Parquet schema, the order a
JSON or JSON Lines source first writes each key, a connector query'sSELECTlist, or a
CREATE TABLEcolumn list (#1299). A dataset loaded before this release keeps its
alphabetical order and sayscolumn_order: "first_seen"; loading the file again gives the
source order. Nothing restores the old order for new loads. - Text columns with 2 to 32 levels become one-hot categorical features of the M-1 GBDT
(#1297). A model trained on the same data can now use columns it used to drop with
not numeric; categorical columns are not supported yet, so its features and scores differ. The
model stores the fitted levels, andscore,explainandPOST /v1/models/{name}/predicttake
the original text column: a row missing a categorical column, or giving it a number, is
refused with the column named. An unseen level or a null encodes as all zeros. Models trained
before this release are unchanged. - The dataset contract is typed from the partition catalog where every row is in a partition
that recorded its schema (#1297). The decisions match the 256-row sample in every case but one:
a column the catalog types numeric whose first 256 rows are all null now becomes a feature. upload_dataset's refusal of a body over 16 MiB now namesload_dataset_object(#1303).- A column the caller may not read answers
type,nullableandnull_fractionasnullin
get_dataset, andORDER BYon it is refused, asWHEREandGROUP BYalready were (#1299).
A v0.3.0 node and a v0.4.0 node answer the same SQL with the same rows: a v0.3.0 responder
refuses the new keep and order_by fields of sql.scan, and a v0.4.0 coordinator asks it again
without them (#1299). A whole mixed-version cluster was not tested; upgrade every node together.
Added
load_dataset_object: a dataset from a stored object — #1303
Loads a CSV or JSON Lines object from a bucket as a dataset, past upload_dataset's 16 MiB inline
cap, for Lake's chunked upload (#1302; oramasearch/lake#846). By
@thatjuan.
- A Job using the
add_datasetoperation. The caller also needsobjects:readon the bucket. It
answers arun_idat once; poll it withget_run. - Refusals come before a run is created, in this order:
invalid_input,unauthorized,
not_found(/bucket),conflict(/table_name),engine_error. - The object is read in 4 MiB slices on a worker and handed to the owner in 10,000-row batches.
The table is partition-only: rows live in partition bodies on their owners, andSELECT
gathers across them. A failed load drops its table;remove_objectruns after either outcome. upload_dataset,add_dataset_fileandload_dataset_objectdecode through one line decoder,
so they refuse in the same words.- Limits: 1 GiB per object (
kMaxDatasetObjectBytes), 16 MiB per line. - The source's column order is kept on this path too: the CSV header, or the order the JSON
Lines rows first write each key (ObjectLoad.KeepsTheSourceColumnOrder, added when #1303 was
rebased onto #1299).
Measured by the PR on a 67 MiB CSV (1,405,770 rows, one node, over REST): 41 s, 1.7 GB of node
memory added, a longest owner-loop stall of 2.7 s, against 126 s, 5.8 GB and 8.5 s when the rows
were also copied into the replicated document. See
docs/data/add-dataset.md.
LIMIT pushed to the partitions; OFFSET and FETCH — #1299
FETCH {FIRST|NEXT} [n] {ROW|ROWS} ONLYisLIMIT n;OFFSET mtakesROW/ROWS.TOP nis
still refused.- Over a partition-only table, a non-aggregating
SELECTwith aLIMITsendsoffset + limitand
itsORDER BYkeys to each owner, which returns at most that many rows per partition. The
coordinator still sorts, skips and cuts, so the answer equals the full scan's. - Over a replicated table without
ORDER BY, rows pastoffset + limitare not copied out. An
aggregate is never cut.
See docs/data/query.md and
docs/data/scatter-gather-select.md.
Source column order and typed columns — #1299 (#1021)
- The order is recorded at load time under a new CRDT key
cols:<table-id>, kept intable.json,
migrated bymigrate_to_replicationand removed bydrop_table. - Every summary carries
column_order: "source" | "first_seen". typeis the JSON family (integer,number,string,boolean,object,array,null,
mixed,unknown). A partition-only table is profiled from thecolumn_statsthat
write_partitionednow records; a partition written before hasnull_fractionnull.
See docs/data/column-order.md.
Categorical features and catalog typing — #1297 (#1203)
- A text column is a categorical feature when every value is text, it has 2 to 32 levels, at most
half its values are distinct, and the values are not dates. Anything else stays excluded, and the
run log says why. - Levels are fitted on training rows only (after the
as_ofcutoff). One-hot reads no label, so
no target information leaks into a held-out fold. feature_schema.categorical_encodingholds the fitted levels;feature_schema.inputsis the
predict input contract by original column name.resolve_datasetreads only the columns it cannot type from the catalog, projected with
SELECT <columns>, and says in its first note where the types came from.
Measured by the PR on the 15,000-row demo table: resolve_dataset went from 86–88 ms to
34–39 ms; held-out AUC on the demo is unchanged at 0.731. On the getting-started churn data the
categories lift held-out AUC to 0.845, from 0.464 with the numeric column alone. See
docs/data/categorical-features.md and
docs/data/dataset-path.md.
Documentation
- New:
docs/data/column-order.md,
docs/data/categorical-features.md. - Updated:
add-dataset.md,object-store.md,query.md,scatter-gather-select.md,
system-datasets.md,dataset-path.md,partition-manifest.md,partitioned-write-path.md,
the API pages and the code map.schemas/openapi.json,
schemas/partition_scan_request.schema.jsonandschemas/predict_request.schema.jsonare
regenerated. v0.3.0records its image digest (#1295).
Not included
- Other file formats from an object: Excel (#1307), Parquet one row group at a time (#1306),
a JSON array (#1305), gzip and other delimiters (#1304). LIMITon the dataset-typing read:resolve_datasetstill scans the table and cuts the
typing sample to 256 rows in memory; appendingLIMITto that read is a one-line follow-up now
that #1299 pushes it down.- Class imbalance warnings and weighting (#1298).
- No GPU build, real-checkpoint or external-engine test was run, and the Linux test suite was not
run (see Verification).
Where it has run
This release has not run on a cluster. v0.3.0 (main-d7551c75) was never rolled either: on
2026-10-19 every Zetta on the development cluster still runs v0.2.0. The new capabilities were
exercised by the test suite only, and #1303's 67 MiB load by its author on one local node.
The release image is oramasearch/zetta:v0.4.0, the same digest as main-104e8a98
(sha256:73e613748c1d6114a31fd8c462b55a09109de1b7c73323042429cb8a8509e1a9), built from a detached checkout of the tag on the cluster BuildKit (linux/amd64, Ubuntu
24.04, Clang 19, zetta target only, no tests) with the same compiler warning as on macOS. The Docker
Hub repository answers 404 to an anonymous request. Rolling it is a separate step,
through the operator's default Zetta image, and needs Steve's go-ahead.
Verification
Full test suite, run serially (ctest --test-dir build, no -j) on the release branch,
macOS 14 arm64, Homebrew LLVM Clang, static Release build, 806 s, load average 6 to 21 from other
work on the host:
2,938 tests · 2,921 passed · 0 failed · 16 skipped · 1 disabled
No test was rerun. Skipped: 16 tests need a real model checkpoint, a GPU, a generated corpus
or dataset, or an external OramaCore engine that this host does not configure; the pub/sub
throughput benchmark is disabled.
Before the release, the same tree without the version change ran the suite in parallel
(ctest -j, after #1303 was rebased): 2,930 of 2,938 passed, and the 8 that failed under that load
passed when rerun serially: RestServerFixture.UploadDatasetOverRestAcceptsALargeBody, three
ObjectRoutes latency budgets, ProjectGraph.ReplicatesToPeer, Testkit.DistributedCasesRunOnARealCluster
and two ClusterRouting membership tests (the loaded-host issues under Known issues).
The version test: ReleaseVersion.MatchesTheCMakeProjectVersion,
ReleaseVersion.MatchesTheCommittedOpenApiDocument and ReleaseVersion.IsDocumented pass.
A running node reports 0.4.0: zetta --help prints zetta 0.4.0 — a database node, and a
headless node built from this branch logs zetta 0.4.0: node started.
Build: the same two warnings as v0.3.0, a discarded io_result at
src/core/ml/forecast_execution.cpp:99 and a duplicate-library note from the linker.
Not run: the shared-library build, the Linux test suite (#1296), and any GPU,
real-checkpoint or external-engine test.
Known issues
Security first. These are open in the tracker.
- #1300 (new, found reviewing #1299, older
than it) —analyze_datasetdescribes a column the caller may not read (its null rate, kind and
summary). #1299 closed the same gap inget_dataset. - #819, #793
— some malformed SQL was reported to end the node. On the macOS build ofv0.3.0each case
answeredengine_error; the Linux image was not checked (#1296). - #760 — column masking can be bypassed with
an aggregate. - #786, #835
— mutual-TLS gossip can stall, and a node can silently stop applying inbound sync. - #762 — authorization is off by default.
- #763, #764
— a joining node can create a second owner account, and replicated identity records are trusted
with no proof of origin. - #761, #765, #766, #767
— role management is unbounded, a row policy with no condition allows all rows, any
authenticated user can read the policy catalogue, and blob permissions match by text prefix. - #759, #830
— still open althoughv0.2.0addressed them; to be checked and closed (#1296). - #769, #770, #771, #669
— lower-priority hardening (admin token, join token, advertised addresses, certificates). - #1301 (new) — a column that is boolean in
some partitions and integer in others isintegerfor training butmixedinget_dataset;
and a one-hot feature name (a=b=c) can collide with a column nameda=b. - #1276 —
runcan fail when a system
metric was not sampled, within milliseconds of node start. Retry therun. - Tests that fail under a loaded host and pass alone (see Verification):
#1256,
#1248,
#1124,
#1288,
#959,
#964,
#963. - #1296 —
v0.3.0follow-ups: the Linux
test suite and the two security re-checks above. - Still open from earlier releases: #999
(consent enforcement stays off), #1001,
#977,
#1272,
#1255,
#1174, pub/sub is one node
(#1038,
#1037,
#1036), and the source connectors are
Postgres and Snowflake only (#1014).
Upgrading
- No migration. A
v0.3.0data directory starts as is. Datasets already loaded keep their
alphabetical column order until they are loaded again. - Read
get_dataset'scolumnsas objects (.name), and send categorical columns by their
original name and text value topredict,scoreandexplainfor models trained on this
release. - Lake pairs with this release from lake
v0.3.0: its dataset page reads the typed columns and
asks for its first rows withLIMIT. Lake does not callload_dataset_objectyet
(oramasearch/lake#846). - Upgrade every node of a cluster together: mixed versions were not tested.