Repository navigation
Release v0.2.0
Released: 2026-10-12 · Branch: dev · Previous: v0.1.1 (2026-08-27)
v0.1.1 was one methodology (churn) on one node API. v0.2.0 is the engine a hosted
workspace runs on: 21 methodologies behind one registry, composed solutions that decide and
measure, AutoML studies, pub/sub inside the node, a keyed record store and an object store
for workspace data, agents described by records, and an audit trail that no longer lives in
the replicated catalogue. Every change here was already running on the development cluster
as oramasearch/zetta:dev-2cfdac21 before this release was cut.
236 commits across 63 pull requests, 750 files. The generated OpenAPI has 172 /v1
paths (was 78), and the test sources hold 2,977 test cases (was 2,119).
⚠️ Behaviour change: upgrade every node of a cluster together
Audit events moved out of the replicated catalogue into per-origin segments
(#1164). Mixed versions deliberately do not converge audit history. Stop the old
writers, upgrade every node, then restart the cluster.
- On first start the node migrates the audit events it finds in the old catalogue into its
own segments, under the reserved originlegacy. The migration is restartable, and a
failure leaves the old persisted catalogue in place. - An unreadable store or a failed migration prevents startup. That is deliberate: a node
that started anyway would drop history. - Valid legacy
audit/<uuid>.jsonfiles are folded in and removed; invalid ones stay where
they are and are reported.
There is no flag that restores the old layout. See
docs/governance/audit-trail.md, "Replication and upgrade
policy".
⚠️ Behaviour change: every SQL statement that changes data checks its right
Before this release only INSERT was authorized in the executor (#1229). Now:
| Statement | Right needed |
|---|---|
UPDATE, DELETE, TRUNCATE |
sql:write, at table or database scope |
CREATE / DROP DATABASE and TABLE |
sql:ddl, at table or database scope |
CREATE VIRTUAL TABLE |
flat sql:ddl |
CREATE USER, CREATE TOKEN |
users:manage, tokens:manage |
Column grants and row-level security apply to UPDATE and DELETE; TRUNCATE of a
policy-protected table is refused; a legacy AccessKey needs the flat right. Each such
statement lands on the audit trail as the new data_changed action.
A principal that could run these statements with read rights is now refused. Grant the
right; there is no setting that turns the check off.
⚠️ Behaviour change: a gate check that did not apply says not_applicable
A gate check with nothing to check on a graph used to answer pass, so a churn report read
as about 30 passes that verified nothing (#1240). CheckStatus gains not_applicable,
which never moves the verdict. The report's JSON now lists only the checks that applied,
with the counts applied and not_applicable. A caller that counted pass entries, or
expected every check in the list, sees fewer. Pass include_not_applicable to the gate
capability to get the full list back.
⚠️ Behaviour change: smaller ones that can break a caller
- A compiled pipeline is named from its intent (#1238), not
churn skeletonor
<methodology> pipeline. A second compile in the same project is numbered(2)instead
of failing. Passnametocompile_intentto keep a fixed name; the two delivery
runbooks now do. create_agentandupdate_agenttake a closeddefinition(#1220): unknown
properties are refused at the property. A file sent as{path, content}is refused
instead of being stored empty; the field iscontents.upload_datasetrefuses a taken name withconflict(409) unlessif_existsis
"rename", and the CSV decoder refuses ragged rows, unnamed or repeated header columns
and open quotes, naming the line, instead of padding or truncating (#1221).- A SQL column that no row carries is an error naming the column and the table's
columns (#1218). - A study whose trials all fail ends
failed, with the last trial's reason, instead of
completewith an empty reason (#1217). A positivecost_capunder 60 trial-seconds is
rejected at/budgets/cost_cap, because it could never admit a trial (#1211). - A request no route matches gets an answer (#1011):
405 method_not_allowedfor a
capability called with the wrong method,404 unknown_capabilityor404 not_found
otherwise. It used to get no bytes at all. delete_datasetrefuses while a pending or running run, or a running study, reads the
dataset (#1258), naming each one.
Added
The methodology registry: 21 methodologies, one way to compile and run them
compile_intent and run route through a registry instead of a churn-only path.
Methodologies M-1 to M-21 are registered and runnable: churn classification, forecasting,
segmentation, anomaly detection, entity resolution (distributed across members), survival,
next-best-action, causal uplift, LambdaMART ranking, language, document extraction, graph,
vision, speech, bandits, optimization, Bayesian uncertainty, out-of-fold stacking, label
efficiency, synthetic data with differentially private GBDT, and online streaming learning.
analyze_datasetandplan_featuresdescribe a dataset and plan its features. The plan
is enforced for all 21 methodologies, and a run reports the provenance of what it used.- Text entity keys and boolean columns are accepted, and the original entity ids come back
in every output. This closes thev0.1.1known issue
#957. - A workspace can register its own methodology (
register_methodology,
update_methodology,delete_methodology,record_methodology_run), run by Lake's
runner (#1261, lake#655).
Solutions: from a score to a decision and a readout
A composed solution registry (create_solution, run_solution) executes through governed
foundation stages. Around it: a replicated holdout experiment manager, a versioned decision
policy engine (create_policy, decide, approve_decision), a replicated audit ledger
(list_ledger), a drift and guardrail monitor (get_monitor), point-in-time snapshots
(snapshot), action delivery (enqueue_action_task, deliver_action_task,
record_outcome) and the ROI readout (get_roi_readout).
Runnable: S-1 churn, S-2 lead scoring, S-3 lifetime value, S-4 collections, S-5 demand
forecasting, S-6 replenishment, S-7 KPI anomaly, predictive maintenance, S-9 automated
outbound, S-10 fraud decisioning, S-11 next-best-action, S-12 claims; and the modules B-1
dynamic pricing, B-2 employee attrition, B-3 ticket triage, B-4 document workflows, B-5
supplier risk, B-6 healthcare operations, B-7 credit underwriting and B-8 security
operations.
Studies, run metrics and models
- Run-backed AutoML studies:
create_study,validate_study,list_trials,
promote_trial,stop_study,stop_trial,get_study_analysis,delete_study. Trials
run through the background training driver, under a cost cap in trial-seconds. - Experiments over a workspace methodology's parameters, with trials run by Lake's
runner (lake#689). - Durable run metrics:
log_run_metrics,log_run_params,log_run_artifact,
get_run_metrics,compare_runs. A GBDT run logs its held-out scores, confusion matrix,
ROC and calibration on the Run (#1239). - One metric name. The run report's key (
held_out_auc,held_out_log_loss,
held_out_accuracy,held_out_rmse,held_out_mae) is what a study objective names; the
old spellings (validation.auc,ROC-AUC,auc) are accepted as aliases. - Standalone models:
train_model,retrain_model,register_modeland
move_model_to_projectwork without a project
(lake#654). - Bring a model exactly: ONNX, XGBoost and LightGBM models are imported as they are, and
export_modelwrites one out (lake#681). - Large artifacts move in chunks: a methodology run's artifact over 24 MiB is appended
withappend_methodology_artifact, andread_model_weightsreads weights back in chunks
(lake#690).
Pub/sub inside the node
Stage 1 of the design recorded as decisions D5 to D39 under
docs/plans/pubsub/: typed addresses, an email-style envelope, ring
buffers with a disk tier, topics, subscriptions, inboxes, governance bounds, limits, and
best-effort survival across a restart.
create_topic,update_topic,drop_topic,list_topics,get_topic,browse_topic,
publish_event,receive_events,update_event,delete_events, over REST and MCP with
parity.- A long poll suspends without holding a thread, and the node keeps accepting connections
while polls are held. - Stage 1 is one node. Topic definitions do not replicate
(#1038).
A keyed record store, an object store, and POST sql
- Records (
put_record,get_record,list_records,delete_record,
create_collection) with conditional writes and secondary-index lists, and labelled time
series (create_metric,record_metric,query_metric) (#1231). - Records live in a Bitcask-style log per node under
<data_dir>/records/, not in the
replicated catalogue. The first version kept them in the catalogue: after 200,000 puts
with three live keys the saved document was 141 MiB and the process held 5.2 GB. The soak
that found it is now a test: 1,000,000 puts hold 76 to 120 MiB of log, 121 MiB RSS, and a
6.8 ms read p99 during saves (figures from the soak recorded in commit1b7a12b1). - Objects (
create_bucket,list_objects,delete_object,object_usage, leases and
object_barrier), the capability behind Lake's workspace liquid stores. A bucket's log
stays a prefix after a power loss, a barrier waits for running fsyncs, a failed fsync
fails the log, a corrupt marked log is refused, and each bucket reports prefix durability
and an incarnation that a write checks in the same step. POST /v1/sql, and in the Postgres subset:ORDER BY,LIMITandOFFSET;
count(*)and other aggregates with noGROUP BY;GROUP BYon any type;SELECT *
fillscolumnseven when no row matches.
Deletes and storage accounting
delete_project(#1013), anddelete_dataset,delete_model,delete_runand
delete_index(lake#645), which each
takedry_run.upload_datasetloads a CSV or JSONL file's text over the API (#1012). A dataset records
itssource_format.storage_usagereports disk by kind for a node and for a whole cluster
(lake#658);node_memoryreports memory
by owning subsystem.- Search returns the matched rows' fields with each hit.
Agents
AgentRecorddescribes every agent Lake shows.- Granted Pipelines are tools in native chat.
- An agent record may carry
mcp:<prefix>skills
(lake#652), and Lake's own areas are
agent skills (agent-runtime#7).
Fixed
Saved connector credentials were stored in the clear — #1230
They are now sealed at rest with AES-256-GCM under an HKDF-SHA256 key derived from the
cluster join token, and they are never in the replicated catalogue. Node start re-seals
legacy plaintext and old-key entries, and the secret file keeps retired join tokens so a
rotation does not strand them. No read surface returns a credential: system.connections
and the connection capabilities report only whether one is set.
The audit trail kept a node's memory high and grew while idle — #1164
138,000 audit: keys in the in-memory catalogue held RSS at 730 MB. Events now live outside
the catalogue, polling is summarized instead of logged per call, idle growth is bounded, an
administrator can purge (purge_audit, itself audited), the trail reconciles in linear
time, and index lookups stay bounded.
The API server could go silent or lose work
- A request no route matched got no response at all, and a throwing handler dropped the
connection or ended a headless node (#1011). A handler exception is now anengine_error
naming the capability, and invalid UTF-8 in a response body is replaced, not thrown on. - In-flight requests drain before shutdown; request frames are kept until owner work
completes; the HTTP worker stops when node startup is refused. - An offloaded call runs its gate on the owner thread, and the owner loop wakes for
run_on_ownerwork and drains it on every wake. - A signal handler is not replaced while a dispatch to it is still in flight.
Studies and training
- A 12-row trial ran for ten minutes and slowed workspace reads, because every boosting
round captured the whole Run and Study into the replicated document. Captures are now
written at most once per second: two 200-tree trials on 12 rows went from +2.6 MB and
135 ms per save to +150 KB and 5 ms (figures recorded in commitdc69a420, #1217). run metric logging failedwas reported for any stop, including the study's own wall-time
budget. The train job now records why it stopped.- An experiment's trial models are named from the experiment. A text entity-key mapping is
answered as a count in run and model reads.
Pipelines
- A run says what it read, what it could not use, and when it trained nothing. A run given
no dataset reads its pipeline's own table. - The churn index step indexes the landed table, and M-1 step names fit the dataset.
Replication
Replication caches values and reconciles only the keys that changed; solution records
reconcile incrementally.
Build
Beast2 links with Corosio's OpenSSL provider. On macOS the build links Homebrew's
libunwind with Homebrew's libc++abi: against the system unwinder a matching catch was
skipped and std::terminate() ran.
Documentation
AGENTS.mdfor agents working in this repository.- The pub/sub design, decision by decision, and the stage 1 spec as built.
CLAUDE.mdand the architecture docs cover pub/sub.orama-cloud-v1-operatorisorama-operatorthroughout.
Verification
Full test suite, run serially (ctest --test-dir build, no -j) on the release commit,
macOS with 16 hardware threads, 744 seconds:
2,873 tests · 2,854 passed · 2 failed · 17 skipped (real checkpoints, a GPU or an
external engine not configured) · 1 more disabled (a benchmark)
The two failures are one defect, not load
(#1276): run threw on a system metric
key that had not been sampled. RestClusterFixture.RunReportAnswersForADatasetlessRun
passed 4 of 5 times when rerun alone; Uc1Journey.JourneyUc1EndToEndWithActingKey passed
5 of 5. See Known issues.
The build has two warnings: a discarded io_result at
src/core/ml/forecast_execution.cpp:99, and a duplicate-library note from the linker.
Running in a cluster: this commit's image, oramasearch/zetta:dev-2cfdac21, is the
workspace Zetta on the development cluster (four pods), serving Lake at dd1574fd. The
image was built before the version string moved, so those nodes report 0.1.1.
Known issues
- #1276 —
runfails
intermittently withengine_error(key 'system.cpu.percent' not found), or ends the
process onsystem.throughput.per_second, when a system metric was not sampled for the
step. Seen within tens of milliseconds of node start. Retry therun. - Tests that fail under a loaded host and pass alone. The mutual-TLS mesh test and the
object and record store latency benchmarks
(#1256,
#959), the record store soak
(#1248), multi-member peer discovery
(#1124), and two distributed tests
(#964,
#963). - #786 — mutual-TLS gossip can stall,
because the read pump and the write pump share one OpenSSL stream. Still open from
v0.1.1. - #999 — there is still no reachable
consent approval surface, so consent enforcement stays off by default. Still open from
v0.1.1. - #1001 — the Snowflake connector is
PAT/OAuth only. Still open fromv0.1.1. - #977 — a fresh node has no agent
loaded. Still open fromv0.1.1. - #1272 —
delete_datasetcannot
see the tables read by a run that was created outside theruncapability, so it does not
refuse for those. - #1255 — a text entity-key mapping
is stored inline in the replicated Run and Model records, which grows them with the number
of entities. - #1203 —
dataset_pathdoes not
encode categorical features, and does not type columns from the partition catalogue. - #1174 —
receive_eventson an
exposed topic does not support a 60-second poll. - Pub/sub is one node (#1038), has
no streaming subscriptions (#1037),
and has no per-role delivery bounds
(#1036). - #1014 — the source connectors are
still Postgres and Snowflake only. project(... VERSION ...)inCMakeLists.txtstill says0.0.1. Nothing reads it; the
version a node reports iskVersionininclude/zetta/version.hpp, which is0.2.0.
Upgrading
- Upgrade every node of a cluster together, then restart it. Mixed versions do not
converge audit history. - The first start migrates. It moves audit events out of the catalogue and re-seals
saved connector credentials. If the audit migration fails the node does not start, and
the old catalogue is left in place. - Keep the cluster join token, and the retired ones in the secret file. Connector
credentials are sealed under a key derived from it. - Grant
sql:write,sql:ddl,users:manageandtokens:manageto whoever runs
those statements today. - Check callers against the smaller behaviour changes above: gate check counts,
pipeline names,definition.files[].contents,upload_datasetname conflicts, and
unrouted requests answering 404 or 405.