Skip to content

Release v0.2.0

Choose a tag to compare

@homer6 homer6 released this 12 Oct 19:41
· 272 commits to main since this release
4015ef7

Released: 2026-10-12 · Branch: dev · Previous: v0.1.1 (2026-08-27)

v0.1.1 was one methodology (churn) on one node API. v0.2.0 is the engine a hosted
workspace runs on: 21 methodologies behind one registry, composed solutions that decide and
measure, AutoML studies, pub/sub inside the node, a keyed record store and an object store
for workspace data, agents described by records, and an audit trail that no longer lives in
the replicated catalogue. Every change here was already running on the development cluster
as oramasearch/zetta:dev-2cfdac21 before this release was cut.

236 commits across 63 pull requests, 750 files. The generated OpenAPI has 172 /v1
paths (was 78), and the test sources hold 2,977 test cases (was 2,119).


⚠️ Behaviour change: upgrade every node of a cluster together

Audit events moved out of the replicated catalogue into per-origin segments
(#1164). Mixed versions deliberately do not converge audit history. Stop the old
writers, upgrade every node, then restart the cluster.

  • On first start the node migrates the audit events it finds in the old catalogue into its
    own segments, under the reserved origin legacy. The migration is restartable, and a
    failure leaves the old persisted catalogue in place.
  • An unreadable store or a failed migration prevents startup. That is deliberate: a node
    that started anyway would drop history.
  • Valid legacy audit/<uuid>.json files are folded in and removed; invalid ones stay where
    they are and are reported.

There is no flag that restores the old layout. See
docs/governance/audit-trail.md, "Replication and upgrade
policy".

⚠️ Behaviour change: every SQL statement that changes data checks its right

Before this release only INSERT was authorized in the executor (#1229). Now:

Statement Right needed
UPDATE, DELETE, TRUNCATE sql:write, at table or database scope
CREATE / DROP DATABASE and TABLE sql:ddl, at table or database scope
CREATE VIRTUAL TABLE flat sql:ddl
CREATE USER, CREATE TOKEN users:manage, tokens:manage

Column grants and row-level security apply to UPDATE and DELETE; TRUNCATE of a
policy-protected table is refused; a legacy AccessKey needs the flat right. Each such
statement lands on the audit trail as the new data_changed action.

A principal that could run these statements with read rights is now refused. Grant the
right; there is no setting that turns the check off.

⚠️ Behaviour change: a gate check that did not apply says not_applicable

A gate check with nothing to check on a graph used to answer pass, so a churn report read
as about 30 passes that verified nothing (#1240). CheckStatus gains not_applicable,
which never moves the verdict. The report's JSON now lists only the checks that applied,
with the counts applied and not_applicable. A caller that counted pass entries, or
expected every check in the list, sees fewer. Pass include_not_applicable to the gate
capability to get the full list back.

⚠️ Behaviour change: smaller ones that can break a caller

  • A compiled pipeline is named from its intent (#1238), not churn skeleton or
    <methodology> pipeline. A second compile in the same project is numbered (2) instead
    of failing. Pass name to compile_intent to keep a fixed name; the two delivery
    runbooks now do.
  • create_agent and update_agent take a closed definition (#1220): unknown
    properties are refused at the property. A file sent as {path, content} is refused
    instead of being stored empty; the field is contents.
  • upload_dataset refuses a taken name with conflict (409) unless if_exists is
    "rename", and the CSV decoder refuses ragged rows, unnamed or repeated header columns
    and open quotes, naming the line, instead of padding or truncating (#1221).
  • A SQL column that no row carries is an error naming the column and the table's
    columns (#1218).
  • A study whose trials all fail ends failed, with the last trial's reason, instead of
    complete with an empty reason (#1217). A positive cost_cap under 60 trial-seconds is
    rejected at /budgets/cost_cap, because it could never admit a trial (#1211).
  • A request no route matches gets an answer (#1011): 405 method_not_allowed for a
    capability called with the wrong method, 404 unknown_capability or 404 not_found
    otherwise. It used to get no bytes at all.
  • delete_dataset refuses while a pending or running run, or a running study, reads the
    dataset
    (#1258), naming each one.

Added

The methodology registry: 21 methodologies, one way to compile and run them

compile_intent and run route through a registry instead of a churn-only path.
Methodologies M-1 to M-21 are registered and runnable: churn classification, forecasting,
segmentation, anomaly detection, entity resolution (distributed across members), survival,
next-best-action, causal uplift, LambdaMART ranking, language, document extraction, graph,
vision, speech, bandits, optimization, Bayesian uncertainty, out-of-fold stacking, label
efficiency, synthetic data with differentially private GBDT, and online streaming learning.

  • analyze_dataset and plan_features describe a dataset and plan its features. The plan
    is enforced for all 21 methodologies
    , and a run reports the provenance of what it used.
  • Text entity keys and boolean columns are accepted, and the original entity ids come back
    in every output. This closes the v0.1.1 known issue
    #957.
  • A workspace can register its own methodology (register_methodology,
    update_methodology, delete_methodology, record_methodology_run), run by Lake's
    runner (#1261, lake#655).

Solutions: from a score to a decision and a readout

A composed solution registry (create_solution, run_solution) executes through governed
foundation stages. Around it: a replicated holdout experiment manager, a versioned decision
policy engine (create_policy, decide, approve_decision), a replicated audit ledger
(list_ledger), a drift and guardrail monitor (get_monitor), point-in-time snapshots
(snapshot), action delivery (enqueue_action_task, deliver_action_task,
record_outcome) and the ROI readout (get_roi_readout).

Runnable: S-1 churn, S-2 lead scoring, S-3 lifetime value, S-4 collections, S-5 demand
forecasting, S-6 replenishment, S-7 KPI anomaly, predictive maintenance, S-9 automated
outbound, S-10 fraud decisioning, S-11 next-best-action, S-12 claims; and the modules B-1
dynamic pricing, B-2 employee attrition, B-3 ticket triage, B-4 document workflows, B-5
supplier risk, B-6 healthcare operations, B-7 credit underwriting and B-8 security
operations.

Studies, run metrics and models

  • Run-backed AutoML studies: create_study, validate_study, list_trials,
    promote_trial, stop_study, stop_trial, get_study_analysis, delete_study. Trials
    run through the background training driver, under a cost cap in trial-seconds.
  • Experiments over a workspace methodology's parameters, with trials run by Lake's
    runner (lake#689).
  • Durable run metrics: log_run_metrics, log_run_params, log_run_artifact,
    get_run_metrics, compare_runs. A GBDT run logs its held-out scores, confusion matrix,
    ROC and calibration on the Run (#1239).
  • One metric name. The run report's key (held_out_auc, held_out_log_loss,
    held_out_accuracy, held_out_rmse, held_out_mae) is what a study objective names; the
    old spellings (validation.auc, ROC-AUC, auc) are accepted as aliases.
  • Standalone models: train_model, retrain_model, register_model and
    move_model_to_project work without a project
    (lake#654).
  • Bring a model exactly: ONNX, XGBoost and LightGBM models are imported as they are, and
    export_model writes one out (lake#681).
  • Large artifacts move in chunks: a methodology run's artifact over 24 MiB is appended
    with append_methodology_artifact, and read_model_weights reads weights back in chunks
    (lake#690).

Pub/sub inside the node

Stage 1 of the design recorded as decisions D5 to D39 under
docs/plans/pubsub/: typed addresses, an email-style envelope, ring
buffers with a disk tier, topics, subscriptions, inboxes, governance bounds, limits, and
best-effort survival across a restart.

  • create_topic, update_topic, drop_topic, list_topics, get_topic, browse_topic,
    publish_event, receive_events, update_event, delete_events, over REST and MCP with
    parity.
  • A long poll suspends without holding a thread, and the node keeps accepting connections
    while polls are held.
  • Stage 1 is one node. Topic definitions do not replicate
    (#1038).

A keyed record store, an object store, and POST sql

  • Records (put_record, get_record, list_records, delete_record,
    create_collection) with conditional writes and secondary-index lists, and labelled time
    series (create_metric, record_metric, query_metric) (#1231).
  • Records live in a Bitcask-style log per node under <data_dir>/records/, not in the
    replicated catalogue. The first version kept them in the catalogue: after 200,000 puts
    with three live keys the saved document was 141 MiB and the process held 5.2 GB. The soak
    that found it is now a test: 1,000,000 puts hold 76 to 120 MiB of log, 121 MiB RSS, and a
    6.8 ms read p99 during saves (figures from the soak recorded in commit 1b7a12b1).
  • Objects (create_bucket, list_objects, delete_object, object_usage, leases and
    object_barrier), the capability behind Lake's workspace liquid stores. A bucket's log
    stays a prefix after a power loss, a barrier waits for running fsyncs, a failed fsync
    fails the log, a corrupt marked log is refused, and each bucket reports prefix durability
    and an incarnation that a write checks in the same step.
  • POST /v1/sql, and in the Postgres subset: ORDER BY, LIMIT and OFFSET;
    count(*) and other aggregates with no GROUP BY; GROUP BY on any type; SELECT *
    fills columns even when no row matches.

Deletes and storage accounting

  • delete_project (#1013), and delete_dataset, delete_model, delete_run and
    delete_index (lake#645), which each
    take dry_run.
  • upload_dataset loads a CSV or JSONL file's text over the API (#1012). A dataset records
    its source_format.
  • storage_usage reports disk by kind for a node and for a whole cluster
    (lake#658); node_memory reports memory
    by owning subsystem.
  • Search returns the matched rows' fields with each hit.

Agents

  • AgentRecord describes every agent Lake shows.
  • Granted Pipelines are tools in native chat.
  • An agent record may carry mcp:<prefix> skills
    (lake#652), and Lake's own areas are
    agent skills (agent-runtime#7).

Fixed

Saved connector credentials were stored in the clear — #1230

They are now sealed at rest with AES-256-GCM under an HKDF-SHA256 key derived from the
cluster join token, and they are never in the replicated catalogue. Node start re-seals
legacy plaintext and old-key entries, and the secret file keeps retired join tokens so a
rotation does not strand them. No read surface returns a credential: system.connections
and the connection capabilities report only whether one is set.

The audit trail kept a node's memory high and grew while idle — #1164

138,000 audit: keys in the in-memory catalogue held RSS at 730 MB. Events now live outside
the catalogue, polling is summarized instead of logged per call, idle growth is bounded, an
administrator can purge (purge_audit, itself audited), the trail reconciles in linear
time, and index lookups stay bounded.

The API server could go silent or lose work

  • A request no route matched got no response at all, and a throwing handler dropped the
    connection or ended a headless node (#1011). A handler exception is now an engine_error
    naming the capability, and invalid UTF-8 in a response body is replaced, not thrown on.
  • In-flight requests drain before shutdown; request frames are kept until owner work
    completes; the HTTP worker stops when node startup is refused.
  • An offloaded call runs its gate on the owner thread, and the owner loop wakes for
    run_on_owner work and drains it on every wake.
  • A signal handler is not replaced while a dispatch to it is still in flight.

Studies and training

  • A 12-row trial ran for ten minutes and slowed workspace reads, because every boosting
    round captured the whole Run and Study into the replicated document. Captures are now
    written at most once per second: two 200-tree trials on 12 rows went from +2.6 MB and
    135 ms per save to +150 KB and 5 ms (figures recorded in commit dc69a420, #1217).
  • run metric logging failed was reported for any stop, including the study's own wall-time
    budget. The train job now records why it stopped.
  • An experiment's trial models are named from the experiment. A text entity-key mapping is
    answered as a count in run and model reads.

Pipelines

  • A run says what it read, what it could not use, and when it trained nothing. A run given
    no dataset reads its pipeline's own table.
  • The churn index step indexes the landed table, and M-1 step names fit the dataset.

Replication

Replication caches values and reconciles only the keys that changed; solution records
reconcile incrementally.

Build

Beast2 links with Corosio's OpenSSL provider. On macOS the build links Homebrew's
libunwind with Homebrew's libc++abi: against the system unwinder a matching catch was
skipped and std::terminate() ran.


Documentation

  • AGENTS.md for agents working in this repository.
  • The pub/sub design, decision by decision, and the stage 1 spec as built.
  • CLAUDE.md and the architecture docs cover pub/sub.
  • orama-cloud-v1-operator is orama-operator throughout.

Verification

Full test suite, run serially (ctest --test-dir build, no -j) on the release commit,
macOS with 16 hardware threads, 744 seconds:

2,873 tests · 2,854 passed · 2 failed · 17 skipped (real checkpoints, a GPU or an
external engine not configured) · 1 more disabled (a benchmark)

The two failures are one defect, not load
(#1276): run threw on a system metric
key that had not been sampled. RestClusterFixture.RunReportAnswersForADatasetlessRun
passed 4 of 5 times when rerun alone; Uc1Journey.JourneyUc1EndToEndWithActingKey passed
5 of 5. See Known issues.

The build has two warnings: a discarded io_result at
src/core/ml/forecast_execution.cpp:99, and a duplicate-library note from the linker.

Running in a cluster: this commit's image, oramasearch/zetta:dev-2cfdac21, is the
workspace Zetta on the development cluster (four pods), serving Lake at dd1574fd. The
image was built before the version string moved, so those nodes report 0.1.1.


Known issues

  • #1276 — run fails
    intermittently with engine_error (key 'system.cpu.percent' not found), or ends the
    process on system.throughput.per_second, when a system metric was not sampled for the
    step. Seen within tens of milliseconds of node start. Retry the run.
  • Tests that fail under a loaded host and pass alone. The mutual-TLS mesh test and the
    object and record store latency benchmarks
    (#1256,
    #959), the record store soak
    (#1248), multi-member peer discovery
    (#1124), and two distributed tests
    (#964,
    #963).
  • #786 — mutual-TLS gossip can stall,
    because the read pump and the write pump share one OpenSSL stream. Still open from
    v0.1.1.
  • #999 — there is still no reachable
    consent approval surface, so consent enforcement stays off by default. Still open from
    v0.1.1.
  • #1001 — the Snowflake connector is
    PAT/OAuth only. Still open from v0.1.1.
  • #977 — a fresh node has no agent
    loaded. Still open from v0.1.1.
  • #1272 — delete_dataset cannot
    see the tables read by a run that was created outside the run capability, so it does not
    refuse for those.
  • #1255 — a text entity-key mapping
    is stored inline in the replicated Run and Model records, which grows them with the number
    of entities.
  • #1203 — dataset_path does not
    encode categorical features, and does not type columns from the partition catalogue.
  • #1174 — receive_events on an
    exposed topic does not support a 60-second poll.
  • Pub/sub is one node (#1038), has
    no streaming subscriptions (#1037),
    and has no per-role delivery bounds
    (#1036).
  • #1014 — the source connectors are
    still Postgres and Snowflake only.
  • project(... VERSION ...) in CMakeLists.txt still says 0.0.1. Nothing reads it; the
    version a node reports is kVersion in include/zetta/version.hpp, which is 0.2.0.

Upgrading

  1. Upgrade every node of a cluster together, then restart it. Mixed versions do not
    converge audit history.
  2. The first start migrates. It moves audit events out of the catalogue and re-seals
    saved connector credentials. If the audit migration fails the node does not start, and
    the old catalogue is left in place.
  3. Keep the cluster join token, and the retired ones in the secret file. Connector
    credentials are sealed under a key derived from it.
  4. Grant sql:write, sql:ddl, users:manage and tokens:manage to whoever runs
    those statements today.
  5. Check callers against the smaller behaviour changes above: gate check counts,
    pipeline names, definition.files[].contents, upload_dataset name conflicts, and
    unrouted requests answering 404 or 405.