Skip to content

OmniGraph v0.10.0

Latest

Choose a tag to compare

@github-actions github-actions released this 31 Aug 21:29
· 63 commits to main since this release
a625748

OmniGraph v0.10.0

OmniGraph 0.10 adds graph-level Blob delivery, commit diffs and change feeds,
conditional writes, benchmark tooling, and merge/read performance improvements.
It upgrades the storage substrate from v0.9.0's Lance 9 to Lance 11.0.0 while
retaining graph storage format v6, recovery sidecars v9, and Lance
V2_2 data files. Existing v0.9 graphs do not require entity export/import.

Coordinated upgrade required. Public graph-facing APIs now describe nodes,
edges, types, entities, and properties; ambiguous table/row names are removed.
CLI/HTTP JSON and public Rust vocabulary have changed, so upgrade the CLI,
server, and client integrations together. Stop the old fleet before opening the
store with the new one. Do not mix Lance 9/10 binaries with Lance 11 readers or
writers.

Rebuild full-text indexes on every branch that needs search. Old indexes
without compatible analyzer proof now fail explicitly instead of silently
missing matches. Ordinary reads, traversal, and vector search remain available;
historical snapshots are retained but may refuse full-text search. Preserve a
verified whole-root backup for rollback and follow the
upgrade procedure.

Highlights

  • Blob reads through the engine, HTTP, and CLI. Select a node or edge Blob
    property at a branch or snapshot. blob get streams bytes; blob stat reads
    metadata. HTTP GET/HEAD support ranges and conditional delivery. Managed
    readers are bounded; valid empty values remain distinct from null through
    compaction. Whole-object external references return their URI without target
    I/O; HTTP redirects, while the CLI never follows them. See Blobs.
  • Secure external-Blob ingress. New URI references are denied by default and
    require configured, normalized allow bases. Server execution never admits
    file://; direct-store CLI access remains deny-only. Existing references
    remain readable/exportable. Writes probe allowed sources before durable
    effects and bound new external references across the whole operation.
  • Exact write receipts and conditional mutations. Mutation and Load return
    the GraphCommit published by that attempt, not a later branch-head lookup.
    A no-op mutation returns commit: null. Canonical reads expose their exact
    graph_commit_id; mutate --if-commit and dedicated HTTP conditional routes
    reject stale tokens before effects with HTTP 412 / CLI exit 4. Older servers
    reject the dedicated route rather than executing an unguarded write. See
    mutations.
  • Commit diffs, change feeds, and baselines. Embedded, HTTP, and CLI surfaces
    provide exact first-parent entity diffs, bounded at-least-once feed pages,
    and streamed baseline/reset handshakes. Cursors bind graph, branch lifetime,
    filters, and a captured cut; consumers own their checkpoints. Retention gaps
    return typed HTTP 410, not a partial diff. Install the complete baseline
    durably before saving its terminal resume cursor. See
    changes and change feeds.
  • Less work for merges and reads. Eligible merges classify candidates from
    fragment lineage instead of scanning every entity; unsupported cases retain
    the full-scan fallback. Warm catalog refresh reads newly published commits
    incrementally. Ordinary node scans omit unneeded properties where safe, and
    constructive multi-type mutations prepare independent datasets concurrently
    while still publishing one graph commit.
  • Safer branch reuse and merge behavior. Deleting and recreating a branch
    no longer reuses its predecessor's native paths/cache identity. Deletion
    acknowledges graph publication; cleanup reclaims the retired forks.
    Branches whose edits net to zero can be merged, and traversal now includes
    stored self-loop edges. See branching.
  • String predicates and stricter query validation. String contains and
    starts_with provide exact, case-sensitive substring/prefix matching, with
    index acceleration where available and correct scan fallback. Missing
    required parameters fail before execution; undeclared variables in inline
    property matches, including negation, fail during typechecking. Nullable
    parameters remain optional. Blob projection, ordering, aggregation, and
    newly declared Blob uniqueness constraints are rejected explicitly.
  • Initialization and recovery hardening. Graph creation publishes its
    initial contents and schema stamp together. An initialization failure no
    longer deletes a graph that actually committed; typed outcomes distinguish
    committed, indeterminate, and claimed roots. A live write handle can heal
    stranded, effect-free recovery intents without reopening, within the existing
    single-mutation-process boundary.
  • Lance correctness fixes. Includes stable-row-ID delete/index alignment,
    Blob compaction/selection, decoder hardening, and overwrite/restore identity
    high-water marks. Ambiguous commit outcomes remain Unknown, never a reason
    to replay automatically. OmniGraph retains its final-output-partition fence
    for exact nearest ranking. The h2 0.4 dependency line advances to 0.4.16;
    the AWS SDK's separate legacy h2 0.3 line is unchanged.

Developer-facing

  • Declarative benchmarks. omnigraph-bench runs YAML-defined merge cases
    from identical frozen fixture state in isolated workers, recording phase
    timings, logical store calls, peak RSS, fixture identity, and source/binary
    provenance. Clean release builds can archive complete or explicitly censored
    canonical JSON; a disposable OmniGraph projection provides bounded queries
    over that archive. Censored runs remain failures. Local records remain
    claim_eligible: false without a controlled, digest-bound build receipt;
    logical calls are not physical cloud requests. The separate FinGraph
    node-and-edge diagnostic supports qualified APFS or EC2 instance-store
    NVMe/XFS fixtures but does not publish durable telemetry or dispatch AWS jobs.
    See the harness guide and
    fixture commands.
  • Merge timing hooks. MergeWriteProbes::merge_timing_snapshot exposes
    stable phase identifiers, total/max microseconds, and completed intervals.
    TableWalk identifies general three-way staging walks; proven-insert paths
    emit none. Normal production execution leaves the task-local probe unset and
    performs no timing clock reads.
  • Compiled operation descriptors. lint --json now includes ordered result
    fields and compiler-owned read/write dependencies, including nested negation,
    inserted-edge endpoints, and node-delete cascades. Invalid queries emit no
    partial descriptor. Server/SDK descriptor projection is not included.
  • Typed storage failures. Rust callers can inspect StorageFailureKind
    (Transient, Configuration, NotFound, Precondition, Permanent,
    Unknown) instead of parsing Lance error strings. Positive transient
    evidence is not replay authorization; erased or ambiguous evidence remains
    Unknown. Generic storage errors still map to HTTP 500.

Compatibility

CLI, HTTP, and embedded API

  • Breaking graph vocabulary. Graph-facing names now distinguish nodes/edges,
    types, entities, properties,
    graph-manifest versions, and published dataset versions. Legacy aliases such
    as table_key, row_id, ambiguous manifest_version, rows_loaded, and
    export --table are removed rather than dual-emitted. Update consumers to
    the release's OpenAPI contract and
    CLI reference; this is not a rolling-safe
    v0.9/v0.10 boundary. embed --json uses record-named counters, and cluster
    observations use graph_manifest_version.
  • New branch lives use internal native refs named {name}.{ulid};
    native_dataset_branch reports that name. Existing bare refs remain valid.
    Public branch paths ending a segment in . plus 26 upper-case letters/digits
    are reserved and refused. See RFC 0042.
  • OmniError::Lance(String) becomes OmniError::Storage(StorageFailure).
    Direct storage consumers must handle StorageError::Backend, replace tuple
    Io matches with Io { failure, source }, and remove
    CreateIfAbsentUnsupported matches. Exhaustive error matches must also cover
    new initialization, Blob, precondition, change-feed, and full-text rebuild
    outcomes; see the error definitions.
  • The embedded read_blob API exposing Lance's BlobFile is removed. Use
    read_blob_at; each managed range is at most 4 MiB. CLI/HTTP stream larger
    values. CLI get outputs raw bytes and a failed transfer can leave a partial
    file; use temporary-file replacement when required. Treat resolved Blob
    snapshot witnesses and ETags as opaque, not as commit IDs or content hashes.

Full-text indexes and retained history

rebuild-full-text-indexes replaces every planned index on the selected branch
from current entities and publishes them in one graph commit. It uses the
default English analyzer, warns that custom tokenizer settings are replaced,
and refuses unknown physical index kinds before publication. Rebuild each live
branch that needs search; other branches and historical snapshots are not
rewritten. Restoring an older snapshot can require another rebuild.

Ordinary optimize preserves existing full-text indexes rather than folding
uncertified postings. Unindexed entities remain searchable through a scan; use
the explicit rebuild command to refresh coverage. Other index reconciliation
and compaction continue normally. See maintenance.

Blob identity and rollback

  • New or schema-rebuilt fields carry stable property-lifetime metadata.
    Schema-preserving writes retain an upgraded dataset's old unmarked schema;
    full-type Overwrite adopts the new metadata. An unmarked pre-0.10 Blob field
    is readable only at its exact current physical dataset entry; older entries
    are refused even if no rename occurred. Historical property renames and
    older branch-owned snapshots can also be refused when identity cannot be
    proven. Current type aliases can resolve pure type-renamed history; retired
    aliases do not remain available. Details: Blob identity.
  • Readers hold their selected version but do not pin it against branch deletion
    or destructive cleanup. Quiesce readers when they must finish. Persisted
    ranged external descriptors are refused, never widened to whole objects.
  • A pre-existing v6 schema with a Blob uniqueness constraint remains openable
    for inspection/export, but that constraint is not made functional. Rebuild
    with the invalid constraint removed.
  • Configuring external_blobs changes cluster state in a way v0.9 rejects.
    A downgrade also loses default-deny external ingress. Removing the field is
    not a full-text rollback: restore the whole pre-upgrade graph backup with
    the old fleet. Follow cluster rollback.

Support boundaries and distribution

  • Azure Blob storage is a preview, not production-supported. Native
    az:// roots, Azurite qualification, and a managed-identity smoke proof are
    available. The disruptive live-Azure qualification matrix remains pending.
    Preview writes must use omnigraph-azure-admission; its lease admits one
    writer process and is not a distributed engine fence.
  • One mutation process per graph remains the supported topology. Conditional
    writes and recovery do not introduce distributed writer fencing. Full-text
    rebuilding is direct-storage maintenance, not an HTTP endpoint; --as is
    attribution, not server-policy authorization.
  • crates.io publication remains paused. Historical omnigraph-* registry
    packages are frozen at 0.8.0 and are not this release. Binaries ship through
    GitHub Releases, the installer, Homebrew, and Docker. See
    distribution policy.