Skip to content

v3.1.0

Choose a tag to compare

@isc-tdyar isc-tdyar released this 19 Sep 04:21
· 674 commits to main since this release
20725bd

Graph scope is now enforced where the data actually lives. Erasure, verification
and temporal deletion each get one owner that reaches every store, and the
graph-key derivations that call sites used to re-implement inline moved behind
Graph.KG.GraphKey. Additive at the interface: no method was removed and no
signature lost a parameter.

Read first if you call drop_graph(). It is now a deprecated alias for
erase_graph() and behaves differently under the same name — it removes the ^KG
adjacency and the temporal trees it used to leave behind. If you relied on the
globals surviving a drop_graph, that is gone. Details in the first entry below.

Added — erasure has one path, and it reaches every store (ADR-0004)

erase_graph(graph=None) removes one graph's content from every store that holds
it; erase_all() also clears the stores no per-graph erase can reach (rdf_labels
and rdf_props are keyed by node id alone, ^NKG ignores graph_id).

drop_graph() is now a deprecated alias for erase_graph(), to be removed in the
next major release.
The name survives, the behaviour does not: it used to delete rows from five tables
and leave ^KG/^NKG answering traversals for a graph with no rows — its own
comment said so ("BYPASS: SQL rows deleted without touching ^KG/^NKG; flag stale").
It now removes the adjacency too, so there is no stale flag for a caller to
remember. Callers that relied on the globals surviving a drop_graph will see the
difference.

Graph.KG.Eraser is that single path, Graph.KG.GraphStores declares the ^KG
inventory it walks, and erasure removes content, not ledger history (ADR-0004).

Added — verify_graph(), which can see the drift verify_sync cannot

verify_sync compares one SQL count against ^NKG's edgeCount: graph-blind, and
one-directional by design, because ^NKG interning is append-only and ignores
graph_id so its counter over-counts. It flags sql_edges > global_edges and stays
quiet on the reverse — which is exactly what every deletion path produced. After the
old drop_graph, sql_edges sat below global_edges and verify_sync reported
in sync.

engine.verify_graph(graph=None) is scoped to one graph, bidirectional, and reads
^KG directly. Graph.KG.GraphVerify holds the check. verify_sync keeps its
contract.

Added — delete_edge_temporal()

The inverse of create_edge_temporal. Removing a single temporal edge previously
meant reaching for a time-based sweep — purge_before, purge_raw_before,
purge_bucket_range — and taking its neighbours with it. Exposed on the facade as
engine.temporal.delete_edge_temporal(source, predicate, target, timestamp, graph=None).

purge_raw_before, purge_bucket_range and get_edge_attrs gain graph=None, so a
purge can be scoped to one graph instead of the default one.

Changed — graph-key derivation has exactly one owner (ADR-0003)

Graph.KG.GraphKey owns the three derivations: ForIndex (default graph =
integer 0), ForLedger ($Char(1)), ForName (the SQL '' spelling). Call sites
that re-derived a key inline agreed with it right up until one of them didn't.

validate_graph_name() mirrors GraphKey:Validate in Python, so a name that cannot
be a subscript raises ValueError naming the graph instead of a RuntimeError
carrying an IRIS <THROW> from several frames down a DBAPI round trip.

Two graph-scope bugs fixed on the way: delete_edge(all_graphs=True) built its index
key from a graph_id assigned only in the other branch, so it raised
UnboundLocalError inside a try that logs and swallows — every row left SQL and
every ^KG entry stayed. Deletion predicates now COALESCE both sides, because the
default graph has two spellings (create_edge writes ''; every INSERT that omits
the column writes NULL) and a predicate matching one of them deletes half the
default graph and reports success. list_graphs() excludes both spellings.

Changed — NodeNotFoundError is a ChangesetOperationError

Spec 217 made it a sibling, so the most common authoring mistake was the one
rejection that would not tell you which operation was at fault. It now carries
op_index and reason alongside missing_node, and its message is unchanged.
except ChangesetOperationError now catches it.

Changed — a namespace is an IVG namespace only once the classes are deployed

The non-USER deployment docs named a specific HealthShare customization namespace as
the place to connect. Following that gets whatever that namespace happens to hold, and
in the common case that is a DDL-only shell: rdf_edges.graph_id nullable instead of
required, primary key edge_id instead of ID, no Graph.KG.Edge/Eraser/
TemporalIndex, no native acceleration, and no schema migration
(tighten_graph_id_column, add_graph_id_to_nodes) ever applied. README now states
the rule, gives the one-line catalog check
(SELECT COUNT(*) FROM %Dictionary.ClassDefinition WHERE Name = 'Graph.KG.Edge'), and
notes that CPF global mapping brings the data but not the behaviour.

tests/integration/test_namespace_isolation.py had been building that shell itself and
asserting isolation against it, which is why its temporal cases had always skipped. The
secondary namespace is now opt-in via IVG_SECONDARY_NAMESPACE, and the suite skips —
rather than creating a shell — unless the namespace already carries Graph.KG.Edge and
a graph-scoped TemporalIndex. Against a properly deployed secondary namespace all 15
cases now run, temporal isolation included.

Fixed — bulk_create_edges_temporal upsert semantics (spec-221 US-3)

upsert=True updated an existing edge's weight through create_edge_temporal but
skipped it through the batch path: Graph.KG.TemporalIndex.BulkInsert carried its own
copy of the store-write body and kept the pre-spec-221 meaning, so a re-ingested
sample silently kept the first weight. BulkInsert now takes mode and resolves it
exactly as InsertEdge does, both write through one private WriteStores(), and
mode= threads from bulk_create_edges_temporal through bulk_write_temporal_edges
to BulkInsert — it was accepted at the engine and dropped before the store.
mode="skip" keeps the old batch behaviour under its real name. The per-edge fallback
in bulk_write_temporal_edges also dropped graph and mode, writing to the default
graph in insert-only mode while reporting every edge as written; it now forwards both.

As on the single path, an overwritten weight does not adjust the ^KG("tagg") bucket
aggregates or the HLL sketch (spec-221 FR-004/FR-011).

Upgrade tests against artifacts real releases wrote

tests/fixtures/snapshots/ now holds frozen v2.16.0 and v2.20.0 artifacts — the
archive each release's own save_snapshot produced, plus every ^KG/^NKG node in
that release's layout, captured from a scratch namespace running that tag's compiled
ObjectScript. Regenerate with scripts/fixtures/generate_old_snapshot.py.

Two e2e suites consume them, one per upgrade path:
tests/e2e/test_upgrade_snapshot_restore_e2e.py (restore an old archive onto the
current schema) and tests/e2e/test_upgrade_migrate_globals_e2e.py (load old
globals verbatim, then MigrateToGraphScoped). See
docs/migration/upgrading-from-older-releases.md
for what old archives cannot carry and which hazards apply to pre-2.17 databases.

Fixed — restore_snapshot

  • A v2.16 archive spells the default graph NULL in rdf_edges.graph_id; the live
    column is required, so those rows could not land. They are now translated to
    ADR-0003's ''. Previously 1 of 4 edges restored, silently.
  • Row insert failures were swallowed by a logger.debug. The result now carries
    failed_rows — per-table counts of what the archive held and the database refused
    — and logs a warning. restored_tables alone cannot tell a dropped row from an
    archive that never held it.
  • Embedding restores read the metadata column into a local and inserted
    (id, emb), losing every embedding's provenance. Both the merge and non-merge
    paths now write it.

Fixed — save_snapshot metadata

iris_version was the literal "unknown" in every archive ever written: the
version bridge was called by a name the module never imported and the except ate
the NameError. ivg_version was hardcoded "1.58.0" and is now the installed
package version, so an archive can be attributed to the code that wrote it.

Fixed — get_embedding_dimension(cursor, table_name) reads the table you name

It accepted table_name and ignored it: the dictionary query hardcoded
Parent = 'Graph.KG.kgNodeEmbeddings', so every caller got the node width no matter
what they asked about, and a table that cannot exist got a confident number rather
than None. On a deployment whose node tables are 384 and whose
kg_EdgeEmbeddings is 768 — the shape reported against 2.18.6, and reproduced on
ivg-iris-enterprise — anything sizing an edge write from this answer sized it
wrong and the write failed downstream, far from the call that chose the number.

It now resolves the class behind the table from %Dictionary.CompiledClass
(SqlSchemaName/SqlTableName), falling back to the name IRIS derives by
convention (schema _→., underscores dropped from the table part) only when the
catalog cannot be read. The default argument is Graph_KG.kg_NodeEmbeddings, so
existing callers keep the answer they had. None now means "no emb column with a
declared width here", not "use the default" — callers that treated a falsy return as
384 need to look again.

resolve_table_class() and derive_class_name() are public on GraphSchema for
callers that need the mapping itself.

Fixed — one default embedding dimension for the whole library

Four entry points carried three different literals, so the width a table got
depended on which one created it and nothing recorded which one that was:
get_base_schema_sql, AdminSchemaRequest, EngineStatus and the CLI's
--embedding-dim said 768; get_procedures_sql_list said 1000. All five now
read iris_vector_graph.constants.DEFAULT_EMBEDDING_DIMENSION, which is 768 — the
value that shipped. A test scans the package so a sixth site cannot reappear.

Nothing was ever built at 1000: get_procedures_sql_list never interpolates
embedding_dimension at all — the generated kg_KNN_VEC says
TO_VECTOR(:queryInput, DOUBLE) with no length — so the disagreement never reached
SQL. Its docstring claimed a DECLARE ... VECTOR(DOUBLE, N) clause that does not
exist, and now says the parameter is unused. initialize_schema is unaffected; it
still refuses to guess and raises ValueError when no dimension was configured or
inferred.

Fixed — the empty-table dimension migration covers kg_EdgeEmbeddings

initialize_schema's auto-ALTER walked the two node tables and, in its
db_dim is None branch, skipped the edge table entirely — so a namespace whose edge
column was created at another width kept it. The mismatch branch did issue the ALTER,
under a bare except Exception: pass that discarded the failure, which is the other
half of why this was quiet for weeks.

The block is now _migrate_vector_dimensions(cursor, dim), returning a report of
unchanged / altered / needs_manual_migration / failed. All three vector
tables are covered in both branches, a failed ALTER is recorded and logged rather
than swallowed, a non-empty table at the wrong width is logged CRITICAL and left
alone, and table names go through the engine's _t() so a prefixed namespace is
migrated instead of Graph_KG regardless of where the engine points.

It still compares each column against the dimension this engine was configured
with, so it cannot see a second writer that disagrees. Two processes initializing the
same namespace at different widths remains undetectable here; the ledger is where
that would have to be caught.

Known issues

Open findings, with what is verified and what is still a decision, are in
docs/KNOWN_ISSUES.md. The ones that affect this release:
five integration files segfault in the driver on their first test and give no
result, so the suite has to be run as separate processes (~66 failures and 26
errors as chunks, all pre-existing at v3.0.1, all tests written against a pre-v3
API); _detect_arno probes with a node that does not exist and so disables a
healthy Arno; Graph.KG.EdgeScan.MatchEdges still overloads graph key 0;
USE GRAPH predicates match one of the default graph's two spellings; four
graph_id writers in cypher/translator.py and bulk_loader.py are still
graph-blind; conftest.py cannot tell a stopped container from a missing one and
falls through to whatever answers localhost:1972, so confirm the container is up
before trusting a gate number; and a named graph cannot carry its own embedding model
or width — the embedding tables have no graph_id, so multiple models means a
namespace per model.