These files say what a statement in Drivine's flavour of Cypher gives on each store Drivine runs it on. The graph engines run Cypher themselves. On Postgres, Oracle and SQLite, Drivine compiles it to SQL. The three Drivine libraries (drivine4j, drivine for TypeScript, drivine-python) run the same files, so a statement gives the same rows from each of them.
queries.yaml: statements that read, run over the graph the file'sgraphwrites. The graph is emptied and written again before each case.writes.yaml: statements that write, run from an empty graph. Each case's steps run in turn, so a step reads what the steps before it wrote.temporal.yaml: openCypher's temporal values (dates, times, moments and durations), run from an empty graph. A graph engine's driver gives a temporal value as a value of its own, and a SQL store gives its ISO text, so the cases read a value's fields.refusals.yaml: statements each SQL store's compiler refuses, and its message. They are compiled with no store.writevalues.yaml: the values aCREATEor aMERGEwrites, as a table: each position a write takes a value in, by each form of value, and Neo4j's verdict on each. See Values a write makes.nesting.yaml: where a statement reads a variable aWITHdrops, as a table: each outer context, by each nesting of subqueries, comprehensions andCALLs up to two deep, by each reference, and Neo4j's verdict on each, with a sample three deep. See Nestings of a dropped variable.rebinding.yaml: where a subquery binds again a variable it has from outside it, as a table: each context (CALL (x),CALL (*),CALL (), andCALL ()within anEXISTS { … }or aCOUNT { … }, and those two alone), by each binder, by a binding and two controls, and Neo4j's verdict on each, with a sample two deep. See Rebinding a variable from outside a subquery.schema.yaml: Neo4j's schema commands (CREATE INDEX,CREATE CONSTRAINT,DROP …,CALL db.awaitIndexes), and the schema spec a SQL store applies each as, or its refusal. They are read with no store. See Schema commands.store.yaml: the store layout's DDL: the tables, the probe that finds them made, and each index and constraint as each store makes, drops and lists it, for the graphdrivineand a set of schema specs, with a read for each index that the compiler must answer from it. Read with no store. See The store layout. It also holds the store's schema: how each store's listings are read back, whether an item is a spec's and in its shape, and on a SQL store of which form, what each store can do, and what its errors say. See The store's schema.marker.yaml: the schema marker (_DrivineSchema) and the stamps: the statements each store reads and writes the marker and the store's stamp (DrivineStore) with, the marker's inventory as it is written and read back, the version token, and the form of the stamp each node carries. Read with no store. See The schema marker and the stamps.migrate.yaml: a store's layout migrated, from layout 1 to layout 2: each row of a node's or a relationship's properties as layout 1 held it, on each SQL store, rewritten as layout 2 holds it, under what the migration is told of the properties' kinds. Read with no store. See Migrating a store's layout.plans.yaml: the schema plan: what a schema manager runs on each store, round by round, for an enforce, a recreate, an ensure or a remove, what the store answers, and what the manager reports, its log lines among it. Read with no store. See The schema plan.host.yaml: what each library's own code does with the values compiled SQL hands it: the functions it registers on SQLite, and how it reads a column of a statement's rows. Read with no store. See Host functions and readers.search.yaml: full-text search: the nodes a search finds on each store, over the graph the file'sgraphwrites, and how closely each other store ranks a corpus,search/docs-corpus.json, as Neo4j's full-text index ranks it. See Full-text search.among.yaml: a search among ids, a library'sloadNearestAmong: for each of a set of nodes, its nearest among the others, each pair once or both ways, on each store, over the graph the file'sgraphwrites, with each neighbour's score. See A search among ids.vector.yaml: vector search in Cypher: the nodesdb.index.vector.queryNodes()and Cypher 25'sSEARCHfind on each store, with their scores, over the graph the file'sgraphwrites, and whatvector.similarity.cosine()andvector.similarity.euclidean()give. See Vector search.fragments.yaml: the parts of a statement a library writes itself, its object manager's filters and orders, as the compiler's module gives them on each SQL store, and the rows Neo4j keeps or orders for each. See Fragments.results.yaml: a statement's columns, and its values with their kinds kept, a node with its labels and a relationship with its type, as each library's call that gives them reads them (drivine4j#741), over the graph the file'sgraphwrites. See Columns and typed values.cli/: thedrivineCLI's contract, which runs Cypher on every store and prints its results as cypher-shell does, andcli.yaml, the cases each build of it runs. Seecli/README.md.model/: the model file each library writes of an application's fragments, views and how they are stored, which Drivine Studio reads. Seemodel/README.md.
A runner empties the whole graph before each case, not just the nodes that carry the file's label. The statements match plain labels, such as MATCH (p:Person), so a node another test left behind would show up in their rows.
- name: "Optional match"
description: "OPTIONAL MATCH keeps the row, with null where it finds nothing."
steps:
- cypher: "MATCH (p:Person) OPTIONAL MATCH (p)-[:KNOWS]->(f) RETURN p.name, f.name ORDER BY p.name"
rows: [["Carrie", null], ["Hugo", null], ["Keanu", "Carrie"], ["Laurence", "Keanu"]]A case has a short name, unique in its file, that a library's runner shows and skips by, and a one-line description of what it checks. Then its steps, run in turn.
A step has:
| Key | What it is |
|---|---|
cypher |
The statement. |
parameters |
The values bound to its parameters. |
rows |
The rows it gives. A row of one column is its value, and a row of several is a list of their values, as PersistenceManager.query gives them. |
error |
The message it fails with, a list of messages any one of which passes, or true for a failure of any message. |
only, except |
The stores the step runs on, or doesn't. A store is neo4j, memgraph, falkordb, neptune, postgres, oracle or sqlite. The group graph names the graph engines, and sql names the stores Cypher is compiled for. |
unordered |
The rows are compared in any order. |
unorderedLists |
Each list within the rows is compared in any order. |
keyOrder |
Each map's keys are compared in order too. |
note |
Why the step is there. It is not read. |
A step with no rows and no error is run, and what it gives is not checked.
On Neo4j a statement runs in the database's default language, Cypher 25 on Neo4j 2026.10. A step that pins what Cypher 5 gives, where the two differ, starts its statement with CYPHER 5 and runs only: [neo4j].
- A whole number and the same number written with a fraction are different values:
2is not2.0. {$repeat: "x", times: 3000}is the textxwritten 3000 times, in parameters and rows alike.{$float: "NaN"},{$float: "Infinity"}and{$float: "-Infinity"}are NaN and the infinities, in parameters and rows alike: not every YAML parser reads.nanand.inf. NaN in the rows is the same as NaN there.- Any other map is the map it is, even one keyed with
$, such as{$numberDouble: "1"}. - A date or a moment whose year a store doesn't hold fails there, where Neo4j works it out (#263; working such years out is #264). Postgres holds years from -4712 (4713 BC), Oracle from -4711, SQLite from -9999, each up to 9999. A value made of fields of a year outside those fails by name, as does one worked out on Postgres or SQLite whose year is past 9999, or on SQLite before -9999; Oracle fails on such a year, and Postgres and Oracle on one before their first, as they work it out. A text written in the statement,
date('+10000-01-01'), is read as it is. - A temporal value with a nanosecond past the microsecond, a seventh, eighth or ninth digit of its fraction of a second that isn't 0, is kept in its text on the SQL stores, written, read back, compared and sorted, and its fields read off the text; but where it would be worked out through the store's timestamp, which keeps microseconds, it fails by name there, where Neo4j works it out (drivine4j#421; working it out is #246): moved by a duration, measured by
duration.between, made of a map's fields over one that gives its time (localtime({time: x})) or over a moment with an offset moved to anothertimezonebeside ananosecond, or a field read off the timestamp, asdayOfWeek,weekorepochSecondsis. A value made of ananosecondfield keeps all nine digits in its text, and a truncation, to a unit of a microsecond or coarser, or a conversion reads its text cut to the microsecond, a time keeping its fraction, so each gives Neo4j's answer (#302). A date takes a duration's whole days alone, so a duration's nanoseconds are no matter beside one. - A duration multiplied or divided by a float is worked out exactly on the SQL stores, where Neo4j works it out in binary floating point:
duration('P1M') / 2.5isP12DT4H11M38.4Sthere, andP12DT4H11M38.399999999Son Neo4j, a month being 2,629,746 seconds. The exact answer is kept (#254): such a step pins Neo4j's answeronlyon Neo4j, and theirs apart. - A
sum()or anavg()of anything but integers, floats and nulls fails by name on the SQL stores, whose sum adds the numbers alone and gave 0, and the average null (#281): a duration as not yet worked out, where Neo4j gives one (#282), and any other kind as Neo4j fails it. It fails as the statement compiles where the statement makes the items of such a kind, and as it runs where it reads one. A step pins Neo4j's durationonlyon Neo4j, and the SQL stores' failure apart. - A temporal value made of a map of its fields that a parameter, a parameter's key or an expression gives, not written in the call, is refused by name on the SQL stores, where it gave null and Neo4j makes it (#328): as the statement compiles where the compiler knows the map, and as it runs where it reads one, as an item of a list. A step pins Neo4j's answer
onlyon Neo4j, and the SQL stores' refusal apart. - A temporal value made of a list, a number or a truth value fails by name on the SQL stores, where it gave null, as Neo4j fails it (#331): as the statement compiles where it is written or given, so never null, and as it runs where it is read, a variable's, a function's or a
CASE's, which gives null where it is null, as Neo4j's does; a truth value a comparison makes,date(n.x > 1), is refused as it compiles, though it may be null. Neo4j fails only a row that reaches the call, so a statement that gives no rows is refused on the SQL stores where Neo4j gives none. +of a truth value and a number, a truth value, a map or a temporal value, or of a map and a number, gave null on the SQL stores, where Neo4j refuses it (#341): it is refused as the statement compiles where the compiler knows both kinds. A truth value put together with a text is its text,'a' + trueisatrue, as in Neo4j, where the SQL stores gave null.- A map beside a text or a temporal value in
+,*or/of a temporal value other than a duration, andNOTof a parameter given a value that is no truth value or of a text put together, gave null on the SQL stores, where Neo4j refuses each as it plans the statement (#345): each is refused as the statement compiles. A truth value read as the statement runs (a property, a variable or an item) beside a number the statement makes or gives, in+,-,*or/, fails the statement there, where it gave null, as Neo4j fails it, by one test of the side read:n.flag + 1,UNWIND $l AS b RETURN b + 1. Beside a number the statement makes that is null as it runs (n.flag + toFloat(n.missing)), it is null, as in Neo4j. Beside a side of a kind left open too (n.flag + n.n), a map or a temporal value (n.flag + date('2024-01-01')), or worked out rather than read (head($l) + 1), it still gives null on the SQL stores, where Neo4j fails; so does a text, a list or a map read as the statement runs in-,*or/, and a truth value in%or^. A test of each side's kind in every+(#341) grew a sum of sums at every level. - A temporal value given at an offset with seconds, a parameter or a value a save writes, is refused by name on the SQL stores, where Neo4j holds it, as the store holds an offset to the minute (see "A time" in Values in the store):
{"$temporal": "DATE_TIME", "text": "1971-01-01T00:00:00-00:44:30"}, or a JVMOffsetDateTimeorZonedDateTimethere, as java.time writes a zone's local mean time (Africa/Monroviabefore 1972), is refused with "A value given as a DATE_TIME isn't one: 1971-01-01T00:00:00-00:44:30 (the store holds an offset to the minute)." So is any other value given as a temporal value whose text is no value of its kind,{"$temporal": "DATE", "text": "2024-13-01"}. Each was bound, or saved, as its text, equal to no temporal value (#357).host.yaml's "Values a save writes" pins the refusal; the spec's statement files take no parameter given as a temporal value, so a statement's is pinned by the module's tests. - Quote every string. A YAML 1.1 parser, such as SnakeYAML or PyYAML, reads a bare
y,yes,nooronas a boolean. writes.yamlbinds and expects a whole number beyond 2^53. A library whose language reads YAML numbers as doubles skips those steps, named in its own runner. Otherwise a library skips nothing: a store that differs says so withexcepthere.
results.yaml is run through each library's call that gives a result's columns and its values with their kinds kept, not through PersistenceManager.query. A library that has no such call yet doesn't run it. Its cases are as queries.yaml's, and each step has columns too: the result's column names, in order, as a graph engine names them, given for a result with no rows too, and empty for a statement that returns nothing. A row of one column is still its value, and a row of several a list of them.
In its rows:
{$t: "node", labels: […], properties: {…}}is a node, with its labels in any order;{$t: "relationship", type: "…", properties: {…}}is a relationship;{$t: "path", nodes: […], relationships: […]}is a path, its nodes and relationships in the order it walks them;{$map: {…}}is the map it holds, for a map with a$tkey of its own.
On a SQL store, the compiler gives the columns, and compiled typed (typed) it tags each node and relationship as above in the columns that hold them: see the compiler's README. It is opt-in, and untyped every statement compiles as before.
A SQL store runs Neo4j's schema DDL through its library's schema machinery: a CREATE is ensured as the schema API ensures a spec, under an owner of its own, ddl, so the item has the name, the expressions and the inventory entry the API gives it. writes.yaml runs such statements on Neo4j and the SQL stores alike. schema.yaml pins how each library reads one, with no store:
- name: "Range index"
description: "CREATE [RANGE] INDEX over one property or several is a range index."
steps:
- ddl: "CREATE INDEX person_name IF NOT EXISTS FOR (p:Person) ON (p.name)"
create: {kind: "RANGE_INDEX", label: "Person", properties: ["name"], alias: "person_name", ifNotExists: true}A step has ddl, the statement, an optional note, and one of these, which says what it reads as:
| Key | What it reads as |
|---|---|
create |
A spec to create: its kind (RANGE_INDEX, UNIQUENESS_CONSTRAINT, FULLTEXT_INDEX, VECTOR_INDEX or TEXT_INDEX, as a store's catalog names it), label and properties; alias, the statement's name where it is not the name the schema API gives the spec; ifNotExists; and options: a full-text index's analyzer, and a vector index's dimensions, similarity (COSINE or EUCLIDEAN), hnswM, hnswEfConstruction and quantizationEnabled. A key left out is null, false or empty. The spec itself has no name: the schema API's is the one it is made under, and the alias is kept beside it, so that a DROP by it finds the item. |
drop |
A drop of the item by its name: kind (INDEX or CONSTRAINT), name, and ifExists. |
await |
true: a wait for indexes, which does nothing on a SQL store. |
reserved |
Drivine's own uniqueness constraint, of the schema marker's scope or the store stamp's key, by any name: its label and properties. It is made as the library makes it for itself, and recorded in no owner's inventory. |
cypher |
true: not a schema command, but Cypher, compiled as any other statement. |
error |
The message it is refused with. A SQL store's compiler refuses it with the same message (see refusals.yaml). |
Each library makes the same tables, indexes and constraints on a SQL store, and the same schema statements on a graph engine, so a store one library made is read alike by another. store.yaml pins them: drivine4j's statements, the reference, as its schema grammars gave them when they were captured, for the graph drivine. The compiler's module (compiler/, org.drivine.cypher.store.ddl) writes them, and its tests check it gives these, byte for byte, on every target.
A function whose body the module generates is held in store.yaml by a marker, not by its text: the function of temporal arithmetic Postgres and Oracle keep (TemporalArithmeticFunction, liberation-data/drivine4j#418) is the compiler's own temporal arithmetic, some 70 to 90 KB on each store, and changes wherever that does. SQLite keeps none: its compiled SQL calls the host function drivine_temporal_arithmetic, which host.yaml pins. <TemporalArithmeticFunction.postgres(drivine)> is the CREATE OR REPLACE FUNCTION Postgres's ddl holds, and <TemporalArithmeticFunction.postgresEnsuring(drivine)> the same statement as its ensuring block runs it; <TemporalArithmeticFunction.oracle(drivine)> is the function as Oracle's functions hold it, and <TemporalArithmeticFunction.oracleClob(drivine)> that text as the CLOB literal its ensuring block runs. A runner expands each marker with the module's GeneratedFunctions.expand, or the JSON request {"expand": "<statement>"}, before it compares: the one place a marker is read. store.yaml's generated pins the SHA-256 of what each marker stands for; the digest of Postgres's whole DDL at each PostgresDdl.VERSION, which a body that changes without the version changing fails, as a store made at that version runs the DDL again only when it changes; and the version each Oracle function's first line ends with (-- drivine:<hash>), the first twelve hex digits of its text's SHA-256, by which Oracle's ensuring makes again a function whose body has changed.
graph: "drivine"
tables:
postgres: {ddl: [...], ensuring: "DO $ensure$ ..."}
oracle: {ddl: [...], functions: [...], ensuring: "DECLARE ..."}
sqlite: {ddl: [...], made: "SELECT CASE WHEN ...", file: "SELECT file FROM pragma_database_list ..."}
catalog: {item: {...}, aliases: [...], stores: {postgres: {indexes: {RANGE_INDEX: "...", ...}, constraints: "...", aliased: "...", unaliased: "..."}, ...}}
names:
- {store: "postgres", graph: "drivine", kind: "ix", parts: ["Person", "name"], name: "drivine_ix2_Personname_44ce5436"}
refusals:
- {store: "postgres", graph: "1graph", refusal: "'1graph' cannot name a graph on Postgres: ..."}
cases:
- name: "Range index"
description: "A range index over one property."
spec: {kind: "RANGE_INDEX", label: "Person", properties: ["name"]}
stores:
postgres:
create: ["CREATE INDEX IF NOT EXISTS drivine_ix2_Personname_44ce5436 ON drivine_node (((props -> 'name'))) WHERE labels @> '[\"Person\"]'", "INSERT INTO drivine_schema ..."]
drop: ["DO $forget$ ..."]
neo4j:
create: ["CREATE INDEX `Person_name_range` IF NOT EXISTS FOR (n:`Person`) ON (n.`name`)"]
drop: ["DROP INDEX `Person_name_range` IF EXISTS"]
dropUnnamed: {error: "Cannot drop index on Person[name]: ..."}
read:
cypher: "MATCH (n:Person {name: $x}) RETURN n"
parameters: {x: "Ada"}
indexedProperties: [["Person", "name"]]
holds:
postgres: ["(props -> 'name')", "labels @> '[\"Person\"]'"]| Key | What it is |
|---|---|
tables |
What makes each SQL store's tables, base indexes and functions (ddl, and Oracle's functions), each statement safe to run again, and how a library finds them made. On Postgres, ensuring is one block: the probe of the node table's comment, which holds the version token (drivine:12), the graph's advisory lock, and the ddl. On Oracle, ensuring probes the property graph's doc property and each function, and makes what is missing, or a function not of its version. A function whose compile fails (ORA-24344, or ORA-04021 and ORA-04031, a busy library cache or a short shared pool) is compiled again after 0.5 s, then 1 s, and after its third try the block fails with ORA-20818, naming the function and what user_errors says of it (liberation-data/drivine#818). On SQLite, made counts what of the ddl is there, or is -1 for tables made without AUTOINCREMENT, and file names the database's file. |
catalog |
Each store's queries that list its indexes of each kind and its constraints; on a SQL store, the statements that set and take away the aliases of the record of item; and Oracle's labelled, the labelled indexes a library tells the compiler of as indexedProperties. On a SQL store a listing of range indexes or of constraints, and on Postgres and Oracle one of vector or full-text indexes, gives, after the records, a row for each object under a name Drivine gives that kind's objects (see Forms). |
names |
The name a SQL store gives an object of kind made for parts, a label and properties, for a graph: <graph>_<kind><layout>_<readable>_<hash>, where layout is the store layout the object holds values in, 2 (see Values in the store). readable is the parts' ASCII letters and digits run together, cut to at most 40 and to what leaves room, within the store's longest name (63 on Postgres and SQLite, 128 on Oracle), for the rest and a _t. hash is Java's String.hashCode of the parts joined by U+0000, over their UTF-16 code units, as Integer.toHexString writes it: unsigned, in lower case, with no leading zeros (drivine_ix2_name_337a8b). |
refusals |
Why a store refuses a graph's name, or null where it takes it. A graph's name is at most the store's longest name less 16 (47 on Postgres and SQLite, 112 on Oracle): the longest a name adds is _uq2__, a hash of 8 with nothing readable, and the _t, _i or _r of an object made beside it, so each fits. |
cases |
An index or a constraint: its spec, as schema.yaml writes one, with ignoreCase and the engines' vector options (quantizationEnabled, Neo4j's; efRuntime, FalkorDB's) among its options. A spec that ignores case is made over the lowercased copies of its properties, under a name of its own, as the schema API makes it. existing is what the store has of the same kind on the label already, which only FalkorDB reads. |
Each case's stores hold, for postgres, oracle, sqlite, neo4j, memgraph and falkordb, what makes the item (create), and on a SQL store the form its record says it is made in, with the digest of its create: SHA-256 of the statements' UTF-8, each after the first behind U+0000. A change to what a create makes is a new form, so its record tells it apart from an item made before: the module's SqlStoreDdl.CURRENT_FORM goes up, and every case's form with it. A test fails where a create changes and its form doesn't. The form is 2, layout 2's DDL; 1 was drivine4j's DDL as it was pinned when records began to say their form. Then what drops it by its name (drop), and, where it differs, what drops it with no name known (dropUnnamed). On Postgres, claim and release take and give up an ensure's turn to build it, binding $at, when it was claimed (see "An index built online", in plans.yaml's section). A uniqueness constraint's SQL stores have recreate too, what makes it again in its place so that data it refuses leaves it the index it had (drivine-cypher-spec#135 §4): leftover, which drops the unique index a recreate that stopped short left under the name it builds under first, the constraint's index's name and _r (SqlStoreDdl.temporaryName, which fits at the longest a graph's name may be); build, the create's unique index under that name, without its IF NOT EXISTS, while the old one stands (on Oracle within the block that waits out another session's lock); swap, which puts it in the old one's place (Postgres: one block that drops the old and renames it; Oracle: the drop, then the rename; SQLite, which renames no index: the drop, the create's index made again under its name, which the data, held unique by the one just built, can't refuse, then the one built first dropped); finish, which puts the one built first in its place where a recreate was cut off after the old index was dropped (the rename; on SQLite the index made again under its name and the one built first dropped); and remade, which, after the create runs again and finds the index made, records the record as made in the current form as the index under its name, under the item's name, keeping its aliases, as a record made anew would be. The old index the swap drops is the one under the name this version gives it, not one its record names otherwise: the three libraries name it alike. Its form and digest (of leftover, build, swap, finish, then remade) are a create's: what it leaves is the index the create makes, so it is of the create's form. Each is a list of statements, or {error}, the message the store refuses it with. A statement is its text, or on FalkorDB {native: [...]}, a command of its driver (GRAPH.CONSTRAINT), where {graphName} stands for the graph. A constraint's falkordb has constrainable, {error}, where FalkorDB refuses it before its create: GRAPH.CONSTRAINT takes only a plain name for a property. On a SQL store, a create follows the statements that make the store, which the case leaves out: Postgres's and Oracle's ensuring, and SQLite's ddl. On Oracle, each index a create builds over the node table, the range index over every node's property, the labelled index and the unique index, is run within a PL/SQL block that runs it again a tenth of a second after another session's lock turns it away (ORA-00054), for up to 300 tries, and then raises, as ensuring runs its statements: Oracle's DDL does not wait for a lock, and libraries starting together each make the store stamp's key constraint (liberation-data/drivine4j#517). An index another session made after the block's IF NOT EXISTS found none (ORA-00955) is taken as made, as ensuring takes it.
Reads. A range index and a uniqueness constraint are only worth their DDL if a statement is read through them, so each such case has a read: a statement, its parameters, and the indexedProperties a library gives the compiler on Oracle. Compiled on each SQL store, its SQL holds each of holds, the expressions the index is made over as its create writes them, with each column a statement reads of the node, props, labels or stamp, as its alias's (Oracle's index names them as n's). A runner checks each expression is in the case's create, as it is or within a PL/SQL literal, quotes doubled, and that the compiled SQL tests it where a planner reads an index by it: as a conjunct of its WHERE, a label's test, or as the left side of a conjunct's =, and not only within a CASE or beside an OR. A vector and a full-text index have none: their searches are each library's, not compiled Cypher. A case may also have reads, a list of more of them, each read alike: a lookup the index answers by a value the statement works out, such as {name: toUpper($x)}, among them (#238).
A library saves to a SQL store by a plan, a JSON array of steps, which Postgres's write function <graph>_write carries out (PostgresWriteFunctions) and each library carries out itself on Oracle and SQLite, alike. A step that finds a node by its id, node, ref or patch, names its shape s, an index into the plan's shapes ({labels, key}, key the id's stored key, $-escaped as every key is), and carries the id twice:
id: the id as the node holds it, the JSON the store layout writes (Values in the store): a text, a number, a truth value, or a temporal value's tag. The node is found byid, compared as the store's range index or uniqueness constraint is made over the property (Postgres:props -> key; Oracle and SQLite: its class and key), so the index answers it; a node made is written with it.k: the id's text: a temporal id's tag's textv, any other id as->>reads it. Postgres's function reads nok: it finds a node, and keys the nodes one plan makes, byid.
The plan's values, each id and each property a step writes, are written as the module's write gives them for the store (StoreValues.text, or the JSON request {"write": <value>, "store": <store>}), the text a compiled statement that writes binds, so a value saved is held as one CREATE made: on Oracle every float is extended JSON's {"$numberDouble": …}, so Oracle reads the plan as extended JSON (EXTENDED), which makes each one a binary double.
A node, ref or patch step without id is refused by name (drivine-cypher-spec#245): libraries that wrote plans before 0.48.1 sent ref and patch steps only k, and must add id when they take it. An unlink step names the targets of a relationship field it keeps or removes by their ids' keys, ids and held, each as the module's planKeyOf gives it (the JSON request {"idSql": {"store": …, "helper": "planKeyOf", "json": …}}) and as it reads each target back (planKey): a text id marked t:, a temporal target's tag's v, any other as ->> reads it, a number on Postgres as numeric's text. So a text id '2026-01-05' and a date id 2026-01-05, or '1' and 1, are two targets, not one (drivine-cypher-spec#391).
How a value is held in a SQL store's props, and in the JSON compiled SQL passes around, is a contract of its own: every library writes it, and anything else that writes a store directly must write it so too, or the libraries read its values wrongly or refuse them. This is store layout 2, the one the store stamp names (see The schema marker and the stamps). The graph engines hold their own values and are layout 1.
Plain values. A text, a truth value, an integer and a list are JSON's own. Bytes are the list of their signed values, [1, 2, -1] for [1, 2, 255], as a graph engine holds them. A property's value is never a map, and never JSON's null: a property set to null is not there.
Floats. On Postgres and SQLite a float is a JSON number written with a point, 2.0 or 1.5: jsonb keeps the scale of 2.0, and SQLite's json_type calls it real. On Oracle every float is an OSON binary double, type() double: written as JSON_SCALAR(TO_BINARY_DOUBLE(…)), JSON_TRANSFORM(… SET … = TO_BINARY_DOUBLE(…)), or extended JSON's {"$numberDouble": "1.5"} read with JSON(… EXTENDED). JSON's 1.5, even read as extended JSON, is a NUMBER, and so is held as an integer or a decimal, not a float. NaN and the infinities are held as each store holds a float, the spellings its compiled CREATE stores and its reads know: on Oracle each is a binary double, written as extended JSON's {"$numberDouble": "NaN"}, "Infinity" or "-Infinity" and read with JSON(… EXTENDED); on SQLite an infinity is the number its JSON writes for one, 9e999 or -9e999, and NaN, which its JSON holds as no number, is not stored; on Postgres, whose jsonb holds none of them, none is. A write of one the store doesn't hold, a save's as much as a statement's, is refused by name, with the message a statement that writes is refused with ("A statement that writes is given NaN or an infinity, which Postgres's `jsonb` holds as no number: …", and SQLite's of NaN), never stored as null or as a text in its place: host.yaml's "Values a save writes" pins each store's text. A list of numbers that holds a float and a whole number is stored with each of its numbers a float, as Neo4j stores it: [1, 2.5] is [1.0, 2.5], written, given as a parameter or saved; a list of whole numbers alone stays one, and a list a statement only reads keeps its whole numbers (see writes.yaml, A list of whole and fractional numbers stored as a property; one worked out as the statement runs, [1, n.x], is #380's). Nor is a negative zero stored on any SQL store: -0.0 is held as 0, so a uniqueness constraint takes it for 0, where Neo4j's keeps the two apart (see writes.yaml, Unique over numbers). Oracle writes the text of the binary double 2^53 as 9007199254740993, in JSON_SERIALIZE, with EXTENDED too, and in RETURNING NUMBER, though the value held is 2^53: no key or reader uses that text, which is why the index key below reads a double by halves.
Temporal values. A date, a time or a moment, a local one or one with an offset, and a duration are each a tag, a JSON object of exactly these keys:
| Key | What it holds |
|---|---|
$t |
The kind: date, localtime, time, localdatetime, datetime or duration. |
v |
The value's canonical ISO 8601 text, below. Two values of a kind are equal, as Neo4j has them, exactly where their v are. |
s |
Its sort key, below: digits only. A duration has none. |
A reader takes no order of the keys: Postgres's jsonb writes them s, v, $t.
The canonical text v:
- A year is numbered astronomically, as ISO 8601 and Neo4j number it (
0000is 1 BC,-0044is 45 BC), with at least four digits, a-before one below 0, and a+before one above 9999 (+10000). Years run from -999,999,999 to +999,999,999. - A date is
YYYY-MM-DD. - A local time is
hh:mm:ss, the seconds always written, then where the second has a fraction a point and the fewest of three, six or nine digits that hold it, as Java'sInstant.toStringwrites one:10:30:00,10:30:00.500,10:30:00.000001,10:30:00.123456789. - A time is a local time and its offset:
Zfor an offset of zero, however it was written (+00:00,-0000), else±hh:mm. Offsets run from -18:00 to +18:00. A named zone ([Europe/Paris]) is not held: a library writes the offset it has at that moment. An offset with seconds (-00:44:30) is not held either: a value given at one is refused by name (see Values). - A local moment is a date,
T, and a local time; a moment a date,T, and a time. - A duration is its months, days and seconds, kept apart as Cypher keeps them (
P1Dis notPT24H), written as Neo4j writes one:P, then the whole years and the months left of the months, then the days, thenTand the whole hours, the whole minutes and the seconds left, with their fraction and no trailing zeros, of the seconds; each part left out where it is 0, each signed on its own (PT-1H-30M), andPT0Swhere all are 0. A week is 7 days; a fraction of a month is 30.436875 days, and a fraction of a day 86,400 seconds, carried to the part below.
The sort key s is worked out of v alone, never of a store's timestamp, which holds neither every year nor nanoseconds; so every writer's s of a value is the same. Within one $t, two values order as their s do as text, under any collation, as Neo4j orders them. Values of two kinds have no order, and neither have durations.
$t |
s |
Digits |
|---|---|---|
date |
the year plus 1,000,000,000, in 10 digits, then MMDD |
14 |
localdatetime |
the same, then hhmmss and the nanoseconds in 9 digits |
29 |
datetime |
the same of the moment in UTC, its date carried a day back or on where the offset takes it there, then the offset in seconds plus 64,800, in 6 digits | 35 |
localtime |
the nanoseconds of the day, in 14 digits | 14 |
time |
the nanoseconds of the day in UTC plus 64,800 × 10^9 (18 hours), in 15 digits, then the offset as a moment's | 21 |
The UTC year of a moment runs from -1,000,000,000 to +1,000,000,000 at the offsets' limits, so the year plus 10^9 is never negative. A time is compared by its UTC nanoseconds of the day without wrapping at midnight, as Neo4j compares it: time('01:00+02:00') is -1 hour, before time('22:00Z'), and the 18 hours keep every one at least 0. datetime('2024-03-15T10:30:00.5+02:00') is held as {"$t": "datetime", "v": "2024-03-15T10:30:00.500+02:00", "s": "10000020240315083000500000000072000"}.
Every tag in the store carries s, and so does every tag a statement gives on. Inside one expression, compiled Cypher holds a value it works out as it runs, such as a sum, without its s, as {"$t", "v"}, and works out the s once, where the value leaves that expression: into a write, a list or a map, an equality, a grouping, DISTINCT, IN or UNION, or a comparison or an order, which read the s of its v where a tag has none. That form never reaches the store or a row.
Keys of maps. A map in flight, a map literal, a parameter's, properties(n), and a property's name in props, may have a key that begins with $, which a user's map may. Such a key is held with one more $ before it, $t as $$t and $$x as $$$x, and read back with one fewer, so an object whose $t is written with one $ is only ever a tag. A key that does not begin with $ is held as it is. Schema DDL indexes the key a property is held at: an index, a uniqueness constraint, a full-text or a vector index over $x is over $$x (SqlStoreDdl.storedKeys), and a search reads that key too; its name and its record keep $x.
Derived fields. Beside props, a node's row has words, Drivine's own per-node fields worked out of its properties: jsonb on Postgres, JSON on Oracle and JSON text on SQLite, null where there are none. Its shape is {"<property>": <word count>, …}, for each property under a full-text index: the count of its words by TextQuery's rule, a run of Unicode letters, digits and _, which BM25 ranks by (drivine4j#665). The rule's version is recorded in the schema marker's inventory, so a change of the rule fills it again. A full-text index's ensure fills it, and writes keep it in step, in a later change under layout 2; nothing in this layout's first release writes or reads it. It is never a property: properties(n), keys(n) and the libraries' loads never see it. A writer of its own leaves it null. Derived fields have no place in props: a key there that begins with a single $ is only ever a tag's.
The count store. Two tables beside the graph's, made empty with the store: the counts of its labels, its relationship types and its full-text words, kept so a count is read, not counted. Their names carry the layout. The writers, the fold and the readers come in the next change under layout 2; nothing in this layout's first release writes or reads them.
| Table | Columns | |
|---|---|---|
<graph>_count2 |
kind, name, n; primary key (kind, name) |
the folded base |
<graph>_count2_delta |
id, an identity; kind, name, d; index <graph>_count2_dk over (kind, name) |
one row for each change to a count |
kind is label, with name the label; type, with name the relationship type; or words, with name <label>.<property>, the total of words' counts behind BM25's average length. kind and name are text (varchar(16) and varchar(1000) on Postgres, VARCHAR2 on Oracle, TEXT on SQLite), and n and d exact integers (bigint, NUMBER(19), INTEGER). A count is n + SUM(d) over its (kind, name), either missing being 0. A writer never updates a row another may: it appends one delta row for each label or type it changes, in the statement that changes it, with no trigger, d the change (a node of two labels made appends +1 for each, one deleted -1). The fold, in one transaction, locks the deltas it reads, adds them to the base and deletes them, which is safe beside writers, as they only append. A label counts each node once for each label it carries, as MATCH (n:L) does; the store's stamp and marker are counted under their own labels, which count leaves out as it does today. A writer of its own appends its deltas so too, or the counts are wrong.
The lock table. One more table beside the graph's, <graph>_lock2, its name carrying the layout: one column, name, its primary key, a short text (text on Postgres and SQLite, VARCHAR2(128) on Oracle), and one row, name = 'schema', put there with the graph and again harmlessly (INSERT … ON CONFLICT DO NOTHING on Postgres, INSERT OR IGNORE on SQLite, a MERGE on Oracle, run within ensuring, which takes the row as put where another session inserted it between the MERGE's look and its insert, ORA-00001). No schema block or compiled statement writes it. An executor about to run a schema change takes a row lock on that row first, from a connection of its own, and holds it until the change is done, so two processes' schema changes to one graph run one after the other: on Oracle, whose DDL commits as it runs and deadlocked on the library cache lock where two changed one table at once, SELECT name FROM <graph>_lock2 WHERE name = 'schema' FOR UPDATE WAIT 10 (drivine4j#777). A writer of its own only makes the table and its row; it locks the row only if it changes the schema beside a library.
The store's stamp. A SQL store is in layout 2 where its stamp, (:DrivineStore {key: 'singleton'}), has layout: 2, and every value in it is held as above. A library refuses a graph whose stamp names any other layout, or none, unless the graph holds no node but the stamp and the schema marker: then it drops the earlier layout's indexes and stamps it 2 (see The schema marker and the stamps).
Other libraries' forms. Before layout 2, other libraries' forms were told apart (#186); since layout 2, a store holds only the module's. A store of layout 1 is refused unless it is empty, and an empty one is stamped again after every object under a layout-1 name is dropped by its name, so a schema manager never meets an item another library made; its records say the form they were made in (see Forms).
Indexes. Each index and constraint Drivine makes over a property is keyed so two values are one key where Neo4j's = says they're equal: a text equal only to a text, a number to a number of the same value (1 to 1.0, and integers past 2^53 exactly; Neo4j's own keeps 2^62 and -2^63 apart from their floats, as Postgres does, which holds a float as its shortest text, where Oracle's and SQLite's key them as one: see writes.yaml, Unique over numbers), a temporal value only to the same value, offset and all, and a date never to the text that spells it. Its name carries the layout, <graph>_ix2_…, _uq2_, _lx2_, _t2_ and _v2_ (see names), so an earlier layout's index is never taken for this one's. Over the property k:
- Postgres:
(props -> 'k'), the property'sjsonb, whose equality is Neo4j's. A statement compares the property as(props -> 'k') = $p::jsonb. - SQLite: two expressions, the class and the key. The class is
COALESCE(props ->> '$."k"."$t"', CASE json_type(props, '$."k"') WHEN 'integer' THEN 'number' WHEN 'real' THEN 'number' WHEN 'true' THEN 'boolean' WHEN 'false' THEN 'boolean' WHEN 'null' THEN NULL ELSE json_type(props, '$."k"') END); the key isCOALESCE(props ->> '$."k".s', props ->> '$."k".v', props ->> '$."k"'), which SQLite compares as Neo4j does within a class, the integer 1 equal to the real 1.0. A list's key is its JSON text, which differs where its items don't ([1.0]and[1]): a statement compares a value that may be a list with the key as a range, from the empty text to the empty blob where it is one, so the index is still searched and the list compared as JSON (#354). - Oracle: two expressions, the class and the key, of Oracle's built-ins alone, for an index over a function the store makes again with
CREATE OR REPLACEwould be disabled (ORA-30554). WithtforJSON_VALUE(props, '$.k.type()' RETURNING VARCHAR2(16))anddforJSON_VALUE(props, '$.k' RETURNING BINARY_DOUBLE), the key isCASE t WHEN 'object' THEN COALESCE(JSON_VALUE(props, '$.k.s'), JSON_VALUE(props, '$.k.v')) WHEN 'number' THEN TO_CHAR(JSON_VALUE(props, '$.k' RETURNING NUMBER), 'TM9') WHEN 'double' THEN CASE WHEN d = FLOOR(d) AND d >= -9223372036854775808 AND d < 9223372036854775808 THEN TO_CHAR(CAST(FLOOR(d / 4294967296) AS NUMBER) * 4294967296 + CAST(d - FLOOR(d / 4294967296) * 4294967296 AS NUMBER), 'TM9') ELSE 'd' || TO_CHAR(d) END ELSE JSON_VALUE(props, '$.k') END: a tag's key or text, read of an object alone, as Oracle's lax path would readsof a list's one item; a number's exact digits, so aNUMBERand a binary double of the same whole value are one key, and any other doubledand its 17 significant digits, which tell any two doubles apart. The class, aVARCHAR2(16), isCASE WHEN <key> IS NOT NULL THEN CASE t WHEN 'object' THEN JSON_VALUE(props, '$.k."$t"' RETURNING VARCHAR2(16)) WHEN 'double' THEN 'number' ELSE t END END, null where the key is, so a value the index holds no key of (a list, an empty text, a text past 4,000 bytes) is constrained by none. A labelled index and a uniqueness constraint's are each of these within theCASE WHENof its label, as before.
The module's ValueLayout, Temporals.key, CypherJson and StoreValues write this, and ValueLayout.read reads it back. host.yaml's key, read and write steps and store.yaml's cases are the conformance checks for a writer of its own: a value whose s, whose tag or whose index expression differs from theirs is a value the libraries read as another.
A schema manager reads what a store has before it makes anything, and decides from what the store says: whether an item it found is the one a spec declares, whether it has the shape the spec declares, and what an error means. Each library decides alike, so a store one library made is ensured alike by another. store.yaml's sections schema, listings, matching, errors and declarations pin drivine4j's decisions, the reference, as its schema grammars made them when they were captured, with no store. The compiler's module (org.drivine.cypher.store.schema, and SchemaItemInfo, EnsureResult, SchemaPolicy, SchemaMode and SchemaDeclarations in org.drivine.schema) makes them, and its tests check it gives these on every target. The listings' statements are the catalog's, pinned once; what this pins is how their rows are read.
They are in store.yaml, and not a file of their own, because they read what store.yaml's statements make and list: a listing's rows are the rows of its catalog queries, and an item is matched against the spec its cases make. marker.yaml is a file of its own because the marker is a contract apart from the DDL; this is not.
schema:
stores:
neo4j: {engine: "Neo4j", supportsIfNotExists: true, supportsNamedItems: true, supportsTextIndex: true, constraintsRequireBackingIndex: false, constraintCreationIsAsync: false, indexOperationsAreAsync: false, awaitIndex: "CALL db.awaitIndex($name, $timeout)", indexState: "SHOW INDEXES ..."}
tuning:
- {spec: {...}, stores: {neo4j: [], memgraph: ["hnswM"], ...}}
samples:
- {spec: {kind: "UNIQUENESS_CONSTRAINT", label: "Person", properties: ["email"]}, limit: 10, stores: {neo4j: "MATCH (n:Person) ...", neo4j4: "...", postgres: "SELECT jsonb_build_object(...", ...}}
listings:
- name: "SHOW INDEX INFO: range, composite, label-only, a vector's backing, and text indexes"
listing: "indexes"
rows: [["label+property", "Proposition", "contextId", 0], ...]
stores: {memgraph: {items: [{kind: "RANGE_INDEX", label: "Proposition", properties: ["contextId"]}, ...], aliases: [[], ...]}}
matching:
pairs:
- {name: "...", existing: {kind: "RANGE_INDEX", label: "Person", properties: ["born", "name"]}, spec: {...}, stores: {neo4j: {identity: true, shape: true}, ...}}
incomplete: [{item: {...}, stores: {oracle: true, ...}}]
builds: [{status: "UNDER CONSTRUCTION", buildFailed: false, buildInProgress: true}]
errors:
- name: "Oracle: ORA-01452, a unique index over rows that repeat a key"
causes: [{sqlState: "72000", vendorCode: 1452, message: "ORA-01452: cannot CREATE UNIQUE INDEX; duplicate keys found"}]
stores: {oracle: {violation: true, blockedByIndex: false, buildTimeout: false}, ...}
declarations:
- {name: "...", specs: [...], items: [...]}
- {name: "...", specs: [...], conflict: "Conflicting schema declarations for ..."}A spec is written as cases write one, with a constraint's nodeIdProperty, the property a violation's sample names each node by, beside. A schema manager hands every rule a spec as it is made, as drivine4j's do: one that ignores case over the lowercased copies of its properties, so that its item is found, matched and sampled over them, not over the properties declared. An item, what a store has, is its kind, label and properties, and what the store gives of its name, dimensions, similarity, analyzer, status, quantizationEnabled, hnswM and hnswEfConstruction; a key left out is null, a value the store doesn't give.
| Key | What it is |
|---|---|
schema.stores |
What each store can do. supportsIfNotExists: its DDL takes IF NOT EXISTS, and where it doesn't, an item is found before it is made or dropped. supportsNamedItems: it keeps an item's name. supportsTextIndex: it has Neo4j's TEXT index, which is refused elsewhere. constraintsRequireBackingIndex: a uniqueness constraint needs a range index over its properties made first. constraintCreationIsAsync and indexOperationsAreAsync: a create or drop returns before it is done, and the listing is polled until it is (a constraint OPERATIONAL or FAILED, an index no longer being built, or gone). awaitIndex: what waits for an index named $name to be built, for $timeout whole seconds, after it is made; and indexState: what gives its state, read when that wait fails for another reason than its timeout. |
schema.tuning |
What a vector spec pins that each store can't make, named as the spec names it, for a library to warn of. |
schema.samples |
What samples the keys that repeat for a uniqueness constraint, limit of them at most, with limit of the nodes of each: each row {properties, count, nodes}, each node its elementId and, where the spec has one that isn't a key, its nodeIdProperty. neo4j4 is Neo4j 4's, which names a node by id(n), not elementId(n). One refilled samples a constraint that ignores case, given as declared, its keys read as its properties lowercased as the copies are filled in (lower, LOWER, SQLite's drivine_lower) rather than as its copies: what a refill the old index refused would have written. |
listings |
A listing's rows, as a driver gives them, and for each store that reads them the items they read as, in order, and the aliases each item's record keeps, the other names it goes by; or invalid: true, where the store refuses them. listing says which of the catalog's queries gave them: indexes or constraints. |
matching.pairs |
An item a store has and a spec, and on each store whether the item is the spec's (identity), and if so whether it has the spec's shape (shape); an item that is the spec's but not in its shape is drift. |
matching.incomplete |
Whether each store makes an item it listed again, as one that lacks part of what it makes. |
matching.builds |
What a listed item's status says of its build, on a store that builds after the statement returns, or of another session's build under way (Postgres's BUILDING): buildFailed, and buildInProgress. |
errors |
A failure's causes, the chain of errors from the outermost in, and on each store whether it says the data violates a uniqueness constraint being made (violation), that a range index is in the way of the constraint (blockedByIndex), that a wait for an index ran out (buildTimeout), or that an index over the same expressions is there already, so a unique index built anew can't stand beside the old one (alreadyIndexed, Oracle's ORA-01408). |
declarations |
An owner's specs, in the order declared, and the items they make, or the conflict they are refused with. |
forms |
A SQL store's listing (store, and listing, which of its catalog's queries), its rows as the store gave them, captured from the store, listed, the items its records read as, its objects' rows left out, and for each of items, a spec, the form the rows say its item is of: current; {stale: <found>}; {newer: <n>}, the later form its record says; or {mismatch: {name, reason, droppable}}. A fresh one is a store the module made, every item of it current, each column of its indexes the column the module writes. |
columns |
Each column the module makes a range index's and a uniqueness constraint's indexes with, by the index's name, on each SQL store, in that store's canonical form (see Forms), so a change to the DDL a column is made with shows here. |
doubleKeys |
On Oracle, the key a range index and a uniqueness constraint hold of a binary double that is no whole number, 'd' || TO_CHAR(d), as Oracle writes each double given as text: captured from the store, so a change to its default format shows here. A library's runner checks each on Oracle with SELECT 'd' || TO_CHAR(TO_BINARY_DOUBLE(:double)) FROM dual. |
indexKeys |
What a range index or a uniqueness constraint of the layout 2 holds of a property, value by value: on Oracle and SQLite its class and its key, as each store's index expressions give them (IndexKey.column), and so what a lookup compares with them. The module works them out of a value given as a statement compiles (IndexKey.of), and the compiler binds the key once, where Oracle's key written over the bound value read it in each branch of its CASE (#286); null where it works out none (a float, a list, a map, an empty text or one past 4,000 bytes), and the store's SQL over the value is compared. keys names the properties a runner stores them under, a key that begins with $ among them, as stored. A library's runner stores each value under each key and checks each store's index expressions over it give the class and key pinned: the drift guard between the module's keys and the stores'. A value pinned null is checked on neither: the store's index may still hold a key of it, a float's say, which the SQL over the value finds. |
A cause. Each error of the chain is written as a library's driver gives it. A SQL driver's error gives its sqlState, and its vendorCode, the driver's own number, 0 where it gives none; a Neo4j driver's error, which Memgraph's are too, gives its code, N/A where it gives none, as drivine4j's driver does, and its gqlStatus; every error gives its message, and any other error only that. A vendorCode or a sqlState is what says an error is a SQL driver's, and a code that it's a Neo4j driver's. drivine4j's Neo4j driver gives the status 50N42 to an error a server sent none for, and a library whose driver gives none reads it as 50N42.
Identity. An item is a spec's when its kind and label are the spec's, and its properties are the spec's in any order. On FalkorDB, which keeps one range index and one full-text index for a label over all its properties, a range or full-text item is the spec's when it holds all the spec's properties, with or without others.
Shape. A vector index is in its spec's shape when its dimensions are the spec's, always compared, so an index whose size isn't given is drift; and when its similarity, hnswM and hnswEfConstruction are the spec's, each where the store gives one and the spec pins one (the store's own options for these win on that store); and on Neo4j, its quantizationEnabled, by the same rule. FalkorDB's efRuntime is never compared. A full-text index is in its shape when the spec asks for no analyzer, the store gives none, or they are the same. Every other kind is always in its shape.
Listings. A row is what the listing's statement gives as one: the value of its one column, a map (a SQL store's JSON, decoded), or, where it gives several columns (Memgraph's SHOW INDEX INFO), the list of them. A number is handed over as a number, whatever type the driver gives it as (a JavaScript Neo4j driver's Integer too), and read as its whole part, 8.0 as 8. Neo4j's listing gives a map a row: a relationship's index or constraint is passed over, and a type it doesn't manage (LOOKUP, POINT); BTREE is a range index; a uniqueness constraint is any type that holds UNIQUENESS; its similarity is read in any case, and one it doesn't know is null; quantization is vector.quantization.type other than NONE, in any case, where it is given, and vector.quantization.enabled where it is not. Memgraph's gives a vector index as a map and the rest as lists of columns, and a label with or without a leading :. FalkorDB's gives a row a label, and its range properties are one item, its full-text properties another, and each vector property an item of its own, each with the row's status; a vector's options are the options of its property where they hold the dimension, or else the row's options where those hold it, or else the options of its property. A SQL store's gives a record a row, and after the records, where its listing names objects, a row for each object under a name Drivine gives (object, its name), which is read for an item's form and is no item; Oracle's alone gives a status, INCOMPLETE for a range index that lacks its labelled index, which is made again.
Forms. Each record the module writes says the form its item was made in, "form": 2 first in its details. A form is how a version writes an item's DDL: the same item, of the same kind, label, properties and uniqueness, made with other expressions. The form is a counter of its own, apart from the store layout: the module's SqlStoreDdl.CURRENT_FORM, which went from 1 to 2 with layout 2's DDL. A schema manager reads an item's form (StoreSchema.formOf) from its record and what the store has under its names:
- First, each object the store has under a name Drivine gives one of the item's must be the one the item is made as: an index, on the node table, unique where the item is a uniqueness constraint and not where it isn't, over the item's properties in order (each column compared with the column the module writes for it, below), and over the nodes of its label alone, or of every node for Oracle's index over the property of every node (
ix). Where it isn't, the item is a mismatch, whatever its record says. On Postgres and SQLite the names are a range index's (ix) and a uniqueness constraint's (uq); on Oracle, a range index'sixand labelledlx, and a constraint'suqand theixit makes too. On Postgres a vector index (v) must also be HNSW, with the operator class of its similarity (vector_cosine_ops,vector_l2_ops), over the embedding of its property, of its size; and a full-text index (t) GIN, withgin_trgm_ops, over the text of its properties in order; each read at the key it is held at ($xat$$x). On Oracle a vector or a full-text index is a table beside the nodes (v,t), its trigger (…_t) and its index (…_i): the table must hold its column (EMBEDDING, aVECTOR;DOC,JSON), the trigger must be on the node table, test no label but the item's, and read the item's properties, in order, at the keys they are held at ($xat$$x), and no others (one that tests a label neither library's way, or none, is not told apart by it), and the index must be of its type (VECTOR; Oracle Text'sDOMAIN) on the table. SQLite makes no object for either: only their records say their form. An object that isn't there is no mismatch: the listing leaves out a record whose object is gone, and Oracle lists a range index that lacks its labelled indexINCOMPLETE. AnINCOMPLETErange index of another form, stale or newer, is not made whole: the module holds only this version's DDL, and would complete it in this version's form, which a stale or a newer item never is here. It is kept,AlreadyMatching, with its form's line atWARN, and a recreate makes it whole. - Then the record: a form equal to the current one is current, an earlier one is stale, and a later one (
form 3, written by a newer version during a rolling deploy) is a newer version's: kept and used as it is, said so once atWARN(… was made by a newer version of Drivine (form 3; this version makes form 2): this version uses it as it is.), with no recreate, as this version would make it again in an older form, and never a problem. A newer form's record is trusted so long as the structure fits: its expressions, which this version may not read, aren't compared, and only that it is an index, on the node table, as unique as the item, by its access method, is; so an older version in a rolling deploy never drops a newer one's index, nor reports it drift. On Oracle a newer vector or full-text index's index over its table beside the nodes must be of its type (VECTOR;DOMAIN), on that table, and its table's columns and its trigger's text aren't read. A name a non-index holds, or an index of the wrong uniqueness, is still a mismatch. A record with no form is current, and so is an item no record lists: nothing is read from an object's text to tell another library's form (see Other libraries' forms).
A remove drops by the record. A known limit: a record's form is taken at its word.
Columns. A range index's or a uniqueness constraint's column is compared with the expression the module writes for it, from the same builder its DDL uses (Postgres's jsonb of the key, SQLite's and Oracle's class and key, Oracle's index over every node, labelled index and unique index), each put in its store's canonical form: its tokens, literals kept as they are, with what the store adds or drops as it prints a column again taken out of both (Postgres's casts, ::text, and parentheses; Oracle's upper case and quotes, a table's alias, FORMAT OSON, RETURNING VARCHAR2(4000), NULL ON ERROR, FALSE ON ERROR, TYPE(LAX), JSON_EXISTS2(…)=1, CASE JSON_EXISTS2(…) WHEN 1 THEN, a path's comment, numbers written as binary doubles, parentheses; SQLite's statement as it was written). A column equal to the item's is the item's. One that matches the module's column written for another property, or another label (on Oracle, for every node too), reads that property; a column of the table reads as itself, and an expression the module never wrote but this reads by its path (a stamp, a layout-1 index) as its property. Anything else is unknown, and no mismatch: a false mismatch would drop a good index under recreateOnDrift. store.yaml's columns pins each column's canonical form, and its fresh forms check a store the module made gives each back so.
The listing gives what the store has under the names after the records, a row a type of object under a name, each a map with object, its name. On Postgres, pg_class: type, its relkind, which a mismatch names as a word (r a table, p a partitioned table, f a foreign table, t a TOAST table, v a view, m a materialized view, S a sequence, c a composite type; i and I an index); and of an index, from pg_index and pg_am, its table, unique, its method, its opclasses, each of its columns as pg_get_indexdef writes it, and its predicate, where, as pg_get_expr writes it. On Oracle, user_objects: type, its object_type; of an index, from user_indexes, its table, uniqueness and indexType, and its columns from user_ind_columns, each an expression's text as the hidden column Oracle makes for it holds it (user_tab_cols.data_default_vc, null where it keeps none), or a plain column's name; of a table its tableColumns, each its name and type; and of a trigger its table and its source, line by line, from user_source. On SQLite, sqlite_master: type, its table, sql, the statement that made it, and unique from pragma_index_list. Each is read by its name in any case, as each store folds an unquoted name to one.
An item of another form is kept and used as it is: a schema manager says so once, at WARN, naming the item, the form it expected and the one it found, and that SchemaManager.recreateAll(<owner>) makes it again. It is never a problem, so it fails no owner under FAIL_FAST, but for a Postgres uniqueness constraint of another form whose index was left invalid, which holds nothing unique and is drift (see the schema plan); it is never reported Created; and it is never written to the marker as stale, since the next run reads it again from the store (drivine-cypher-spec#135). A mismatch is drift, as an item of another shape is on any store (see the schema plan).
Errors. On Neo4j, a violation is a driver's error of the chain with the code Neo.ClientError.Schema.ConstraintValidationFailed or the status 22N79 or 22N80; or a chain of driver's errors whose every status is 50N11 or 50N42, one of them with the code Neo.DatabaseError.Schema.ConstraintCreationFailed. An index in the way is the code Neo.ClientError.Schema.IndexAlreadyExists, and a wait run out Neo.ClientError.Procedure.ProcedureTimedOut. On Memgraph and FalkorDB, a violation is a chain whose messages, joined by |, hold constraint and violat, in any case. On Postgres, it is a SQL error of the SQL state 23505; on Oracle, of the vendor code 1 or 1452; on SQLite, a SQL error whose own message holds UNIQUE constraint failed. No store but Neo4j has an index in the way or a wait. On Oracle, an index over the same expressions there already is the vendor code 1408; no other store refuses a second index so. A violation is a suspicion only: a schema manager reports one where the sample finds a key that repeats, and fails with the error where it finds none.
Declarations. An owner's declarations of one kind, label and set of properties, as each is made (one that ignores case over the lowercased copies), are one item, kept in the order first declared. They must agree, as declared and not as made, on all but a vector index's options for a store, and a constraint's nodeIdProperty: the one declaration that pins a store's options wins, and two that pin different ones are a conflict; otherwise the first constraint with a nodeIdProperty wins, and otherwise the first. The item kept is a declaration as declared. The conflict's message names the kind, label and properties (as made, the copies where it ignores case), the reason, and each distinct declaration it is over, once, as drivine4j writes it: where they disagree on the rest, without a vector index's options for a store or a constraint's nodeIdProperty. The owner's version token is marker.yaml's versions.
Each library keeps three things of its own in a store, and reads what another library kept: the schema marker, a node (:_DrivineSchema {scope, version, items, appliedAt}) for each owner of a schema catalog, holding the version token it last applied and the inventory of the items it applied; the store's stamp, one node (:DrivineStore {key: "singleton", storeId, assignedAt, engine, layout}); and the stamp on each node a save writes. marker.yaml pins them: drivine4j's, the reference, as it gave them when they were captured, for the graph drivine. The compiler's module (org.drivine.cypher.store.marker, and org.drivine.cypher.store.Stamps) writes them, and its tests check it gives these, byte for byte, on every target. Running the statements and reading their rows stay each library's.
They are a file of their own, not a part of store.yaml, because they are a contract of their own: store.yaml is the DDL a store is made with, and this is what the libraries keep in it once it is made, which changes apart from the DDL and is read on every enforce. The constraint each node is kept single by is made as store.yaml makes any, so its statements are pinned here too, as the module's DDL gives them.
graph: "drivine"
marker:
label: "_DrivineSchema"
scopeProperty: "scope"
defaultScope: "schema"
separator: "\n"
runtimePrefix: "runtime:"
staleRuntimePrefix: "runtime-stale:"
constraint: {kind: "UNIQUENESS_CONSTRAINT", label: "_DrivineSchema", properties: ["scope"]}
scopesCypher: "MATCH (m:`_DrivineSchema`) RETURN DISTINCT m.scope"
stores:
postgres: {prepare: "ensuring", read: {version: "...", items: "..."}, readParameters: ["scope"], merge: "...", mergeParameters: ["scope", "version", "items", "now"], deleteAllButNewest: "...", delete: "...", scopes: null, itemsMark: "", constraint: [...]}
keys:
- {spec: {kind: "RANGE_INDEX", label: "Person", properties: ["name"]}, key: "RANGE_INDEX:Person:name:Person_name_range"}
inventory:
- name: "Stale items"
record: {keys: [...], runtime: [...], stale: [...]}
items: "RANGE_INDEX:Person:name:Person_name_range\nruntime-stale:UNIQUENESS_CONSTRAINT:Person:id:Person_id_unique"
read: {keys: [...], runtime: [...], stale: [...]}
scopeFor:
- {owner: null, scope: "schema"}
reads:
- {stored: "items:", read: {keys: ["items:"], runtime: [], stale: []}, oracle: {keys: [], runtime: [], stale: []}}
versions:
- {versions: ["v2", null, "v1"], token: "v1|v2"}
storeStamp: {label: "DrivineStore", keyProperty: "key", key: "singleton", constraint: {...}, parameters: [...], restampParameters: ["key", "layout"], stores: {...}}
storeLayout: {cases: [{name: "...", store: "postgres", stamps: [{layout: null}], foreignNodes: false, objects: [...], verdict: {stamp: {drops: [...], assign: false}}}], firstConnection: [{name: "...", store: "postgres", setup: [...], refused: true, refuse: "...", absent: ["{graph}_lock2"]}]}
nodeStamps: {property: "__drivine.stamp", token: 16, separator: ":", neverStamped: "-", tokens: [...], stamps: [...]}| Key | What it is |
|---|---|
marker |
The marker's label, the property that keys it to its owner, the scope of the owner with no name, and the constraint that keeps one marker for each scope. Each store's statements: read gives the version or items of the scope's newest marker; merge writes it, and makes it where there is none; deleteAllButNewest removes every other marker of the scope, and constraint makes the constraint that keeps one (when a library runs them is its schema manager's, today as drivine4j's SchemaManager decides, and #135's once the manager is in the module); delete removes it; and scopes reads the scope of every marker in SQL, on SQLite, where scopesCypher is read on the others. prepare names the statement of store.yaml's tables a library runs first: Postgres's and Oracle's ensuring, SQLite's first ddl, and none on a graph engine. On Postgres, claim and release take and give up an ensure's turn to build the marker's constraint, as a store.yaml case's do. Every statement but those binds the scope as $scope; mergeParameters are what merge binds, and releaseParameters what release binds. itemsMark is what the inventory is kept behind on the store. constraint is what makes the marker's constraint, without the statements that make the store first, as store.yaml writes a case's create. |
keys |
Each spec's inventory key: <kind>:<label>:<properties joined by ",">:<name>, the name the spec's own, or the one derived from its label and properties where it has none or an empty one. A spec that ignores case is keyed as it is made, over the lowercased copies. |
scopeFor |
An owner's name, null for the owner with none, and the scope its marker is kept under. |
inventory |
A record of keys, the runtime ones and the stale ones; its items, as every store but Oracle keeps them; and what is read back from them. |
reads |
A stored text as a store gives it back, or null where no marker or no items came back, and what is read from it: null for no inventory known, which is not an empty one. oracle is what Oracle reads, where it differs, its mark taken off first. |
versions |
The version tokens of an owner's catalogs, null for one that sets none, and the token the owner records: null where none sets one. |
storeStamp |
The stamp's label, the property it is merged on and that property's value, its constraint, and what assignIfAbsent binds: storeId, a new UUID's text, assignedAt, the moment as ISO-8601 text in UTC, as Java's Instant.toString writes it (2026-11-10T05:00:00.123456Z: the fraction of a second in three, six or nine digits, the fewest that hold it, and none for a whole second), and not a number, as readStamps orders by it, engine, the store's name as the stamp records it, and layout, the store layout the library holds values in: 2 on a SQL store, 1 on a graph engine, whose stamp's statements don't write it. Each store's engine, as the stamp records it, its layout, its prepare, run first as the marker's is, and statements: assignIfAbsent makes the stamp where there is none; readStamps reads every stamp as a map {id, assignedAt, layout}, the oldest first, tie-broken by id, and a library takes the first, so every reader takes the same one; foreignNodes gives a row where the graph holds a node other than the stamp and the schema marker; restampLayout sets the stamp's layout, binding restampParameters, only while it holds none, in one statement; and layoutObjects lists the indexes, tables and triggers Drivine may have made for the graph, each {name, type}. The last two are null on a graph engine. See storeLayout. |
storeLayout |
The verdict on a graph by its store layout (#161), which StoreLayout.verdict gives and each library acts on at its first connection, before any statement runs: before the store's ensuring, ddl or the stamp's prepare makes anything, as this layout's lock table, which they make, is what tells a graph of this layout from an earlier one's where a clear deleted its stamp. A graph whose table of nodes isn't there yet is new, and needs no verdict. Its firstConnection cases hold each library to that order, which no statement of the module's can: each library's runner sets a case up on its store (an earlier layout's table of nodes holding a value, with no stamp), opens a manager of the graph, and checks that its first connection refuses it with the message refuse ({graph} the graph again), not another error, and makes none of absent, this layout's lock table. Each case gives the store, the stamps readStamps read (their layout, null where a stamp has none), whether foreignNodes gave a row, and the objects layoutObjects listed, each {name, type}. Its verdict is "current", the graph to be used; {stamp: {drops}}, the graph, which holds no node but Drivine's own, re-stamped: each of drops run, the earlier layout's indexes, tables and triggers and their catalog records, then restampLayout with what restampParameters names, then the stamps read back and the objects listed again, and judged again, anything but "current" refused, so a store raced by an earlier library is never used on a guess; or {refuse}, the message the library refuses the graph with. Which verdict a graph gets is the layout's verdict's table. A library from before layouts reads no layout, and can't refuse a graph this one wrote: the upgrade notes say not to point one at it. |
nodeStamps |
The node stamp's property and form, two tokens of token hexadecimal digits joined by separator. tokens are the text of a token of 64 bits, as Java's String.format("%016x") writes it. Each of stamps is a stored stamp, its nodeToken (what comes before its first colon, or all of it), its linksToken (what comes after, or null for nothing), whether it is twoTokens (33 long, with its colon at 16), whether it hasNodeToken (longer than 16, with its colon at 16), and the stamp it is withoutLinks. neverStamped is what a save reports as the stamp of a node it found with none. |
The inventory. The items are the inventory keys, in the order they were recorded, joined by a newline. A key to be recreated when next ensured is written behind runtime-stale:, whether or not it is marked runtime too; any other key ensured at runtime behind runtime:; and any other as it is. A stale key is read back as a runtime one too, and a key marked runtime or stale that isn't among the keys is not written. An empty inventory is an empty text, so an inventory of one empty key is read back as an empty one. A key holding a newline, which a label can hold, is read back as two keys: the separator is not escaped, as drivine4j writes it. Oracle keeps an empty text as null, so it keeps the items behind items:, and takes that off as it reads them; a text with no mark is read as it is.
The version token is each catalog's token once, sorted by its UTF-16 code units, as Java sorts strings, and joined by |.
The moment. On Postgres, Oracle and SQLite, merge binds appliedAt as $now, milliseconds since the epoch by the caller's clock. The graph engines write their own clock's, with timestamp(), as drivine4j always has, and bind none. That split is kept on purpose: one clock for every store would be a change of its own, made in every library at once. On a SQL store the newest marker of a scope is the one with the largest appliedAt, a marker with none or with one that isn't a number coming last (Oracle reads a text of a number as the number), and of two with the same, the one made last (nid). A graph engine orders by appliedAt as it orders any value, which every marker it writes has.
A SQL store's layout is 2, a graph engine's 1, and a stamp with no layout, or one that is no number, is 1: on a graph engine that is its own, on a SQL store an earlier one. The verdict is the first row of this table that the graph meets, and every graph meets one: the module's StoreLayoutTableTest asks it of every combination of an engine, the stamps (none, this layout's, an earlier one's, one with no layout, one whose layout is no number, a later one's, this one's beside a later one's or beside one with no layout), whether the graph holds values, the objects the store lists (any set of an earlier layout's generated names, <graph>_ix_…; this layout's lock table, <graph>_lock2; its generated names, <graph>_uq2_…; a later layout's, <graph>_ix3_…, and its lock table, <graph>_lock3; a migration's checkpoint, <graph>_migrate2; another graph's whose name begins with this one's; entries with no name, or one that is no text), and whether the library gives the store's DDL, and holds each to its row. DDL of another store than the graph's is the library's error, thrown. The values are whether the graph holds a node other than Drivine's own; the objects are known as a layout's only by the names it generates, where the library gives the DDL, and a graph engine lists none.
| # | Stamps | Values | Objects | Verdict |
|---|---|---|---|---|
| 1 | of two layouts or more | any | any | refused: stamped by two versions at once |
| 2 | of a later layout | any | any | refused: written by a later Drivine, to be upgraded |
| 3 | any | any | a later layout's (<graph>_ix3_…, <graph>_lock3) |
refused, as row 2 |
| 4 | any | any | a migration's checkpoint, <graph>_migrate2, on a SQL store |
refused: a migration to this layout is under way, to be resumed (see Migrating a store's layout) |
| 5 | of this layout | any | any else | "current" |
| 6 | of an earlier layout, on a SQL store | none | any else | {stamp: {drops}}: the earlier layout's objects dropped, then restampLayout. Nothing is migrated |
| 7 | of an earlier layout, on a SQL store | some | any else | refused: its layout and this one named, and the reload |
| 8 | none or of an earlier layout, on a graph engine | any | any | "current": a graph engine holds its own layout, with a stamp or without, and lists no objects |
| 9 | none | none | none of an earlier layout's | "current": new, or cleared; the stamp is assigned as ever |
| 10 | none | none | an earlier layout's, whatever of this one's beside | {stamp: {drops}}: those dropped, then judged again; the restamp finds no stamp, and changes nothing |
| 11 | none | some | this layout's and an earlier one's | refused, naming both |
| 12 | none | some | this layout's only | {stamp: {drops: [], assign: true}}: a graph of this layout whose stamp a clear (MATCH (n) DETACH DELETE n) deleted with Drivine's own nodes, stamped again by assignIfAbsent, with the library's own id and moment, in place of restampLayout |
| 13 | none | some | an earlier layout's only, or none | refused, as row 7, as layout 1's |
After a stamp verdict's statements a library reads the stamps back, lists the objects again, and asks again: the graph is then "current", which the test checks of every stamp verdict, run as a library runs it, and anything else is refused, so a store raced by another library is never used on a guess. Asked with the objects first listed, a graph of row 10 would be told to stamp again. A drop that fails leaves the graph refused. An earlier layout's objects beside a stamp of this layout (row 5) are the schema plan's: a record it holds of another form is made again by it. A library from before layouts reads no layout, and can't refuse a graph this one wrote: the upgrade notes say not to point one at it; one that stamps a new graph between this library's verdict and its assignIfAbsent leaves a stamp of layout 1 there: on the next connection the graph is re-stamped where it holds nothing yet (row 6), and refused where it holds values (row 7).
A graph of an earlier layout that holds values is refused (the layout's verdict); a migration moves it to this one (#256), a step from each layout to the next, of which there is one: layout 1 to layout 2. Its first part, pinned here, is the rewrite of a row: the module's LayoutMigration.rewrite, or the JSON request {"rewrite": …}, gives each node's or relationship's props, as layout 1 held them, as layout 2 holds them, so every library migrates a row alike. Its second, the run around it, from the judgement of the graph to its stamp, is below (The migration's run).
Layout 1 held a value as the store's JSON alone: a temporal value as its ISO text, bytes as base64 text where a library wrote them so, and on Oracle every float a save wrote as a NUMBER. Which texts were temporal values, and which whole numbers floats, is lost, so a migration is told them: a declaration of a label's property, or a relationship type's, as date, localtime, time, localdatetime, datetime, duration, float or bytes, which a library gives from its model, and the command line as it is told. A text is never taken for a temporal value by its shape, as Neo4j orders a text shaped like a date as a text (#161). A property of no declaration is rewritten only where no declaration is needed:
- a property whose name begins with
$, or a key of a map in its value, is held with one more (Keys of maps); - on Oracle, a
NUMBERwith a fraction is a float, as no integer has one, and is a binary double. So a float id with a fraction, keyed as itsNUMBERin layout 1 (4.5), is keyed as layout 2 keys a lookup of it (d4.5E+000).
A declared property's value is made one of its kind, or of a list of them. A temporal kind's is its text, which must be a text of that kind as ISO 8601 writes it, 10:30 and 2024-03-15T10:30 without their seconds too: a space in place of the T between a date and its time is read as the T (Python's str), and a named zone after an offset (+01:00[Europe/Paris]), which layout 2 holds none of, is dropped; the value is its tag, canonical, with its sort key. A float's is a number, or NaN or an infinity as the texts "NaN", "Infinity" and "-Infinity" a layout-1 save wrote them as. Bytes are base64 text, or a list of whole numbers from -128 to 255, those from 128 read as unsigned, and are the list of their signed values. A declared value that is none of its kind (a text of another kind, a named zone with no offset, an offset with seconds, a number declared a date, a text of no base64), or that the store's layout 2 can't hold (NaN on Postgres and SQLite, an infinity on Postgres), is kept as it was and named with why, as is a property a node's two labels declare two kinds; declarations of one label's property as two kinds are refused. A migration refuses to write a row with such a value, unless it is told to keep them as they are, and then names each.
A row is read into the values it holds, and written as StoreValues.text writes those values for the store, exactly what a layout-2 save or CREATE writes of them, so a row migrated is held as the same values written anew; but a list of whole and fractional numbers, which a save and CREATE store with each number a float (#377), is kept as it was, as the migration writes only the rows a declared change is made in. Each store gives a row's text as it reads its JSON back: Postgres its jsonb's text (props::text), SQLite its text, and Oracle JSON_SERIALIZE(props EXTENDED), whose binary doubles are extended JSON's {"$numberDouble": …}, and which reads the rewrite back with JSON(… EXTENDED). A row with nothing to change needs no write.
from: 1
to: 2
declarationRefusals:
- {name: "...", declarations: [{label: "Person", property: "born", kind: "date"}, ...], refusal: "The property born of Person is declared date and datetime: ..."}
rewrites:
- name: "A property declared a date"
labels: ["Person"]
declarations: [{label: "Person", property: "born", kind: "date"}]
stores:
postgres:
props: '{"born": "2024-03-15", "name": "2024-03-15"}'
result: '{"born":{"$t":"date","v":"2024-03-15","s":"10000020240315"},"name":"2024-03-15"}'
changes: [{property: "born", change: "temporal"}]
invalid: [{property: "...", kind: "...", value: '<its JSON text>', message: "..."}]| Key | What it is |
|---|---|
declarationRefusals |
Declarations a migration refuses, and its message (LayoutMigration.refusal). |
labels |
The node's labels, or the relationship's type, the row is of: a declaration of any of them applies. |
declarations |
What the migration is told: each {label, property, kind}, the property named as the user names it ($x, not $$x). |
stores |
For each SQL store the case is pinned on, the row's text as that store gives it (props), and the text layout 2 holds it as (result), compared exactly: the rewrite's text, the JSON request's value. |
changes |
What was changed, property by property, in the row's order, each escapedKey, double (an Oracle NUMBER made a binary double), temporal, float or bytes; none where the row needs no write, the JSON request's changed false. |
invalid |
Each declared value kept as it was: its property, its declared kind, its JSON text and why. |
The module's LayoutMigrator, or the JSON request {"migrate": …}, runs a migration as the schema plan runs (The schema plan): a round at a time, the library running each round's actions and giving back what came of each, the planner deciding the next from its state alone, which crosses any boundary as JSON. A migration runs while nothing else uses the graph. Its steps:
- Judge. The stamps (
readStamps), whether the graph holds values (foreignNodes), the objects the store lists (layoutObjects), Drivine's own nodes and the size of the graph (own,sizes) are read. A graph the verdict refuses as holding values of layout 1 (rows 7 and 13: itsrefusegivesfrom: 1) is migrated; one that holds a migration's checkpoint is resumed; a current graph (or one of this layout whose stamp a clear deleted, row 12), one that holds no values, and any other refused graph are done with, ascurrent,emptyorrefused. - Scan. Every node, then every relationship, is read in batches by its id (
readBatch,$afterthe last id read and$limitthe request's batch size) and rewritten, with nothing written. The report counts what would change, lists every value not of its declared kind, and names the candidates to declare: each label's undeclared property whose values all read as one temporal kind (texts, or lists of them), or as whole numbers and floats both, counting no value of Drivine's own nodes or of a node another of whose labels declares the property. A graph with a value not of its kind is refused here (invalid), a dry run too, unless the request keeps such values. A dry run then ends with the check of step 5, of possible collisions, read-only, with the report. - Begin. The library makes the checkpoint table,
<graph>_migrate2, first, so a graph stopped at any later moment is refused as part way through a migration (row 4), never read by this layout's lock table as one of this layout whose stamp is gone (rows 11 and 12); then makes the store's tables as this layout makes them (prepare: all of what makes the store's tables at a first connection, Postgres's and Oracle'sensuringand every statement of SQLite'sddl, not only the stamp'sprepare, which on SQLite makes the node table alone; run without the verdict), takes the lock a schema change takes (lock, on a connection of its own, in a transaction held open until the migration is done, and taken once: a migration begun again from its scan holds it already, from its resume; none on SQLite, whose library holds the database's write lock), and, under it, makes the checkpoint table again and reads it, with the stamps and the uniqueness constraints the catalog records (uniques). A checkpoint already there, read under the lock, is another migration's, which is resumed, not begun again. The first checkpoint holds the uniqueness constraints, and then the earlier layout's indexes, tables and triggers are dropped, with their catalog records, before any row is written (StoreLayout.drops, as for a graph that holds no values, the records first, as a migration stopped after an object's drop lists it no more and would never forget its record): a layout-1 unique index would refuse a rewrite that makes two of its keys one (two spellings of one moment) in the write's transaction, so every resume would fail there before the check could name them, and Oracle's layout-1 triggers would keep their side tables in layout 1. A resume runs the drops again before it writes, each safe to run again. A resume prepares too before it takes the lock, as a migration stopped once its checkpoint table was made may have no lock table yet. - Write. Every node, then every relationship, is read again in batches after the checkpoint's id, and each batch's rewritten rows (
update), the values it keeps (each a row of the checkpoint table namedkept:<table>:<id>:<property>), and the checkpoint after its last id are written in one transaction (on Oracle a library binds the checkpoint's$stateas a CLOB, as it may pass 4,000 bytes). So each row is rewritten exactly once, as a rewrite is not to be run again on a row it wrote (it would escape a tag's$tagain), and a migration stopped at any moment, and begun again, resumes strictly after the last batch it committed, under the lock, from the checkpoint it read there. - Check. Each uniqueness constraint the catalog recorded as the migration began, kept in its checkpoint, is checked for keys that repeat, by the schema plan's own sample of what a constraint would refuse (
sampleConflictsQuery), the store's SQL, run uncompiled as the schema plan runs it (aqueryof purposecollisions). After the writes it is exact, over layout 2's values, as the index an application's schema plan makes would refuse them: keys that repeat end the runcollisions, with samples, every value rewritten and the checkpoint kept, so the graph stays refused as part way through a migration. The user corrects the rows with the store's own tools (psql,sqlplus,sqlite3), keeping their values in layout 2, and resumes; the migration offers no statement of its own to run on a graph part way through one. In a dry run the check reads the constraints from the catalog (uniques) and runs over layout 1's values, listing possible collisions: it misses values that become equal only once rewritten (two spellings of one moment) and counts a date and the text that spells it as one. - Finish. Any of the earlier layout's objects still listed are dropped, with their records, as in step 3; every stamp is given this layout (
stampLayout, or, where the graph has none, the stamp'sassignIfAbsentwith the library's own id and moment:assignStamp), and the checkpoint table dropped last (on Oracle withPURGE). The graph is then judged again, and must be current, an ordinary graph of this layout (migrate.yaml'sfinished): the run endsmigrated. No index or constraint is made: an application's schema plan makes its declared ones at its next start, as it makes any it finds missing, and until one does, statements give the same answers, unindexed, so slowly on a large graph; a graph migrated with no application has none. The count store is left empty, as on any fresh store of this layout, its counts seeded when Drivine begins to keep them (#349). Every report from the check on (a dry run's,collisionsandmigrated) says both, as itsnotes; one that ends before it, refused orinvalid, wrote nothing for them to be of.
The checkpoint, the row checkpoint of the checkpoint table, holds the layouts it migrates between, the table and the id the write is after (no table once every row is written), the digest of the declarations, whether it keeps values not of their declared kind, the report so far, the uniqueness constraints the catalog recorded as the migration began (each its label, then its properties), which the check reads, and the fingerprint of the graph as the migration found it: the SHA-256 of Drivine's own nodes and the graph's size, as canonical JSON, none of which a rewrite changes. A resume reads them again, and refuses a graph whose fingerprint differs, one written to under the migration, or declarations, or an option to keep values not of their kind, that differ from those the migration began with. Once every row is written (no table), a resume checks neither: what is left reads the graph as it then is, a row corrected after a collision, or a stamp given this layout before the checkpoint was dropped. While the checkpoint table is there the verdict refuses the graph (row 4), as part way through a migration. Under the lock, a migration that judged the graph beside another that has since finished, and dropped its checkpoint, finds the stamp of this layout and no checkpoint, beginning or resuming, in the table it made again, or made and stopped (the graph refused as part way through until it is begun again): it drops that empty checkpoint table, and ends refused, having written nothing.
An action is one of a query, whose rows the planner reads, each one JSON object read into a map; an execute, in auto-commit; a transaction of executes, all of them or none; prepare; lock; and assignStamp. Each is answered rows, ran or failed, with the store's message, which ends the migration as failed. migrate.yaml's statements pin each store's statements for the graph drivine, and the module's LayoutMigratorTest runs a migration against a store kept in memory, stopped after each action of each round and begun again, and requires every row rewritten exactly once and the rows a migration never stopped gives.
The report a migration ends with, and a dry run's whole answer, is {status, message, tables: {nodes: {rows, changed}, edges: …}, properties: [{label, property, changes: {<change>: <count>}, invalid: [{table, id, kind, value, message}]}], observed: [{label, property, classes: {<class>: <count>}}], candidates: [{label, property, kind, values}], kept: [{table, id, labels, property, kind, value, message}], collisions: [{label, properties, sample}], notes: [<text>, …]}, its status one of current, empty, refused, inProgress (a dry run of a graph part way through a migration), invalid, dryRun, collisions, migrated and failed. A property of a node of two labels is counted under both.
A schema manager decides, from what a store says, what to make, drop, recreate and record, and what to report: drivine4j's SchemaManager, IndexManager and ConstraintManager. The compiler's module (org.drivine.cypher.store.plan) decides it once for every library, as a planner with no store: SchemaPlanner(store, markers), whose start(request, now) and resume(state, outcomes, now) each give a round of actions for the library to run, or the plan done and its report. plans.yaml pins its decisions: drivine4j's, the reference, captured from its managers over a scripted store, with no store, for the design's cases 1 to 29 (drivine-cypher-spec#135), and the module's own for case 30, the form check, and case 31, the safe recreate of a uniqueness constraint (whose Oracle, Postgres and SQLite listings and errors are captured from each store), and for the public IndexManager and ConstraintManager calls (item). The module's tests check the planner gives them, round by round, on every target.
It is a file of its own because it is a contract of its own: decisions, not text. Every statement it names is pinned once, by store.yaml or marker.yaml, and what it pins is when each runs, what it is answered, and what comes of it.
graph: "drivine"
database: "default"
cases:
- case: "2"
name: "An unchanged version"
description: "Everything there already: nothing is made, and the marker is written again as it was."
stores: ["postgres", "oracle", "sqlite", "neo4j", "memgraph"]
catalogs:
- {owner: null, version: "v1", items: ["Range index", "Uniqueness constraint"]}
steps:
- operation: {enforce: "all"}
runs:
- stores: ["postgres", "sqlite"]
rounds:
- now: 1000000
actions:
- {execute: {marker: "prepare"}}
- {query: {marker: "read.version", scope: "schema"}, rows: ["v1"]}
- {query: {marker: "read.items", scope: "schema"}, rows: ["RANGE_INDEX:Person:name:Person_name_range\n..."]}
- {query: {listing: "indexes.RANGE_INDEX"}, items: [{kind: "RANGE_INDEX", label: "Person", properties: ["name"], name: "Person_name_range"}]}
- {query: {listing: "constraints"}, items: [...]}
- now: 1000000
actions:
- {execute: {marker: "deleteAllButNewest", scope: "schema"}}
- {writeMarker: {scope: "schema", version: "v1", items: "RANGE_INDEX:Person:name:Person_name_range\n..."}}
done: 1000000
report:
owners:
- {owner: null, results: [{spec: "Range index", alreadyMatching: {...}}, ...], marker: {version: "v1", items: "..."}}
log:
- ["INFO", "SchemaManager", "Applying schema for owner 'schema' on 'default': 2 items (forceRecreate=false, versionChanged=false, version=v1)"]
- ["INFO", "SchemaManager", "Schema for owner 'schema' on 'default': 0 created, 2 already matching, 0 recreated, 0 needing attention"]| Key | What it is |
|---|---|
stores |
The stores the case runs on. |
catalogs |
The startup catalogs, each its owner (null for the unnamed one, whose marker's scope is schema), its version, and its items. |
policy, buildTimeoutMillis |
The policy, where it isn't the default (FAIL_FAST, nothing recreated, a sample of 10, no index replaced), and the wait for an index built in the background, where it isn't 60 seconds. |
steps |
The operations run in turn, on one store and one manager: each its operation (enforce or recreateAll, of all or {owner}; ensure or remove, of an owner's items; item, a call, ENSURE, RECREATE or DROP, of one spec, with no marker), the runtime items the manager holds then, by owner, and its runs; for an ensure or a remove, runtimeAfter, the runtime items drivine4j's manager holds after it; or, for an ensure refused before anything is applied, refused, the message. |
runs |
The stores that run the step alike, each round's actions with the now it is planned at, the done moment the plan ends at, and the report. |
report |
Each owner applied, in order: what each of its items gave, its unbuilt indexes, its orphans, the marker it wrote (null for none), its failure, and for a remove what it removed; for an item, its result, or whether it dropped one; the database's failure, or the store's error it rethrown, from the action of that round; and every log line at INFO and above, its level, logger and text. |
A spec is a store.yaml case's, by its name, or written as its cases write one. An item, what a store has, is written as store.yaml's listings write one, and a report's item also gives its owningConstraint, the uniqueness constraint whose index it is (A constraint's own index, below), null where it is none's. In a log line, {engine} is the store's name.
Actions. Each round's actions are what the planner gives, in order, and each is answered as the store answers it:
{execute: {create: <case>}},{drop: <case>}and{dropUnnamed: <case>}are astore.yamlcase's statements on the store, a create after the statements that make the store, as its cases leave them out (Postgres's and Oracle'sensuring, SQLite'sddl);{marker: "constraint"}ismarker.yaml's constraint, after them too;{marker: "prepare"}and{marker: "deleteAllButNewest", scope}are the marker's. One no case pins (a FalkorDB constraint's backing index, an index dropped by the name the store gave it) is given whole,{statement}or{native}, with itspurposewhere it isn'tDDL.{leftover: <case>},{build: <case>},{swap: <case>}and{finish: <case>}are a constraint case'srecreatesteps on the store, and{remade: <case>}itsremade, of purposeALIASES. Each runs as one action a statement, andfailed: {at, causes}fails the oneatcounts to.{query: {listing: "indexes.<KIND>"}}and{listing: "constraints"}arestore.yaml'scatalogqueries, answered asitems: each store's rows of them, which read back to them (the module's tests write them, and check every store's rows read back so overstore.yaml's listings).{listing: "onlineBuilds"}is Postgres'scatalogquery of the online builds under way on the node table, answered asrows, a row for each and none where there is none: read before a drop of an index left invalid, and before an ensure run again after a deadlock, which waits until it gives none.{claim: <case>}and{marker: "claim"}take an ensure's turn to build an item, as its case andmarker.yamlpin it (claim), aLISTING, answered asrows:[at], one row of when it was claimed, as text, where the turn is taken, and none where another session holds it;{release: <case>, at}and{marker: "release", at}give it up (release), aLISTINGtoo, bindingat, so it runs where the take ran, on the connection of a caller's transaction too, answered asrows(the claim given up, or none).{marker: "read.version"|"read.items", scope}and{marker: "scopes"}are the marker's,{sample: {spec, limit}}isstore.yaml's sample (refilled: true, its refill's), and{indexState: <name>}Neo4j's; each is answered asrows.delayis the wait before a poll.{readLowerCopies: {label, properties}}, a library's read of its record of the lowercased copies, before it fills in those ofpropertieson the nodes oflabel, is answered asrows, each a record,[label, property].{awaitIndex: {name, timeout}}, Neo4j's wait,{fillLowerCopies: {label, properties}}, the lowercased copies a library fills in and records, and{writeMarker: {scope, version, items}}, the marker's merge with what it binds (the items behind the store's mark, and on a SQL store the round'snow).
An action answered failed gives its causes, as store.yaml's errors write them; the library stops there, and the actions after it in the round are planned and not run. An action given no answer otherwise ran.
Rounds. The library runs a round's actions in order, stopping at the first that fails, and hands back what came of each. The first round reads what the plan decides from: the marker's prepare, each owner's marker (its version where it is versioned or ensuring, and its items), and each listing the items are found in, each statement once; for a remove, the scope of every marker. The rounds after it walk the owners sorted by name, the unnamed one first, each apart from the others, and within an owner the indexes before the constraints, each in the order declared, each finished before the next. A listing read is used again until the plan's own DDL runs, one that failed only once, as drivine4j reads it again after, and a marker's until its scope's marker is written or the read fails (an owner named schema shares the unnamed one's scope, and reads what it wrote, or reads again what failed for it); everything else drivine4j reads after its DDL (an item found after it is made, a constraint's sample, an index's state, a poll of a store that builds in the background) is read then. A state between rounds is JSON, which a library may hold anywhere.
Running a round. In the module's API each action is a PlanAction, and its Purpose says where the library runs it, as drivine4j does:
Query: the statement with itsparameters, after waitingdelayMillis(a poll), compiled first where it iscompiled(the marker's scopes, Cypher, on Postgres and Oracle). A listing (LISTING), an index's state (STATE) and a sample (SAMPLE) run on a schema connection, a marker's read (MARKER,SCOPES) on the auto-commit manager. It is answeredRows: each row a plain value, as Listings says (a single column's value, a map, or a list of columns), a number as a number.Execute: DDL (DDL,ALIASES,PREPARE) in auto-commit, on a schema connection of its own, a FalkorDBNativestatement as its driver's command; a marker's statement (MARKER) on the auto-commit manager. After DDL or aliases, whether it ran or failed, the library tells its own caches that the store's schema changed, as drivine4j'sSchemaGrammar.changed()in afinally. It is answeredRan.ReadLowerCopies, the library's own read of its record of the lowercased copies, as drivine4j'sLowerCopyCatalog.stored(), on a schema connection, after what its statements need made where they need it (drivine4j'sprepare, run once). It is answeredRows, each row a record as the list of its label and its property, as drivine4j'sentriesreads the store's rows; the planner reads those oflabelandproperties, and leaves the ones recorded out of the fill.AwaitIndex(Neo4j), the statement with itsnameandtimeout, on a schema connection, thetimeoutbound as an integer, whichdb.awaitIndextakes (on the JavaScript driver anInteger: a plain number is sent as a float);FillLowerCopies, the library's own lowercased copies ofpropertieson the nodes oflabel, filled in and recorded through the auto-commit manager, each of them, recorded or not, after the samepreparewhere it hasn't run (a fill can come with no read before it); andWriteMarker, the marker's merge with itsparameters, on the auto-commit manager. Each is answeredRan.
An action that throws is answered Failed, its causes as a cause is written, the outermost first; the library keeps the error, and the round stops there. The library runs a failed action again only as an Execute's retry says, and swallows no failure: the planner decides which reads as nothing (a listing, a state, a sample), which runs again (the merge) and which is thrown. An error the library raises itself as it has a connection, before a statement reaches the store, which refuses the graph for every owner alike (drivine4j's ReusedIdsException: a SQLite graph whose tables give a deleted row's id again, refused on every connection) is not answered: the library throws it out of the round, as it is, and plans nothing more, as drivine4j's applyToDatabase rethrows it past its owners' isolation, where the planner would take it, answered Failed, as a store's error, and on a marker's read as the owner's failure, the other owners going on. Every other error of an action, the store's or the library's, is answered Failed. now is the library's clock, in milliseconds since the epoch, at each call: the planner times its waits from it, and a SQL store's marker records it.
The report. The library logs each log line at its level, under its logger, in order. Then, where rethrown is given, it throws the error it kept for the action that failed and ended round round (rethrown.action), as it got it: drivine4j doesn't catch it. Otherwise, where failure is given, it throws its own error (drivine4j's DrivineException) with that message; otherwise the operation succeeded. An ensure or a remove is checked with validate on every database before any is planned (an ensure's and a remove's items merged, the indexes before the constraints, as drivine4j's catalog merges them, two that conflict refused), and a refusal is thrown with nothing run; an ensure or a remove of no items plans nothing. A call plans its databases one at a time, in order of their names, as drivine4j sorts them, and the first whose report throws ends the call: the databases after it aren't planned.
The runtime items. The library holds the items ensured at runtime, each a RuntimeCatalog of an owner on the databases its ensure named, starting with none, and the module says how they change: SchemaPlanner.runtimeAfter(operation, runtime, databases), the operation given to databases (null for all), as the call names them, before the library resolves them to its registered databases. An ensure's items are merged, the declaration that pins engine options or names a constraint's node id property kept, and each replaces the owner's item of its identity on those databases, in its place; a remove takes the owner's items of its items' identities, from every target where it is given to all databases, and from the same databases' otherwise. For an ensure, the library checks each database the call resolves to with validate, in order of their names, over what runtimeAfter gives, SchemaPlanner.runtimeOn(database, runtime) giving each its runtime, and keeps the items only where none is refused (the first refusal is thrown, and it, or a target the library refuses, leaves them as they were), and then plans those databases; for a remove, it checks with validate, keeps what runtimeAfter gives before it resolves the databases (a target refused then leaves the items taken), and then plans each database over it. A library merges no items itself.
What drivine4j decides, which the plan keeps. A version that changed recreates the owner's items, but not one another owner declares; a first version is adopted. An ensure records the version the marker has, and recreateOnStartup doesn't apply to it. A runtime item a later manager hasn't been given is kept in the inventory; a recreate marks it stale, and the next ensure of it recreates it. A key recorded and no longer declared is an orphan, reported and left; a marker with no inventory has none. Only a versioned owner keeps its marker single. Under FAIL_FAST, a drift, a violation, an index in the way or an index not built fails the owner once every item has been tried, and its marker isn't written; under WARN, each is logged and the marker written. A failure drivine4j's manager catches (the marker's, the lowercased copies' fill and record, a TEXT index off Neo4j, a wait that ran out) ends the owner there, its marker unwritten, and the other owners go on; one it doesn't (DDL the store refused for a reason the planner doesn't read, a read of the copies' record) ends the plan, and the library throws the store's error. A remove keeps an item another owner declares or records, its properties in any order.
An index built online. An ensure builds an index so that writes to the graph go on while it is built. On Postgres a range, vector or full-text index is made CREATE INDEX CONCURRENTLY IF NOT EXISTS, a statement of its own, and a uniqueness constraint's index CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS; on Oracle a range index's two indexes are built ONLINE, and a uniqueness constraint's, its range index and its unique index, blocking writes, as before: Oracle's online build of its unique index over keys the data repeats is not refused as a blocking one is, and sessions ensuring it at once then take turns on the table (ORA-00054, waited out), where online builds of its range index beside the blocking build of its unique index deadlock in the library cache (ORA-04020). The index a recreate builds beside the old one is built as before, blocking writes, on both: another session never sees it part built, or left invalid, and so never takes it for the constraint, nor drops it, mid-recreate. Where the store can't build online, the build is run again as one that blocks writes, and so is each after it in the plan, said once at WARN ({engine} can't build an index online here (<the store's message>), so it is built by a statement that blocks writes to the graph's nodes until it is done., then what lets it build online). Postgres refuses CONCURRENTLY in a transaction (25001): Ensure the schema outside a transaction to build it with writes going on. Oracle refuses ONLINE on an edition without it (ORA-00439): An edition with online index builds (Enterprise Edition) builds it with writes going on. A library that ensures the schema on a connection a transaction holds takes a savepoint before a Postgres index build, so the transaction outlives the refusal. A Postgres build cut short or failed leaves its index invalid under its name, which IF NOT EXISTS would pass over: the listing gives its record the status INVALID, and the index too among the objects under Drivine's names, as one cut short before its record was written has none. One invalid while a build of it is under way (pg_stat_progress_create_index; a build another role runs in the database, whose index Postgres hides, is taken as maybe its) is listed BUILDING instead, recorded or not, as its record is written once it is done: an ensure waits for that build, for as long as it waits for an index of its own to build (a uniqueness constraint's for 30 seconds), and reads how it ended, rather than take it as made or drop it, and one still building past that is said so (an index's, as one still being built is) or thrown (a constraint's). An index of another shape under the name is drift at once, being built or not, and isn't waited for; nor is the index a recreate builds first taken as holding the constraint while it is BUILDING. An ensure, or a recreate, drops an index left INVALID and builds it again; the drop waits until no online build of any index is under way on the node table (pg_stat_progress_create_index, as below), and is a block that, where the index is still INVALID (no build of it under way: one another session began since is left, and an ensure of a uniqueness constraint waits for it in the listing, as above), holds the index's table (LOCK TABLE … IN ACCESS EXCLUSIVE MODE) and drops it (DROP INDEX, the one name) only if it is still invalid then, so a valid index another session built meanwhile is left: reads and writes of the graph's nodes wait while an invalid index is dropped, and only then. The block never waits on the table for a build, which would hold up every read and write of it until that build ended, and whose last phase, waiting for the block's snapshot, would deadlock with it. It is said at INFO (Completed Postgres <kind> on <label>[<properties>]: its build had been cut short or had failed, which left it invalid, so it was dropped and built again). A uniqueness constraint's index left so is dropped and built again too by an ensure, said as an index's is (Completed Postgres UNIQUENESS_CONSTRAINT on …); a recreate builds the new one beside it and its swap drops it, as below. A build the data refuses, over a key it repeats, leaves its unique index invalid as well: an ensure drops it by the constraint's name, as above, once the sample bears the violation out, as a 23505 can also be another session's build under the name (pg_class), whose index is left. An ensure whose create passes over an index another session is building, or whose build left it so (IF NOT EXISTS), waits for that build in the listing it reads after, and then finds it valid, or invalid, or not there at all where that session has dropped it since; and makes the constraint again itself, once, as one found invalid, and where it is again not made throws (Postgres UNIQUENESS_CONSTRAINT on <label>[<properties>] was not made: …): it is never reported Created, nor AlreadyMatching, over an index that isn't there, is still being built, or holds nothing unique. Two online builds on one table can deadlock on Postgres, as Drivine's indexes are over an expression and partial, which CONCURRENTLY waits on: Postgres cancels one (40P01), and that ensure, of an index or a constraint, said at INFO (Postgres cancelled a statement of the ensure of <kind> on <label>[<properties>], deadlocked with another session's online build on the same table: ensuring it again once no build is under way there), waits until no online build is under way on the node table (pg_stat_progress_create_index, a hidden build in the database taken as maybe one, for as long as an index's build is waited for), and is run again, which then finds the other session's build of the item, or none, as above; after 3 tries it fails (Postgres cancelled the ensure of <kind> on <label>[<properties>] 3 times, deadlocked each time with another session's online build on the same table (<the store's message>). Nothing was made: ensure it again once those builds have ended.). So that sessions ensuring one item at once, as replicas booting together do, don't all build it at once, each deadlocking with the others and then each dropping and building again the index a cancelled build left invalid, an ensure on Postgres builds an item in its turn: where it would build or drop the item's objects, it first claims its turn, a row of the catalog of the kind CLAIM:<kind>, keyed as the item, which no listing reads (store.yaml's and marker.yaml's claim, an insert that gives a row, at, when it was claimed, where the turn is this ensure's, and none where another session holds it). It polls for its turn while another holds it, for 60 seconds at most, and then reads the store again from the start, as another session may have made the item meanwhile, which it then finds; and it gives up its turn once it ends, made or failed (release, binding at, which gives up only the claim taken then). A claim older than 60 seconds with no online build under way on the node table is taken over, as its session ended, or failed, without giving it up. Where the wait runs out it ensures without its turn, as before turns, said at INFO (Postgres <kind> on <label>[<properties>]: another session's ensure of it held its turn to build it for PT1M: ensuring it without), and where the claim can't be written, at once. An online build reads the table twice; on Postgres it waits for each transaction older than it, so a REPEATABLE READ or SERIALIZABLE transaction left open holds it up until it ends. How an index is built is no part of its form (store.yaml's digest is of the create's blocking statements). SQLite builds an index holding the database's write lock throughout. Neo4j and FalkorDB build theirs in the background, as before; Memgraph's are built as before.
A constraint's own index. Neo4j lists a uniqueness constraint's own index among its indexes, a range index over the constraint's properties under the constraint's name, on 4.4 and 5 alike. An ensure of a range index over them finds it, AlreadyMatching; a drop and a recreate (the public drop() and recreate(), a remove, recreateAll, a version change, recreateOnDrift, recreateOnStartup) that find a range index read the constraints too, and one of a uniqueness constraint of its name they leave to the constraint, as Neo4j refuses to drop it ("Index belongs to constraint"); the constraints are read, not SHOW INDEXES' owningConstraint, which Neo4j 4.4 hasn't: the drop reports nothing dropped and the recreate AlreadyMatching, with the constraint as owningConstraint, each said at INFO (Range index on <label>[<properties>] is the index of constraint <name>, so it is not dropped: drop the constraint to drop it, and … not recreated: …). On Postgres and SQLite, whose listing keeps a uniqueness constraint's unique index as the constraint's, an ensure of a range index over one property that finds none listed reads the constraints, and one over that label and property serves it: AlreadyMatching, under the constraint's name and with it as owningConstraint, nothing made, said at INFO (Range index on <label>[<properties>] is served by <name>'s index: nothing made beside it), and where it ignores case its copies are filled in only where unrecorded, as for one listed. A drop or a recreate there reads no constraints: it finds no range index listed, so a drop drops nothing and a recreate makes one. Oracle lists the range index's record a constraint over one property serves, and, that record gone, makes the range index again, as its index over every node's property, which the constraint's index doesn't stand for, went with it; Memgraph's constraint is not taken to serve one (drivine-cypher-spec#325).
Forms. On a SQL store an item is read with its form (Forms), from the listing it was found in, and nothing more is read for it. One of another form is kept, AlreadyMatching, with a line at WARN under IndexManager or ConstraintManager: {engine} <kind> on <label>[<properties>] is of another form than this version makes (form 2 expected, found <found>): it is kept and used as it is., then SchemaManager.recreateAll("<owner>") makes it again. (recreateAll(null) for the unnamed owner), or, for a public call, Call recreate() to make it again. One a newer version made (a record of a later form) is kept, AlreadyMatching, with a line at WARN: {engine} <kind> on <label>[<properties>] was made by a newer version of Drivine (form <n>; this version makes form 2): this version uses it as it is., and no recreate is suggested. A Postgres uniqueness constraint of another form, older or newer, whose index was left INVALID (or ends so, once a build of it under way is waited for) is not kept as AlreadyMatching: nothing holds the data unique, and this version never makes another version's item whole, so it is drift, with a line at WARN under ConstraintManager, Postgres UNIQUENESS_CONSTRAINT on <label>[<properties>]: its index was left invalid in another version's form, so nothing holds the data unique. Call recreate() to make it again in this version's form (destructive)., and nothing is run on it: under FAIL_FAST it fails the owner, under WARN it is logged and the marker written, and a recreate (recreateOnDrift, a version change, recreateAll) makes it again in its place, in this version's form. A mismatch is drift, with a line at WARN that names what stands under the name and what it is, never Created and never made over: under FAIL_FAST it fails the owner, under WARN it is logged and the marker written, and a recreate (recreateOnDrift, a version change, recreateAll) drops it by its name, DROP INDEX IF EXISTS <name>, logs that it dropped an index Drivine didn't make under its name, at INFO, reads the listing again, and makes the item, where it is an index (a uniqueness constraint on a SQL store is made in its place, below); where it isn't, a recreate drops nothing and the item stays drift.
A uniqueness constraint made again. On a SQL store, every recreate of a uniqueness constraint (recreate(), recreateAll, recreateOnDrift, a version change, recreateOnStartup, a stale runtime item, and so a mismatch's drop and a form-stale item's) makes it in its place, as store.yaml's recreate pins it (drivine-cypher-spec#135 §4), so data it refuses never leaves it with no index. First, what stands under its names that isn't an index fails it as drift, as above, with nothing run; another index under one of its names but its unique index's (Oracle's index over the property of every node) is dropped by its name and said so at INFO. Where nothing stands under the constraint's index's name and the index a recreate builds first does, unique and of the constraint's shape, and not left invalid (Postgres's INVALID, an online build of it cut short, which this version's recreate never leaves, building it blocking, but a REINDEX CONCURRENTLY or another version's can; it holds nothing unique: an ensure makes the constraint, a recreate drops it, leftover, before its build, and a remove drops it as below), a recreate was cut off after its old index was dropped: A recreate finishes it: finish puts it in its place, said at INFO (Put {engine} index <name>_r in the place of <name> for <kind> on <label>[<properties>], finishing a recreate that was cut off), it is never dropped first, and the constraint's create and remade follow, the listing read again, Created, and Recreated. An ensure (and an enforce's) that finds it so keeps it, AlreadyMatching, as it holds the data unique, runs nothing (but, where it ignores case, the fill of its copies an ensure runs where no record is listed), fails no owner, and says so once at WARN: {engine} <kind> on <label>[<properties>] is held by <name>_r, the index a recreate of it builds first, and nothing stands under <name>: that recreate was cut off, or is still running. It is kept and used as it is. SchemaManager.recreateAll(<owner>) finishes it. (Call recreate() to finish it. for a public call). It doesn't finish it, as the recreate may be running still in another session, between its drop and its rename: a finish then would rename the index that session is about to, whose rename would then find none, and leave the constraint with no index. A remove (and the public drop()) that finds it so drops it, leftover, and forgets the record, drop, with Dropped {engine} <kind> on <label>[<properties>], held by <name>_r, the index a recreate of it that was cut off built first at INFO (plans.yaml case 31t): left standing, it would go on refusing a key the data repeats for a constraint no one declares. Two recreates running at once are still a race of that kind, which the plan doesn't guard against: a library runs one recreate of an owner at a time, and two processes recreating at once may leave a constraint with no index, or a leftover, that the next recreate makes good. Otherwise: leftover (what a recreate left under the name it builds under where the constraint's own index still stands, or of another shape: SQLite's swap cut off before its last drop, or Postgres's swap refused, leaves one beside the constraint's index, which the next recreate's leftover drops, and a recreate cut off between its build and its swap leaves it so too. It is unique, and may be of another shape than the constraint's, as one a newer version built, so it may refuse writes the constraint allows: an ensure (and an enforce's) that finds the constraint's own index AlreadyMatching with it beside drops it on Postgres and SQLite, leftover, with Dropped {engine} index <name>_r beside <name> for <kind> on <label>[<properties>]: a recreate built it first and never swapped it in, and left standing it would go on refusing writes at INFO (cases 31x, 31y), as their swap needs it not to be there: Postgres's is one block, which rolls back whole where the index to rename is gone, and SQLite's makes the index again under its name. Oracle's swap drops and then renames, so there it is kept, and said once at WARN ({engine} <kind> on <label>[<properties>] has <name>_r beside <name>, the index a recreate of it builds first: that recreate was cut off, or is still running. It is kept, and refuses writes as it was built to., then what makes it again); it can stand there only beside an index of another shape, as Oracle builds none beside the module's own (ORA-01408); a remove, and the public drop(), run leftover before the drop by the record, so one isn't left refusing keys for a constraint no one declares, case 31w); the lowercased copies filled in again, where it ignores case; build. A refill the old index refuses is a key the data repeats: it is sampled with the keys lowercased as the fill lowercases them (schema.samples' refilled), as the copies hold none that repeats, reported Violation, and thrown where the sample finds none, nothing built, the old index and its record as they were. Data that refuses the build is sampled, and reported Violation, as a create's is, and thrown where the sample finds no key that repeats: the old index and its record are kept as they were. Otherwise swap, with the line of what it dropped where that was a mismatch (at INFO); where the record found lists the constraint's properties in another order than it is declared in (found all the same, as identity is a set; an index under its own name still standing, the one built first beside it is a leftover, not a recreate cut off after its drop, case 31v), its own index, under the name that order gives, is dropped and the record forgotten by it (the constraint case's drop), said at INFO, and its aliases set on the record made (aliased), so one index and one record are left; Oracle's ORA-01408 drops by that record too; then the constraint's create, which finds its index made, remade, the listing read again, and Created {engine} <kind> on <label>[<properties>] at INFO, Recreated from the record as first found. Oracle builds no second index over the expressions of one there (ORA-01408; a plain index over them it builds beside, which the data alone refuses): the build so refused, the index under the constraint's name is dropped first (DROP INDEX IF EXISTS <name>) and the create and remade run as above, a violation of the create sampled as one of the build's. Its line, Dropped {engine} index <name> of <kind> on <label>[<properties>] before making it again: Oracle builds no second index over the same expressions (ORA-01408), so the new one can't be built beside it., goes on at INFO, It was made as this version makes it, so the data it held unique is unique still, but for a write in between, where the index was of the current form; and at WARN, It was <its form>, not made as this version makes it: where the data repeats a key, <label>[<properties>] is left with no index until the constraint is made, where it was of another form, a newer one, or no index Drivine made. A step after the build that fails (the swap, the rename, the record) is the store's error, thrown as any DDL's: Postgres's swap is one transaction, and leaves the old index and the new one beside it; Oracle's, refused at the drop, leaves both, and refused at the rename, only the new one, under the name it was built under, which the next recreate finishes (an enforce keeps it, as above); SQLite's leaves the new one, and the old where its drop hadn't run; and a failed remade leaves the index in place and its record as it was. Each is pinned with what the store gave (plans.yaml case 31e), as are a refill refused for another reason than a key that repeats (31o, the owner failed with the fill's error), a build the data refused whose sample then finds no key that repeats (31p, the refusal thrown), a build refused for another reason (31q, thrown), an ORA-01408 followed by a create a write in between refuses (31r, the violation reported, the constraint left with no unique index), and an ORA-01408 over a newer version's index (31s, at WARN). The graph engines drop and then make, as drivine4j does: Neo4j refuses a second constraint over the same schema, and none of them renames one.
Where reads differ. A plan reads less than drivine4j: drivine4j reads a listing again for each item, and runs the marker's prepare before each owner's first marker statement, where the plan reads a listing once until its own DDL runs, and prepares once. It also reads a listing in the first round that an item made before its use then makes it read again. What is written, and in which order, is drivine4j's, and so is every log line.
Compiled SQL calls functions each library registers on SQLite (drivine_equal, drivine_add …, and regexp, which SQLite calls for its REGEXP operator), and each library reads the rows itself, where a store's JSON holds no NaN or infinity: SQLite writes NaN as the tag {"$numberDouble":"NaN"} and an infinity as 9e999, and Postgres, in the rows of a statement given NaN or an infinity, writes stand-in numbers past any double's. That code is each library's, not the compiler's, so host.yaml pins it, run by each library's unit suite with no container: on an in-memory SQLite with its functions registered, and on its row reader. The functions are moving into the compiler module, as a SQLite extension each library loads (#108): the module runs the function steps of those it implements too, on every target and through SQL on the extension, and lists the rest as not yet ported (see the compiler's README).
- name: "drivine_add"
description: "drivine_add(a, b) is Cypher's a + b of two JSON values, as JSON text: NaN adds as a number …"
pairs:
writes.yaml: ["NaN worked out from stored infinities", "NaN and the infinities written"]
steps:
- function: "drivine_add"
args: ['9e999', '-9e999']
result: '{"$numberDouble":"NaN"}'A case has a name, unique in the file, a description, pairs, and its steps. A step has an optional note and is one of these:
| Keys | What it checks |
|---|---|
function, args, result |
The SQLite function, called with args, gives result. Each argument and the result is a SQL value: a text (the JSON text a function of JSON values takes, a rounding mode, or a regular expression), a whole number, a float, or null. A text result is compared exactly. regexp is called as SQLite calls it for text REGEXP pattern: regexp(pattern, text), the pattern first. |
function, args, error |
The SQLite function, called with args, fails, and its message holds error. |
function, ascending |
The function of one argument, called on each of ascending in turn, gives values that ascend as SQLite orders them, each before the next. For drivine_sort_key of a list, whose text is each library's own: only its order is pinned. |
key, result |
The sort key s of the temporal value whose kind's word and canonical text key gives, {kind, v}, is result, a text of digits, or null for a duration (see Values in the store): as Temporals.key gives it, as each store's SQL works it out, and as any writer of the store layout must. |
write, result |
The value write gives, {store, value}, is held in that store's layout as the JSON text result, compared exactly: the module's write for the store (StoreValues.text, the JSON request {"write": <value>, "store": <store>}), the text a compiled statement that writes binds it as, which a library's save writes too (see Values in the store). store is postgres, oracle or sqlite; value is written as the other files write a value ({$float: "NaN"}, {$bytes: "AQL/"}, {$temporal: "DATE", text: "2024-01-01"}, {$map: {…}} for a map with a key that begins with $). |
write, error |
The value is refused, with the message error, exactly: a float the store holds as no number, with the message a statement that writes is refused with, a temporal value whose text is no value of its kind, an offset with seconds among them, with the message a statement given it is refused with, or a store that has no layout of values to write, left out or a graph engine's. |
read, value |
A column of a compiled statement's rows reads as value, as PersistenceManager.query gives it: a temporal value's tag as its text, and a map's key held with one more $ as the key written (see Values in the store). read has the store (postgres or sqlite), nonFinite (whether the statement was given NaN or an infinity, which the compiled statement says), and the column's raw JSON text: a text, or a list of texts and $repeats written one after the other. |
A step noted drivine4j#421 (layout 2) pins a timestamp of nine fractional digits, which SQLite's timestamps don't have until layout 2: each runner holds it until its store's do, as it holds any step its code can't pass yet. The module's runners (HostSpecTest, check/extension-native.py) run a held step and require it to fail, on its result or as the function fails, and fail where it passes, so the hold comes off once the store keeps nine digits.
Values are written as in the other files: {$float: "NaN"} is NaN, {$repeat: …} a text repeated, and any other map, {$numberDouble: "2.0"} among them, is the map it is. A write step's value is also read as a request's parameter is (see the compiler's README): {$bytes: "<base64>"} is bytes, {$temporal: <kind>, text: …} a temporal value, and {$map: {…}} the map it holds.
Pairs. A host function and the compiled statements that call it are one contract: a change to one alone changes what a statement gives. So each case's pairs names, by file, the store cases in queries.yaml, writes.yaml and temporal.yaml that rely on its steps, and a runner checks, before the case passes:
- that each case named is in its file;
- for each function its steps call, that a step of a paired case, run on SQLite, compiles to SQL that calls it: for
regexp, SQL that holds the operatorREGEXP; - for each
storeandnonFiniteits steps read, that a step of a paired case, run on that store, compiles with thatnonFinite; - for its
keysteps, that a step of a paired case makes a temporal value as the statement runs; - for each
storeitswritesteps name,postgres,oracleorsqlite, that a step of a paired case runs on that store (by itsonlyandexcept) and its Cypher writes, holdingCREATE,MERGEorSET: whether it compiles there or is refused, as a statement given a float its store holds as no number is, so Postgres's refusedCREATEcounts. Awritestep with no store, or a graph engine's, is refused before any store is written, and needs no pair.
A change to a host vector names the store cases it moves, and a change to a paired store case shows which host vectors it leans on. A host case with no pairs, or whose pairs no longer rely on it, fails. But for drivine_host_version's, which no compiled statement calls: a library calls it itself, once on each connection, to check its functions are those the compiler's SQL calls, so its case has no pairs (pairs: {}), and its steps are the only ones a runner checks no pair for.
Regular expressions. On SQLite a regular expression the compiler writes, given to REGEXP, drivine_regex_group or drivine_regexp_replace, is read by each library's own language: Java's, JavaScript's or Python's, each rewritten from Java's as its library rewrites it. So the compiler writes them only with what host.yaml's regexp case pins every library reads alike: ^ for the text's start and \z for its end; each class written out, with \t, \n, \f, \r, \xNN and \uNNNN and ranges of them, and [0-9] for a digit; a backslash before a character of the syntax (^$\.*+?()[]{}|/) for the character; groups, (?:…), alternatives, and ?, *, + and {m,n}. It writes none of what the libraries read otherwise: \s, \d, \w, \b or any other class escape, which are Unicode's on drivine4j and partly ASCII's on Drivine for TypeScript; $, which in Java matches before a line break that ends the text too; \Z; ., which in Java matches no U+0085; a class within a class, or && in one, which Java reads as their union and intersection; a POSIX class ([:alnum:]) or \p{…}; or a flag group. A pattern a statement gives =~ is its own, read as Java reads it: the compiler's group around it, ^(?:…)\z, is written as above, and a leading (?i) in it is pinned. The compiler's tests check every pattern it writes on SQLite over the whole spec.
A full-text search (a library's loadMatching) is given a label, the properties of its full-text index, a query as a user writes it, and topK, and gives the best topK nodes that match, each with a score. On a graph engine the store's own index answers it: Lucene's on Neo4j, Tantivy's on Memgraph, RediSearch's on FalkorDB. On a SQL store Drivine parses the query and scores the nodes itself, by BM25, or through Oracle Text where Oracle has the index. search.yaml pins what a search finds, the same on every store, and how close each other store's ranking must come to Neo4j's. Neo4j's is the one the others are held to.
graph: |
CREATE (:Doc {id: 'graphs', body: 'graph databases store graph data'}), ...
indexes:
- {kind: "FULLTEXT_INDEX", label: "Doc", properties: ["body"]}
cases:
- name: "Words are ORed"
description: "A node that holds any of a query's words is a hit."
steps:
- search: {label: "Doc", properties: ["body"], query: "soup kiwi", topK: 50}
ids: ["kiwi", "soup"]
today: [{stores: ["falkordb"], ids: [], issue: "liberation-data/drivine4j#666"}]
ranking:
corpus: "search/docs-corpus.json"
write: "UNWIND $rows AS r CREATE (:Article {id: r.id, title: r.title, body: r.body})"
index: {kind: "FULLTEXT_INDEX", label: "Article", properties: ["body"]}
search: {label: "Article", properties: ["body"]}
top: 5
recorded: {image: "neo4j:2026.10.1", digest: "neo4j@sha256:…", statements: [...]}
bars:
- {stores: ["postgres", "oracle", "sqlite"], sameTop: 17, overlap: 4.3, target: {sameTop: 21, overlap: 4.5}, measured: {...}}
- {stores: ["neo4j"], sameTop: 21, overlap: 4.95, note: "...", measured: {...}}
- {stores: ["memgraph"], sameTop: 16, overlap: 4.09, note: "...", measured: {...}}
- {stores: ["oracle-text"], sameTop: 5, overlap: 2.57, note: "...", measured: {...}}
- {stores: ["falkordb"], sameTop: 9, overlap: 2.04, note: "...", measured: {...}}
queries:
- {query: "GraphObjectManager", neo4j: ["legacy-graph-object-manager#1", ...]}The stores. A search runs on neo4j, memgraph, falkordb, postgres, oracle, oracle-text and sqlite. Oracle is two stores here: oracle has no full-text index made, and Drivine scores by BM25; oracle-text has the index made, which is Oracle Text, and Oracle Text scores. The group graph is the three graph engines (Neptune has no full-text search, and runs none of the file), and sql is postgres, oracle, oracle-text and sqlite. A step's only and except name them as the other files' steps do.
The exact cases. For each store, a runner empties the graph, makes each of indexes (as the schema API ensures a spec, store.yaml's DDL; on oracle, none), writes graph, and waits for the indexes to hold it (CALL db.awaitIndexes() on Neo4j; Oracle Text's index holds what is committed, so the write is committed first). On Postgres it runs every case twice, with the indexes made and without them: the index only narrows the nodes read, and both give the same hits. Each step is then one search, with no threshold, and its hits named by their id property:
| Key | What it is |
|---|---|
search |
The search: its label, the properties of its index, its query, and topK. |
ids |
The hits, in any order. |
ranked |
The hits, best first. |
error |
The message the search is refused with. |
only, except |
The stores the step runs on, or doesn't. |
today |
Not read. What a store gives today where it differs, and the issue that makes it pass: stores (stores or groups), and ids, ranked or error, or none of them where it gives anything but the step's. A library holds such a step in its own HELD list, so the step says what every store is to give. unfiled marks a difference with no issue yet. |
note |
Why the step is there. It is not read. |
No step rests on how a store breaks a tie. On Neo4j no two hits that a ranked step orders, or that topK keeps or leaves out, score alike; where another store scores them alike, the step leaves that store out (except), and where the tie is only of order and not of which hits topK keeps, a step of ids pins its hits there.
A case says what Neo4j's index gives, with Neo4j's default analyzer, standard-no-stop-words: words ORed, a phrase, + and -, AND, OR and NOT (and &&, ||), word*, field:word, escapes, case and Unicode. A refusal is Drivine's own, not Neo4j's: a query with no word to look for (operators, punctuation or excluded words alone) and an empty one are refused with A full-text query needs a word to look for: '<query>' has none., on every store, where Neo4j gives no hits, every node, or a parse error.
The ranking. Scores differ between stores in scale, so the ranking pins ids and order, not scores. A runner writes the corpus (search/docs-corpus.json, a section of Drivine's docs to a node, {id, title, body}) with write, bound as $rows, makes index and waits for it as the exact cases do (on oracle, none; on Postgres, with it and without it), and asks each of queries for its best top, by search. Over the queries whose neo4j is not empty, it counts those whose top hit is Neo4j's, and the mean of how many of the top five are in Neo4j's top five, as sets. Neo4j is asked too, and has a bar of its own at full agreement with the lists, so a Neo4j image that has drifted from the recording fails loudly: the other stores are measured against the lists, not against Neo4j, and wouldn't show it. On each store of a bar, the count must be at least sameTop and the mean at least overlap. target is where the BM25 stores' bar goes once BM25 measures a text's length in words (liberation-data/drivine4j#665), and the other bars have none yet; measured is what each store gave when the file was recorded, and is not read, and so is a bar's note. Each store's bar is what it gave when the file was recorded, rounded down, so a change can't make any store rank further from Neo4j, and no higher than the store gives however it breaks a tie between hits that score alike, at the top or at the fifth, so a runner needn't break ties of its own (Oracle Text's and FalkorDB's bars are lowered so, and Memgraph's is the worst of its recorded runs, as their notes say; on Neo4j no two of a query's first six hits scored alike); Oracle Text's is its own, as it isn't BM25, and Memgraph's and FalkorDB's are their search engines'.
The corpus was drivine4j's src/test/resources/fulltext/docs-corpus.json at dd4a66f0, byte for byte (SHA-256 865f814ceaba8ade6ba604dcb3614aa344529c0f701991c40b6c169ee4c9c0e5); drivine4j's FullTextRankingAgreementTest now reads this copy: the sections of Drivine's own docs at 3a8b0b12, as that test says, each a node. Its queries are that test's.
Recording Neo4j's ranking again. recorded names the image the neo4j lists were recorded on, by digest, the version it reported and the date, and the statements that record them: the index made and filled, and each query asked of it. When Neo4j's pin moves, run them on the new image, through the test lock, and write the lists again with the new image's name and digest; a change of a list is Neo4j's and is said so in the pull request. drivine4j's Neo4jRanking asks Neo4j the same way.
A vector index finds the nodes of its label whose embedding, a list of numbers, is nearest a vector. In Cypher, Neo4j searches one with the procedure db.index.vector.queryNodes(name, k, vector), which yields each node and its score, and with Cypher 25's SEARCH clause, which binds them to a MATCH's variable; vector.similarity.cosine() and vector.similarity.euclidean() score two lists with no index. A graph engine runs the statements itself. On a SQL store Drivine is to compile them, to a search of the store's own vector index where it has one (#160). vector.yaml pins what each gives, and Neo4j's answer is the rule.
graph: |
CREATE (a:Chunk {id: 'a', embedding: [1.0, 0.0, 0.0], lang: 'en'}), ...
indexes:
- {kind: "VECTOR_INDEX", label: "Chunk", properties: ["embedding"], name: "chunk_cos", options: {dimensions: 3, similarity: "COSINE"}}
cases:
- name: "A filter after the k nearest"
description: "A WHERE on what a search finds is applied to its k nearest, after they are found, so it may leave fewer than k."
steps:
- cypher: "CALL db.index.vector.queryNodes('chunk_cos', 2, [1.0, 0.0, 0.0]) YIELD node, score WHERE node.lang = 'en' RETURN node.id, score"
rows: [["a", 1.0]]
- cypher: "MATCH (c:Chunk) SEARCH c IN (VECTOR INDEX chunk_cos FOR [1.0, 0.0, 0.0] LIMIT 2) SCORE AS s WHERE c.lang = 'en' RETURN c.id, s"
rows: [["a", 1.0]]
today: [{stores: ["sql"], issue: "liberation-data/drivine-cypher-spec#160"}]The stores. The file runs on neo4j, postgres, oracle and sqlite, and sql names the last three. Memgraph and FalkorDB run none of it: each has a vector search procedure of its own (vector_search.search on Memgraph, db.idx.vector.queryNodes on FalkorDB), and Drivine is to refuse these statements there, naming it (#214). Neptune runs none of it either.
A case. For each store, a runner empties the graph, makes each of indexes, a schema spec as store.yaml writes one, its name the one the statements search by, as the schema API ensures a spec, writes graph, and waits for the indexes to hold it (CALL db.awaitIndexes() on Neo4j). On Postgres it runs every case twice: with the indexes made, and with each index's record kept and its HNSW index (the record's object) dropped, as SQLite always is, so that the search measures every node of the label. A statement finds an index by its name through the record, which holds its label, property, dimensions and similarity, so the record is kept in both runs, and read as it is: the schema listing leaves out a record whose object is gone (Forms), so the runner drops the HNSW index after it makes the indexes, before each case. The index only narrows the nodes read (exactly, on a graph this small), and both give the same rows. A step is a statement, as in queries.yaml, with its cypher, parameters, rows or error, only, except and note, and today, as in search.yaml: what a store gives today where it differs, with rows, error, or neither where it gives anything but the step's, and the issue; a library holds such a step in its own HELD list, and today is not read. The SQL stores compile db.index.vector.queryNodes(), given the graph's vector indexes as its catalog records them (GraphOptions.vectorIndexes, see the compiler's README); every step of SEARCH, vector.similarity.cosine() or vector.similarity.euclidean() that gives rows is held on them today. An error is true: the step's note says Neo4j's code and message, and each SQL store's own message is not pinned yet.
Comparing rows. Rows are compared as queries.yaml compares them, except that each float is compared within 1e-6, as Neo4j's scores are float32s (0.8 comes back 0.800000011920929), and a run of consecutive rows whose last column, the score, is the same number within 1e-6 is compared in any order, as if sorted by id. Neo4j's own order of hits that score alike is not stable: of the same two hits, a search for 10 gave a then d, and one for 2 gave d then a. No step rests on which of two such hits a search keeps: where k cuts between hits, they score apart.
What Neo4j gives. A cosine index scores a node (1 + cos) / 2, from 1.0 for the same direction to 0.0 for the opposite; a euclidean index 1 / (1 + d²). A node is in an index if it has the index's label and its property is a list of numbers of the index's size, and on a cosine index not a list of zeros. A filter on what a search finds, a WHERE after its YIELD, or before or after SEARCH, is applied after the k nearest are found, so it may leave fewer than k. What a search finds is read on as any node is: matched out from, optionally, in EXISTS { }, and aggregated. The cases were recorded on Neo4j 2026.10.1 community, in Cypher 25, its default.
Refused. Of Neo4j's vector search, a SQL store refuses by name what it has no counterpart of (see refusals.yaml): db.index.vector.queryRelationships(), as it makes no vector index on relationships; Neo4j's VECTOR type and its functions, vector(), vector_distance(), vector_norm() and vector_dimension_count(), as an embedding there is a list of numbers; db.create.setNodeVectorProperty(), for a SET of the list; and a k written as a float, 1.5 or 2.0, as Neo4j refuses it. A search by db.index.vector.queryNodes() is compiled, and refuses before it runs what Neo4j fails: a k that is no whole number from 1 to 2147483647, an index the graph has not made, and a vector written or given that is null, of another size than the index's, or of no length on a cosine index.
A search among ids (a library's loadNearestAmong(ids, topK, threshold, searchK, neighbours)) finds, for each of a set of nodes named by id, its topK nearest among the others of the set, each with its score, as vector.yaml scores it. It is the shared module's VectorSearch with an among (see the compiler's README). among.yaml pins what each store gives.
- among: {label: "Doc", property: "embedding", ids: ["a", "b", "c", "d", "e", "f"], topK: 1, neighbours: "after"}
rows:
- {seed: "a", near: [{id: "b", score: 0.9}]}
- {seed: "b", near: [{id: "c", score: 0.98}]}
...Each pair once. With neighbours: "after", each node keeps only the nodes after it in the ids, so each pair comes back once, under the node given first. The direction is kept before each node's topK are cut. So a node whose nearest all come before it still keeps its best after it, and no pair is lost to the cut, as it would be were each node's nearest taken first and the earlier ones dropped after: the file's case "A node's nearest all before it" is that trap. The library puts the ids in its id type's own order first, so the order a caller gives them in changes nothing. The module compares no two ids: it keeps a node's neighbours by their places among the ids alone, so every store agrees.
Exact, or through the index. With no searchK, every pair of the set is measured, through no index, whichever neighbours a node keeps, so the answer is exact on every store, and its cost grows with the square of the set's size (#408). With a searchK, each node asks the vector index for its searchK nearest, where one is made, and keeps those among the set (with neighbours: "after", those after it). On a selective set, a few nodes among many, that misses most of a node's neighbours, and the module warns that it can. A step's warned lists the stores it warns on.
A case. For each store, a runner empties the graph, makes each of indexes, as store.yaml's DDL makes them, writes graph, and waits for the indexes to hold it. It then runs each step through its library's loadNearestAmong, or the module's VectorSearch given the same request, and compares the rows: for each id of a node with an embedding the search can measure, its neighbours, best first, each score within 1e-6. An id of no node, or of a node with no embedding, or by cosine one of no length, has no row. today, as in search.yaml, says what a store gives where it differs, and the issue; it is not read. FalkorDB can't tell that a stored vector has no length, and measures one by cosine (#409).
writevalues.yaml is a table of the values a CREATE or a MERGE writes (liberation-data/drivine4j#814, #817). Its positions are each place a write takes a value: the properties of a node made alone, of a relationship's endpoint and of a relationship, by a CREATE and by a MERGE, each in a FOREACH too, and a MERGE's ON CREATE SET and ON MATCH SET, each a statement over graph with {value} for the value and made the variable the clause makes. Its forms are each value, with {made} for that variable: a pattern, a pattern comprehension, EXISTS { … } and COUNT { … } naming a variable bound before, a pattern of new variables only, the clause's own variable bare, a property of it, a pattern, EXISTS { … } or a pattern comprehension naming it, and a variable bound before read plainly. Its cells are every position by every form, in order, each with its statement and what Neo4j 2026.10.1 gives, each statement on a graph of graph alone, as EXPLAIN and as a run: neo4j: "stored" and its rows, or the GQL code it refuses it with (when it runs where EXPLAIN takes it) and the error the SQL stores refuse it with, before any SQL runs.
Where Neo4j stores a cell, the SQL stores either compile it to Neo4j's rows or, where the cell says ours: "not compiled yet", refuse it as not compiled yet, never with other rows: a variable the clause makes, named at any depth inside a pattern, a pattern comprehension, a list comprehension, a quantifier, reduce() or a subquery in its own properties, which Neo4j reads as a variable bound anew there ([(a)-->(x) | x] matches any node as a), and where it does not refuse it for 42N29 or 42I58, is refused so (drivine4j#814). Such a cell keeps Neo4j's rows as the evidence. Its canaries are 40 nestings three deep of those constructs at those positions, drawn from constructs with seed, each with Neo4j's verdict and rows: each is refused, with the error it pins (on Oracle alone, oracle), or compiled to Neo4j's rows.
The module's WriteValuesTableTest holds each SQL store's compiler to every cell and every canary on every target, and fails on a cell missing, added or out of order, and on a cell or a canary compiled where Neo4j refuses it. Each refused cell and canary is a step of refusals.yaml, and each compiled one a step of writes.yaml with Neo4j's rows, after graph alone, which WriteValuesSpecTest checks, so each library runs them as it runs those files. Beyond the table, refusals.yaml's "Variable a write makes, read in a subquery in its properties" and "Variable a write makes, read inside a pattern or a comprehension in its properties" pin what a subquery and a pattern comprehension within one read of what the clause makes. refusals.yaml's "Node or relationship set as a property" pins what Neo4j refuses when it runs (its 22N01): a node, a relationship, a path or a relationship list written as a property, in a list written or as an entry of a map set, by SET and in what a CREATE or a MERGE makes, and writes.yaml's "A value worked out of a node, a relationship or a path, set as a property" what it stores of one. refusals.yaml's "Map or list of lists written as a property" pins the same refusal of a map, written, projected or bound by a WITH to one written, and of a list written that holds a map or a list, and writes.yaml's "A value read of a map, or an empty list, written as a property" what Neo4j stores beside them. writes.yaml's "A value given that Neo4j stores as no property" pins a map, a list that holds a map or a list, and a list of more than one kind given as a parameter or a row the statement unwinds, which fails as its row is written (drivine4j#820): a parameter's value is known as the statement compiles, so it costs no test as it runs, and a row unwound is tested as it is written, so one a MATCH or a WHERE leaves out fails nothing. writes.yaml's "A value worked out as it runs that Neo4j stores as no property" pins the same of a value the statement works out as it runs, as a key of a row a WITH passes on or an entry of a parameter's map, tested as each row is written, where its kind is not known as the statement compiles.
A pattern, a pattern comprehension, EXISTS { … } or COUNT { … }, at any depth, in the properties of any node or relationship a CREATE or a MERGE makes, within a FOREACH too, is refused as not compiled yet, whatever it reads, where Neo4j does not refuse it first (drivine4j#814). No position is kept, as Neo4j reads such a value at another moment than a SQL store, which reads each row's before it writes it. Neo4j reads a CREATE's before it writes any of its rows, so UNWIND [1, 2] AS i CREATE (m:M {c: COUNT { (:M) }}) stores 0 and 0; and a MERGE's after it makes the node or the relationship, so UNWIND [1, 2, 3] AS i MERGE (m:M {f: size([(x:M)-[*0..1]->() | x])}) makes one node, counting itself, where a SQL store makes three. The table's cells in those positions say ours: "not compiled yet". refusals.yaml's "Pattern or subquery in the properties of a CREATE or a MERGE" holds the steps refused so, with Neo4j's rows in each note, and its "When a write reads its own properties" what each reads, with what the SQL stores gave compiled. What is compiled is the value worked out in a WITH before the write, as the refusal says: writes.yaml's "Work a read out in a WITH before the write".
nesting.yaml is a table of where a statement reads a variable, p, that a WITH drops, or a control, and what Neo4j 2026.10.1 does with each (liberation-data/drivine4j#681, #680). Its lists:
contexts: each outer statement, with{hole}where the reference goes andanchorthe variable in scope there. AWITHthat dropspand neither groups nor is DISTINCT, one that groups and one that is DISTINCT, each with the reference in itsWHEREand in itsORDER BY, and, as controls, a statement with noWITH, in aMATCH'sWHEREand in aRETURN'sORDER BY.nesting: each construct a reference may sit within, thekindof hole it goes in (an expression,expr, or a query's clauses,query) and theholeit holds (expr,patternorquery). In itscypher,{v}is the variable in scope,{d}its depth, which names what it binds, and{col}the column its query returns;anchoris the variable in scope within its hole, andcolumnwhat the hole's query returns, where they are not{v}andx{d}.holes: how each kind of hole holds a reference, and a construct of another kind.references: a quantified path pattern's group namingp, one of new names only, and a read ofp.name, each as a pattern and as an expression.
Its cells are every context by every nesting of depth 0, 1 and 2 that is Cypher, by every reference, in order, each with its statement and what Neo4j gives as EXPLAIN and as a run, over queries.yaml's graph alone: neo4j: "rows" and its rows, or the GQL code it refuses it with, and the error the SQL stores refuse it with, before any SQL runs. A cell Neo4j gives rows for that the SQL stores refuse has both, and a note of why. A cell whose answer Neo4j doesn't give alike from one environment to another says neo4j: "not pinned", with the error the SQL stores refuse it with and a note of why, and no rows: in a WITH that drops p and groups, a read of p in its WHERE within a list comprehension, one deep and two, gives no rows on one Neo4j 2026.10.2 and four on another (#366), so no runner could hold to either. Its canary holds 40 cells three deep, picked at random from its seed, alike.
The module's NestingTableTest builds the cells again from the lists and fails on a cell missing, added or out of order, and holds each SQL store's compiler to every cell and every canary on every target. Each cell the SQL stores refuse is a step of refusals.yaml, and each cell Neo4j gives rows for a step of queries.yaml, run on the SQL stores too where they compile it, but none that is not pinned, which NestingSpecTest checks, so each library runs them as it runs those files.
The table's CALLs name what they import. Each file's case "A CALL in the order of a RETURN that groups or is DISTINCT, and CALL (*) where a projection groups", captured alike, pins the rest of what Neo4j reads there. In the WHERE or the order of a WITH or a RETURN that groups or is DISTINCT, a CALL's brackets read as of that projection, its items and what a subquery reads from outside it, so a subquery's own variable is refused there (Neo4j's 42N44). A CALL (*) { … } there imports it, and refuses only a read of it. A projection that groups within the subquery reads its own items, and a subquery within it, an EXISTS { … } or a COUNT { … }, what is in scope where it is: a comprehension's, a quantifier's, a reduce()'s or a pattern comprehension's variables, and a subquery's own, but not what a CALL (*) hides. One in an aggregate of the order, ORDER BY sum(COUNT { … }), reads what is bound before the projection, as the aggregate does. A procedure's CALL db.labels() YIELD label WHERE … there reads a subquery's own variable as anywhere.
rebinding.yaml is a table of where a subquery binds again a name it has from outside it, name (m1), or a control, and what Neo4j 2026.10.1 does with each (liberation-data/drivine4j#819). Its lists:
contexts: each subquery the binding sits within, eachouteras a whole statement whoseMATCHbinds{o}andinneras the body of another, with{body}where the binder goes: aCALL ({o})that imports it, aCALL (*), aCALL ()that imports nothing, aCALL ()within anEXISTS { … }and within aCOUNT { … }, and anEXISTS { … }and aCOUNT { … }alone. The compiler doesn't parse aCOLLECT { … }yet, so none is in the table.binders: each way a subquery's clauses may bind the name,{v}: aWITH's alias, anUNWIND's, aCALL's item, aFOREACH's variable, aMATCH's pattern variable and anOPTIONAL MATCH's; each asbind, which binds it, andread, which binds a new name and reads{v}.variants: the binding, with the name bound outside (outer: m1); and the two controls, the same binding with the name not bound outside (outer: m0), and the name bound outside and only read.
Its cells are every context by every binder by every variant, in order, each with its statement and what Neo4j gives as EXPLAIN and as a run, over queries.yaml's graph alone: neo4j: "rows" and its rows, or the GQL code it refuses it with, and the error the SQL stores refuse it with, before any SQL runs. Its canary holds 20 statements two contexts deep, one context's inner within another's outer, picked at random from its seed, none a cell's, alike.
What Neo4j refuses (its 42N07): within a subquery, a WITH's or an UNWIND's alias, or a CALL's item, that names a variable the subquery has from outside it, one a CALL imports or one in scope for an EXISTS { … } or a COUNT { … }, and an EXISTS { … }'s or a COUNT { … }'s RETURN alias too. The name stays from outside for the whole subquery, after a WITH that passes it on as it is, WITH m1, too. One a CALL { … }'s leading WITH imports, CALL { WITH m1 … }, Neo4j has declared in the subquery's own scope, as at the top: an UNWIND that binds it again is refused as declared already (its 42N59), as one that binds again a name in scope at the top is, after a WITH that passes it on too, and a WITH's alias projects it anew, as writes.yaml and queries.yaml pin. refusals.yaml pins these beyond the table, in "Rebinding a variable from outside a subquery refused: after a WITH passes it on, and beyond the table". A CALL () within, which imports none, may bind such a name with a WITH or an UNWIND, but a CALL's item within it names none of an EXISTS { … }'s or a COUNT { … }'s. A FOREACH's variable binds the name within the FOREACH alone, as a comprehension's or a reduce()'s does within the expression, and a pattern reads the variable from outside, so Neo4j takes them, as it takes a CALL () at the top level that binds the name. A name a WITH drops is from outside an EXISTS { … } or a COUNT { … } in its WHERE or its order, which Neo4j still reads there, but not where the WITH groups, whose WHERE reads its items alone; queries.yaml's "Rebinding a variable a CALL { … }'s leading WITH imports" pins those Neo4j takes.
The module's RebindingTableTest builds the cells again from the lists and fails on a cell missing, added or out of order, and holds each SQL store's compiler to every cell and every canary on every target. Each cell the SQL stores refuse is a step of refusals.yaml, and each cell Neo4j gives rows for a step of queries.yaml, run on the SQL stores too where they compile it, which RebindingSpecTest checks, so each library runs them as it runs those files.
A library's object manager writes its own SQL for a view or a fragment: the joins, the documents it reads back, the plan of a write. Where that SQL compares, sorts or binds a value, it asks the compiler's module for the part, as a Cypher expression over its own rows, so it compares a temporal value by its key, reads a key that begins with $ as it is held, answers an equality through the store's index, and binds a value, as a compiled statement does (#161). In Kotlin these are SqlCypherCompiler.condition, value and order; through the JSON entry, a fragment request (see compiler/README.md, "Fragments").
fragments.yaml pins them. Its nodes are setup's, of the label Person, and each case is one expression over n, a row people_ of the graph's table of nodes that the library matches by that label:
- name: "A date before a date: a value of another kind is no date, and neither before nor after it"
condition: "n.born < $d"
parameters: {d: {"$temporal": "DATE", "text": "1900-01-01"}}
rows: ["p1"]
sql:
postgres:
- "CASE WHEN ... END"condition,order(withdescending) orvalueis the expression;parametersare written as the JSON entry takes them.sqlis each store's terms, as the module gives them for the graphgraph, with the file'sindexedproperties andvariables. The module'sFragmentsTestholds each target to them.impliedLabelsis, on each store where there are any, the labels the answer says every row the condition keeps carries, by variable ({oracle: {n: ["Person"]}}): on Oracle, those of each equality read through a label's index that is one of the condition's own conjuncts. A library's query need not test them again, and reads them from the answer, not from the terms' text (#373). None on a store the case doesn't name.tableputs each form a fragment takes (compiler/README.md, "Fragments") in each place a library puts it: each condition alone, underNOT, besideORandAND-ed (positions), each value as an order ascending and descending, withrowsfor each in turn anddigests, the first 16 hexadecimal digits of the SHA-256 of each store's terms joined by a line feed, which the module's test holds every target to.rowsis what Neo4j gives for the same expression oversetup's nodes, by id: the nodesMATCH (n:Person) WHERE <condition> RETURN n.idkeeps, or the orderMATCH (n:Person) RETURN n.id ORDER BY <order>, n.idgives. Avaluehas none.
A runner. Each library's runner makes setup's nodes on each SQL store, as a statement it compiles, and for each case asks for the fragment as its object manager does, then runs it in SQL of its own: SELECT /*+ <the fragment's hints> */ ... FROM <graph>_node people_ WHERE <Person among its labels> AND <condition>, or ORDER BY <the order's terms>, <its id>, binding the fragment's parameters under the names it gives. Where the answer's impliedLabels names Person for n, the runner leaves out its test of Person, as a library's query does: setup's node of another label then shows a condition that implies a label it doesn't test. Without its hints, Oracle may give a case the wrong rows. The ids it keeps, or their order, are rows.
These files are a contract between the three libraries. They live in drivine-cypher-spec, and each library copies a tagged version into its tests, unchanged, with a script of its own (scripts/sync-cypher-spec.sh in drivine4j). A change of what the compiler gives or refuses lands there first, in a tagged release, and each library then moves to that tag.
Some tests stay in each library's own suite: values that change on every run (a UUID, the time), parameters of a language's own temporal types, statements built in code (hundreds of CREATEs), and whatever needs a connection of its own (a rolled-back transaction).