fix(driver-memory): refuse the filters the live query path cannot evaluate, compile the one it must (#5324, #5328)

.changeset/memory-filter-refuse-what-it-cannot-evaluate.md

@@ -0,0 +1,51 @@
+---
+"@objectstack/driver-memory": patch
+---
+
+fix(driver-memory): the live query path refuses the filters it cannot evaluate, and compiles the one it must (#5324, #5328)
+
+**This is an observable behaviour change.** Two filter shapes that used to be
+answered *silently* now raise the catalogued `INVALID_FILTER` / 400 every other
+filter refusal in this driver and in `driver-sql` already speaks (ADR-0112):
+
+| filter | before | now |
+|---|---|---|
+| an operator outside the Filter Protocol — `{ name: { $sounds_like: 'x' } }`, `$elemMatch`, `$size`, `$where`, field-level `$not`, … | handed to mingo, which threw a `MingoError` carrying **no `code` and no `status`** — served as a 500-shaped `{ error }` body | `INVALID_FILTER` / 400, naming the operator, the field and its position |
+| a `$between` whose comparand is not `[min, max]` — `{ score: { $between: 5 } }` | the arm was skipped, the constraint **vanished**, and `find` returned `[]` | `INVALID_FILTER` / 400, wording aligned with `driver-sql`'s |
+
+Two more shapes join them, same cause: an undeclared `$`-combinator in a node
+position (`{ $nor: … }`, `{ $where: … }` — `FilterConditionSchema` declares
+`$and`/`$or`/`$not` and nothing else), and a combinator operand that is not a
+filter condition (`{ $or: 'x' }`, `{ $or: [null] }`, `{ $not: 'x' }`).
+
+If a query of yours starts returning a 400, it was already broken — it was
+returning an empty result set or an uncoded 500 for the same input, and
+`driver-sql` was rejecting it. The message names the operator and the path
+(`filter.$or[1].$and[0].stage`).
+
+**`$not` is the opposite change: it now works.** `$not` is a declared combinator
+(`LOGICAL_OPERATORS`), `cel-to-filter` emits it for every CEL `!expr` in an RLS
+read scope, and `driver-sql` / `driver-mongodb` / this package's own reference
+matcher all implement it — but the live query path passed it to mingo, and
+MongoDB has no document-level `$not`, so **every query carrying a negated scope
+threw** `unknown top level operator: $not`. It is compiled to `$nor` with one
+operand, the same rewrite `driver-mongodb` performs, which is NULL-safe by
+construction and therefore lands on the answer #5146 ruled canonical.
+
+Both of this package's filter faces — the live mingo path and the reference
+matcher — now share ONE shape gate, so they cannot answer one filter
+differently again. They did: given a malformed `$between` the live path returned
+NO rows while the matcher returned EVERY row.
+
+The conformance gap that hid all of this is closed too. `FILTER_LOGIC_CASES`
+was run against this backend through the reference matcher only — the driver
+does not call it — so the table's `$not` case had been green for as long as it
+existed while the same filter through `InMemoryDriver.find` threw. The table now
+runs through the real driver, as it does for the other three backends.
+
+Accepted operators are the spec's `FILTER_OPERATORS`, plus `$regex` (produced by
+plugin-auth's ObjectQL adapter, compiled by `driver-sql`) and its `$options`
+companion. `$options` is a modifier, not a predicate: on its own, with no
+`$regex` beside it, it is refused like any other filter this driver cannot
+evaluate — it used to raise the same uncoded engine error on the live path and
+match every row in the matcher.

packages/plugins/driver-memory/src/memory-driver-document-not.test.ts

@@ -0,0 +1,232 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#5324] Document-level `$not` on the LIVE query path — the shape the issue was
+ * filed on.
+ *
+ * # Why this is implemented and not refused
+ *
+ * #5324 offered both directions and deliberately declined to choose. The
+ * evidence chooses: `$not` is a DECLARED combinator (`LOGICAL_OPERATORS` in
+ * `@objectstack/spec/data`, alongside `$and`/`$or`), `driver-sql` compiles it,
+ * `driver-mongodb` translates it, `memory-matcher` evaluates it, and
+ * `FILTER_LOGIC_CASES` — the standard every backend is held to — contains a case
+ * that requires it. Refusing it would have made this driver the only backend
+ * that cannot run a spec-declared operator, and would have left the conformance
+ * table with a case it could never pass. "Refuse what you cannot evaluate" has a
+ * companion clause: what the contract DECLARES, you evaluate.
+ *
+ * So the general refusal in `memory-filter-vocabulary-refusal.test.ts` covers
+ * every operator the Filter Protocol does not declare, and this file covers the
+ * one it does.
+ *
+ * # The rewrite, and why `$nor`
+ *
+ * mingo is a MongoDB-semantics engine, and MongoDB has no document-level `$not`
+ * — `unknown top level operator: $not`, uncoded, was the whole of #5324. The
+ * negation of a whole condition in MongoDB is `$nor` with a single operand, and
+ * that is exactly the rewrite `driver-mongodb` performs for the same reason
+ * (#4405). Nothing else about the condition changes.
+ *
+ * # Why the null cases are the load-bearing ones
+ *
+ * `cel-to-filter.ts` lowers a CEL `!expr` to `{ $not: {…} }`, which is the
+ * ordinary product of an RLS read scope — so this operator decides who sees
+ * which rows. #5146 ruled the JS backends' two-valued reading canonical and
+ * rewrote `driver-sql`'s SQL to match it, because SQL's `NOT (col = x)` is
+ * UNKNOWN for a NULL column and a `WHERE` drops the row. `$nor` is total by
+ * construction and lands on the same answer — asserted below against the exact
+ * fixture and expectations `memory-matcher-not-null-safe.test.ts` pins, so the
+ * live path is held to the ruling rather than merely to "it no longer throws".
+ */
+
+import { describe, it, expect, beforeAll } from 'vitest';
+import type { FilterCondition } from '@objectstack/spec/data';
+
+import { InMemoryDriver } from './memory-driver.js';
+import { match } from './memory-matcher.js';
+
+/** Fields present but null — how a SQL NULL round-trips into a record. */
+const NULLED = [
+  { id: '1', stage: 'won', owner: 'u1', amount: 10 },
+  { id: '2', stage: 'lost', owner: 'u2', amount: 20 },
+  { id: '3', stage: null, owner: 'u1', amount: null },
+  { id: '4', stage: null, owner: null, amount: 40 },
+];
+
+/** The same rows with the null fields ABSENT — the shape a partial write leaves. */
+const MISSING = [
+  { id: '1', stage: 'won', owner: 'u1', amount: 10 },
+  { id: '2', stage: 'lost', owner: 'u2', amount: 20 },
+  { id: '3', owner: 'u1' },
+  { id: '4', amount: 40 },
+];
+
+const ALL = ['1', '2', '3', '4'];
+
+const FIELDS = {
+  id: { type: 'text', name: 'id' },
+  stage: { type: 'text', name: 'stage' },
+  owner: { type: 'text', name: 'owner' },
+  amount: { type: 'number', name: 'amount' },
+};
+
+describe('[#5324] InMemoryDriver.find compiles a document-level $not', () => {
+  let nulled: InMemoryDriver;
+  let missing: InMemoryDriver;
+
+  beforeAll(async () => {
+    nulled = new InMemoryDriver({ persistence: false });
+    await nulled.syncSchema('deal', { fields: FIELDS });
+    for (const row of NULLED) await nulled.create('deal', { ...row });
+
+    missing = new InMemoryDriver({ persistence: false });
+    await missing.syncSchema('deal', { fields: FIELDS });
+    for (const row of MISSING) await missing.create('deal', { ...row });
+  });
+
+  const idsFrom = async (driver: InMemoryDriver, where: unknown): Promise<string[]> => {
+    const rows = await driver.find('deal', { object: 'deal', fields: ['id'], where: where as FilterCondition });
+    return (rows as Array<Record<string, unknown>>).map((r) => String(r.id)).sort();
+  };
+
+  /**
+   * Both readings of "no value" must give the same answer, and the reference
+   * matcher must give it too — the same contract
+   * `memory-matcher-not-null-safe.test.ts` states for its own face, now binding
+   * on the path that actually serves queries.
+   */
+  const matched = async (where: unknown): Promise<string[]> => {
+    const fromNulled = await idsFrom(nulled, where);
+    const fromMissing = await idsFrom(missing, where);
+    expect(fromMissing, 'a null field and an absent field must match alike').toEqual(fromNulled);
+    const reference = NULLED.filter((r) => match(r, where)).map((r) => r.id);
+    expect(fromNulled, 'the live query path and the reference matcher must agree').toEqual(reference);
+    return fromNulled;
+  };
+
+  describe('the shape #5324 reported — every position, not just the top level', () => {
+    it('at the top level', async () => {
+      expect(await matched({ $not: { stage: 'won' } })).toEqual(['2', '3', '4']);
+    });
+
+    it('inside a $or branch', async () => {
+      // The issue measured all three of these throwing `unknown top level
+      // operator: $not`; `normalizeFilterCondition` passed `$not` through
+      // wherever it sat, so nesting never helped.
+      expect(await matched({ $or: [{ $not: { stage: 'won' } }] })).toEqual(['2', '3', '4']);
+    });
+
+    it('inside a $and branch', async () => {
+      expect(await matched({ $and: [{ $not: { stage: 'won' } }] })).toEqual(['2', '3', '4']);
+    });
+
+    it('ANDs with its sibling keys', async () => {
+      expect(await matched({ $not: { stage: 'won' }, owner: 'u1' })).toEqual(['3']);
+    });
+
+    it('nested two combinators deep', async () => {
+      expect(await matched({ $and: [{ $or: [{ $not: { stage: 'won' }, owner: 'u1' }] }] })).toEqual(['3']);
+    });
+
+    it('the RLS shape a CEL `!(stage == "won")` scope lowers to', async () => {
+      // `cel-to-filter.ts` emits exactly this for a negated read scope. On this
+      // driver — the default for dev and test — it used to be an uncoded throw
+      // on every query the scope touched, not a wrong row count.
+      expect(await matched({ $not: { stage: 'won' } })).toHaveLength(3);
+    });
+  });
+
+  describe('the #5146 canon, now answered by the live path too', () => {
+    it('$not over multiple keys matches a record missing EITHER', async () => {
+      expect(await matched({ $not: { stage: 'won', owner: 'u1' } })).toEqual(['2', '3', '4']);
+    });
+
+    it('$not of a $or rejects a value-less record whose OTHER branch matches', async () => {
+      // Record 3 has no stage but owner = 'u1', so the $or holds and the
+      // negation must reject it. This is the case that forced `driver-sql` to
+      // compile its NULL guard onto each leaf instead of beside the `NOT`.
+      expect(await matched({ $not: { $or: [{ stage: 'won' }, { owner: 'u1' }] } })).toEqual(['2', '4']);
+    });
+
+    it('$not of a $and matches every record failing either conjunct', async () => {
+      expect(await matched({ $not: { $and: [{ stage: 'won' }, { owner: 'u1' }] } })).toEqual(['2', '3', '4']);
+    });
+
+    it('a double negation is the positive filter again', async () => {
+      expect(await matched({ $not: { $not: { stage: 'won' } } })).toEqual(['1']);
+      expect(await matched({ $not: { $not: { stage: 'won' } } })).toEqual(await matched({ stage: 'won' }));
+    });
+
+    it('$not of $ne still means "the field IS that value"', async () => {
+      expect(await matched({ $not: { stage: { $ne: 'won' } } })).toEqual(['1']);
+    });
+
+    it('$not of $in matches the value-less records', async () => {
+      expect(await matched({ $not: { stage: { $in: ['won'] } } })).toEqual(['2', '3', '4']);
+    });
+
+    it('$not of an ordering comparison matches the value-less records', async () => {
+      expect(await matched({ $not: { amount: { $gt: 15 } } })).toEqual(['1', '3']);
+    });
+
+    it('$not of $contains matches the value-less records', async () => {
+      expect(await matched({ $not: { stage: { $contains: 'w' } } })).toEqual(['2', '3', '4']);
+    });
+
+    it('$not of a null predicate', async () => {
+      expect(await matched({ $not: { stage: { $null: true } } })).toEqual(['1', '2']);
+      expect(await matched({ $not: { stage: { $null: false } } })).toEqual(['3', '4']);
+    });
+  });
+
+  describe('the boolean identities (#5134)', () => {
+    it('$not: {} matches nothing — NOT TRUE ≡ FALSE', async () => {
+      expect(await matched({ $not: {} })).toEqual([]);
+    });
+
+    it('$not of an empty $or matches everything', async () => {
+      expect(await matched({ $not: { $or: [] } })).toEqual(ALL);
+    });
+  });
+
+  /**
+   * Measured while verifying this fix, and NOT caused by it: three operators
+   * answer a value-less field differently on the two faces, with or without a
+   * `$not` around them. mingo reads `$exists` as key presence and lets `$nin`
+   * match a missing key; the matcher's `value === undefined` guard and its
+   * `typeof value !== 'string'` test answer the opposite.
+   *
+   * This is a SEMANTIC divergence, not a shape one, so the gate this PR adds
+   * neither causes nor cures it — a ruling on which reading is canonical belongs
+   * with the identical matcher-vs-formula divergence already filed as **#5299**,
+   * where this measurement is recorded. Pinned as measured so the fix that lands
+   * there has to move these lines deliberately.
+   */
+  describe('known two-face divergences on a value-less field — pinned, see #5299', () => {
+    const liveVsReference = async (where: unknown) => ({
+      live: await idsFrom(nulled, where),
+      reference: NULLED.filter((r) => match(r, where)).map((r) => r.id),
+    });
+
+    it('$exists on a present-but-null field: mingo says "the key is there", the matcher says "no value"', async () => {
+      expect(await liveVsReference({ stage: { $exists: true } })).toEqual({
+        live: ['1', '2', '3', '4'],
+        reference: ['1', '2'],
+      });
+    });
+
+    it('$nin on an ABSENT field', async () => {
+      const live = await idsFrom(missing, { stage: { $nin: ['won'] } });
+      const reference = MISSING.filter((r) => match(r, { stage: { $nin: ['won'] } })).map((r) => r.id);
+      expect({ live, reference }).toEqual({ live: ['2', '3', '4'], reference: ['2'] });
+    });
+
+    it('$notContains on a null field', async () => {
+      expect(await liveVsReference({ $not: { stage: { $notContains: 'w' } } })).toEqual({
+        live: ['1'],
+        reference: ['1', '3', '4'],
+      });
+    });
+  });
+});

packages/plugins/driver-memory/src/memory-driver-filter-logic-conformance.test.ts

@@ -0,0 +1,107 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#5324/#5328] Filter logical-combinator conformance for the LIVE QUERY PATH —
+ * `InMemoryDriver.find`, through `normalizeFilterCondition` and mingo.
+ *
+ * # Why this file exists at all
+ *
+ * `FILTER_LOGIC_CASES` is the one standard five filter backends are held to
+ * (`@objectstack/spec/data`, #3774). Four of them ran it through the code a real
+ * query executes: `driver-sql` compiles it to SQL, `driver-sqlite-wasm` runs
+ * that SQL on sql.js, `driver-mongodb` translates and executes it, and
+ * `service-analytics` lowers it into its read-scope SQL.
+ *
+ * `driver-memory` ran it through `memory-matcher` ONLY
+ * (`memory-matcher-or-semantics.test.ts`). That file is not a driver test: the
+ * driver does not call `match()` — it imports exactly one symbol from that
+ * module, `getValueByPath`, and filters with mingo instead. So this backend's
+ * half of the conformance table was measured against a REFERENCE implementation
+ * while the half users actually run was never executed against the standard once.
+ *
+ * The cost was not hypothetical. The table's `$not ANDs with its sibling keys
+ * inside a branch` case was green here for as long as it has existed, while the
+ * same filter through `InMemoryDriver.find` threw `unknown top level operator:
+ * $not` — MongoDB has no document-level `$not`, so mingo has none either (#5324).
+ * A conformance suite that green-lights an operator the driver cannot run is
+ * worse than no suite: it is a gate reporting coverage it does not have, which
+ * is the "declared ≠ enforced" shape Prime Directive #10 names.
+ *
+ * So the gap is closed the way the other three backends close it — by running
+ * the table through the thing that serves queries. `memory-matcher-or-semantics`
+ * stays: the matcher is still the reference evaluator, and holding BOTH faces to
+ * the same table is what makes "this package has two filter surfaces" a
+ * statement someone can check.
+ */
+
+import { describe, it, expect, beforeAll } from 'vitest';
+import { FILTER_LOGIC_CASES, FILTER_LOGIC_ROWS } from '@objectstack/spec/data';
+import type { FilterCondition } from '@objectstack/spec/data';
+
+import { InMemoryDriver } from './memory-driver.js';
+import { match } from './memory-matcher.js';
+
+const TABLE = 'conformance';
+
+describe('[#5324] InMemoryDriver.find — filter logic conformance (the LIVE query path)', () => {
+  let driver: InMemoryDriver;
+
+  beforeAll(async () => {
+    driver = new InMemoryDriver({ persistence: false });
+    await driver.connect();
+    // Every fixture column is a plain string — the shared table keeps its
+    // predicates boring on purpose, so nothing here is about coercion. The
+    // declaration is still made, because that is how a real object reaches the
+    // driver and how its field kinds are resolved (#4047).
+    await driver.syncSchema(TABLE, {
+      fields: {
+        id: { type: 'text', name: 'id' },
+        a: { type: 'text', name: 'a' },
+        b: { type: 'text', name: 'b' },
+        c: { type: 'text', name: 'c' },
+        owner: { type: 'text', name: 'owner' },
+        status: { type: 'text', name: 'status' },
+        parent_object: { type: 'text', name: 'parent_object' },
+        parent_id: { type: 'text', name: 'parent_id' },
+      },
+    });
+    for (const row of FILTER_LOGIC_ROWS) await driver.create(TABLE, { ...row });
+  });
+
+  const ids = async (where: FilterCondition): Promise<string[]> => {
+    const rows = await driver.find(TABLE, { object: TABLE, fields: ['id'], where });
+    return (rows as Array<Record<string, unknown>>).map((r) => String(r.id)).sort((x, y) => x.localeCompare(y));
+  };
+
+  for (const c of FILTER_LOGIC_CASES) {
+    it(c.name, async () => {
+      expect(await ids(c.filter), c.note).toEqual([...c.expected]);
+    });
+  }
+
+  /**
+   * The fixture as a whole, so a case that returns nothing because the seed
+   * failed cannot read as a case that correctly excluded everything.
+   */
+  it('the fixture really is all four rows', async () => {
+    expect(await ids({})).toEqual(['1', '2', '3', '4']);
+  });
+
+  /**
+   * The two faces, on the same table, in one assertion.
+   *
+   * `memory-matcher-or-semantics.test.ts` already holds the matcher to these
+   * cases and this file holds the driver to them, so both being green already
+   * implies agreement. Asserting it directly is still worth one test: it is the
+   * invariant #5240 established for this package ("a backend whose two halves
+   * disagree about what a filter MEANS is exactly the divergence the ruling
+   * closes"), and stated here it survives either suite being edited.
+   */
+  it('both filter faces answer the whole table identically', async () => {
+    for (const c of FILTER_LOGIC_CASES) {
+      const live = await ids(c.filter);
+      const reference = FILTER_LOGIC_ROWS.filter((r) => match(r, c.filter)).map((r) => r.id);
+      expect(live, `${c.name}: the live query path and the reference matcher disagree`).toEqual(reference);
+    }
+  });
+});