Skip to content

driver-sql: a declared operator on a multiple: true (JSON array) column silently answers wrong — $in/$eq always zero rows, $nin returns the rows it was asked to EXCLUDE #7398

Description

@baozhoutao

Summary

A declared filter operator applied to a column the operator cannot mean anything for is compiled to SQL and executed anyway, returning a wrong answer with a 200. No error, no warning, nothing for a type checker to catch.

Concretely, on a field declared multiple: true (which driver-sql stores as a JSON text column):

  • $in / $eq / bare equality → always zero rows (fail-closed)
  • $nin → returns the rows it was asked to EXCLUDE (fail-open) ⚠️

The second one is the reason I am filing this rather than logging it as a footgun.

⚠️ $nin on an array column inverts the result — this is the dangerous half

{ members: { $nin: [U1] } } returned exactly the record whose members contains U1.

The mechanism is one line of SQL: the stored column value is the text ["U1","U2"], so
members not in ('U1') is true — the text genuinely is not equal to that uuid. "Exclude these"
therefore compiles to "return everything".

Why this matters more than the $in case:

Measured

Environment: @objectstack/* 17.0.0-rc.5, SqlDriver(better-sqlite3), single-tenant dev runtime,
queries issued over the REST list face (GET /api/v1/data/:object?filter={json}).

Fixture: one record in the table. Its multi-value lookup column members holds ["U1","U2"]
(shown here as U1/U2; they are ordinary record ids).

Multi-value (multiple: true) column — members

filter HTTP rows returned verdict
{members:{$in:[U1]}} 200 0 ✗ silently empty
{members:{$in:[U1,U2]}} 200 0 ✗ silently empty
{members:{$eq:U1}} 200 0 ✗ silently empty
{members: U1} (bare equality) 200 0 ✗ silently empty
{members:{$contains:U1}} 200 1 ✅ the only thing that works
{members:{$contains:U2}} 200 1
{$or:[{members:{$contains:U1}},{members:{$contains:U2}}]} 200 1 ✅ the only "any of" spelling
{members:{$nin:[U1]}} 200 1 ⚠️ inverted — this row contains U1
{members:{$overlaps:[U1]}} 400 undeclared operator, loudly refused
{members:{$containsAny:[U1]}} 400 undeclared operator, loudly refused
{members:{$in:[[U1,U2]]}} 500 DATABASE_ERROR (this is the #5234 face, not this one)

Control — the same object's single-value lookup column owner

filter HTTP rows verdict
{owner:{$in:[U1]}} 200 1

So $in itself is fine. What is broken is $in meeting an array column.

Raw SQL, same table, same row

Reading the column and querying it directly, with no platform in the path:

stored value:  ["U1","U2"]

SELECT COUNT(*) … WHERE members IN ('U1')                 -> 0
SELECT COUNT(*) … WHERE members LIKE '%U1%' ESCAPE '\'    -> 1

That is the whole bug in two lines. driver-sql lowers $in straight to col in (?, ?, …)
(the list('in') arm of applyNormalizedComparison) without consulting the column's type,
and a JSON text column is never equal to any single member it contains. $contains "works" only
because it lowers to LIKE '%value%' and the JSON serialization happens to contain the substring.

Why I read this as a defect and not as documented behaviour

The platform already chose "refuse" on every adjacent seam — this is the one square on the grid
with no gate:

So writes check the field's type against the payload; reads do not check the field's type against
the operator. Given isJsonField(type, field) { return JSON_COLUMN_TYPES.has(type) || !!field.multiple }
already exists in driver-sql, the information needed for the check is present at the point of lowering.

And the failure mode is the worst available one: 200 + an empty array is byte-identical to a
successful query that legitimately matched nothing.
There is no signal for a caller to key on.

What it cost downstream (why I bothered to measure it)

In a downstream business app, a delete-guard was written as:

plans.find({ where: { assignees: { $in: memberIds } } })   // assignees: multiple lookup

The rule it implements is "refuse to delete a team while any of its members is still referenced by a
plan". Because the predicate matched nothing, the guard never fired once since the feature shipped.
It threw no error, logged nothing, and type-checked. It was found only when someone happened to test
the guard's positive path by hand. A full sweep of that codebase then had to be run to prove there
was no second occurrence.

The general shape: this bug class does not produce incidents, it produces rules that quietly do not
exist
. That is why "the caller should know better" is a weak answer here — there is nothing for the
caller to notice.

Asks (either would fix it; the first is the smaller change)

  1. Refuse it. Bring "declared operator × column type it cannot apply to" under the same
    400 INVALID_FILTER envelope that already covers unknown operators and malformed comparands —
    naming the operator, the field, and the fact that the filter was not applied. Minimum viable
    set: $in / $nin / $eq / bare equality / the ordering comparisons, against field.multiple
    (and the other JSON_COLUMN_TYPES) columns.
  2. Or give array columns a real membership operator. Today $contains is declared as a string
    operator ($contains: z.ZodString, in StringOperatorSchema) and works on array columns purely
    because the JSON serialization is text and LIKE '%v%' happens to hit. That is incidental, not
    designed — it is substring matching over a serialization, so it is also sensitive to how the
    column is serialized. There is likewise no way to express "any of these values" in one query;
    the only spelling is an $or of N $contains, since FILTER_OPERATORS is a closed set of 15
    with no $overlaps / $containsAny.

If the ruling is "1 only", that is completely fine for me downstream — a loud refusal is all the
downstream app needs, and $or + $contains is a perfectly workable spelling. The current state
(silently wrong, and inverted for $nin) is the only outcome that cannot be worked with.

Refs

Minor, same code path, mentioned only so it is not lost

The refusal message for an unknown operator advertises $regex as supported:

Unsupported filter operator "$overlaps" on field "members". Supported operators:
$eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $between, $contains, $notContains,
$startsWith, $endsWith, $regex, $null, $exists.

$regex is in RETIRED_FILTER_OPERATORS per the #4706 ruling, so the error text is pointing authors
at a retired operator. Not worth its own issue — just noting it since it is emitted from the same area.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions