Skip to content

sum over a column that is NULL in every row of a group: driver-sql answers null, the engine's in-memory aggregate tier answers 0 — same query, same rows, fork chosen by a driver capability bit #15546

Description

@os-litant

Found while building the platform pin for #15330 (the compiled list-view group queries through POST /data/:object/query). Reported first on #14556 per that card's deliverable 3; filed here as its own card because #15330 is a tests card that changes no behaviour, and its two candidate repair sites were held by open PRs at the time.

The measurement

One ObjectQL engine, one better-sqlite3 SqlDriver, one seeded table, one compiled header query. The only thing that moves between the two runs is whether the driver exposes aggregate — which is exactly what ObjectQL.aggregate forks on (packages/objectql/src/engine.ts, the line reading if (typeof drv.aggregate === 'function' and allStructuredSupported and ...)).

Rows:

id business_unit amount notes
v1 void_unit NULL NULL
v2 void_unit NULL NULL
v3 void_unit NULL NULL
f1 full_unit 10 'x'
f2 full_unit 20 NULL

Query (the output of compileListViewGroupQuery for a view grouping on business_unit with a sum summary on amount), posted through POST /api/v1/data/:object/query:

{ "groupBy": ["business_unit"],
  "aggregations": [ { "function": "count", "alias": "count" },
                    { "function": "sum", "field": "amount", "alias": "sum_amount" } ] }

Answers for the void_unit group — a group with THREE rows, every one of them holding NULL in the summed column:

face sum_amount type
driver-sql pushdown (SQL SUM(amount)) null null
engine in-memory tier (applyInMemoryAggregation) 0 number
the spec's own reducer, packages/spec/src/ui/view-grouping-query.test.ts 0 number

avg, min and max agree (null on both faces), and count_distinct agrees (0 on both). sum is the only divergence. A group with SOME nulls also agrees, because applyInMemoryAggregation's toNumber(null) is 0 and SQL SUM skips NULLs — so the divergence needs a group in which the column is null in EVERY row, which is why nothing has caught it.

Which face is wrong

0, on the reading the platform has already written down. emptyGroupValueFor in packages/spec/src/data/aggregation-policy.ts rules that counting and summing no rows is 0 — "those are measured facts, not missing data" — while averaging, minimising or maximising nothing stays null. Summing three rows that each contribute nothing is the same act as summing no rows: the addend set is empty either way. So the in-memory tier and the spec reducer agree with the declared policy, and driver-sql is the face that disagrees.

Stated as an assumption to be confirmed rather than as a settled reading: emptyGroupValueFor is written for a group the runtime evaluates with an EMPTY row set, and this group is not empty — it is three rows whose aggregand is absent. If the maintainer reads those as different cases, then the ruling to record is which one this is, and the repair may land on the spec reducer instead. Either way the two platform faces must stop disagreeing.

Why it is unmeasured today

packages/spec/src/data/aggregation-conformance.ts is the vocabulary's conformance table and it does not reach this cell. Its stage column is nullable and every count_distinct case turns on it, but its two numeric aggregands are non-null by declaration — score is documented as "a non-null numeric column for the arithmetic aggregates" and flag is 3 true / 3 false. So no case in the table ever asks any face what sum over an all-null column is.

Nearest neighbours, same defect CLASS and different cells, all closed: #11065 (avg over a boolean: driver-memory null vs sqlite a number), #11151 (driver-mongodb, avg/sum over a boolean), #11455 / #11635 (driver-sql boolean aggregands on PG). The shape those cards settled — a face answering null where the ruled answer is a number — is this one, one column type over.

Blast radius

This is on the live compiled-query path, not a corner: a grouped list view with a sum summary on a nullable currency or number column is the ordinary case. The face that answers is picked by ObjectQL.aggregate per query — driver capability, advertised date granularity, reference timezone, per-aggregation filter — so the SAME view renders a blank group total on one deployment and 0 on another, with nothing in the response saying which happened. The measures reconcile everywhere else, which is the property that keeps this class alive.

What would close it

  1. A ruling on which value is correct for a non-empty group with an all-null aggregand (the reading above is a recommendation, not a decision).
  2. The losing face repaired — most likely driver-sql lowering SUM through a COALESCE, matching the identity emptyGroupValueFor already declares, and the sibling SQL drivers checked for the same cell.
  3. A case added to aggregation-conformance.ts so every enrolled face is held to it, since that table is what the enrolled faces are graded against.

Reproduction

The pin for #15330 (packages/rest/src/list-view-grouping-query-door.test.ts) boots the harness this was measured on: ObjectQL + sqlite SqlDriver + ObjectStackProtocolImplementation + RestServer, with the aggregate fork driven by shadowing driver.aggregate with an own property and restored with delete. That file deliberately does NOT pin this cell in either direction — pinning today's divergence would cement it, and the fixture it does pin has no all-null measure column.

Found by #15330. Reported on #14556.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions