Skip to content

fix(objectql,driver-sql)!: a group key is the column's value, in the shape find() presents it (#3849) - #3864

Merged
os-zhuang merged 1 commit into
mainfrom
fix/3849-group-key-coercion
Jul 28, 2026
Merged

fix(objectql,driver-sql)!: a group key is the column's value, in the shape find() presents it (#3849)#3864
os-zhuang merged 1 commit into
mainfrom
fix/3849-group-key-coercion

Conversation

@os-zhuang

Copy link
Copy Markdown
Contributor

Closes #3849. Follow-up to #3848.

Outcome

groupBy: ['qty'] returns 3, not '3'. groupBy: ['won'] returns true/false — not 'true'/'false' on one path and 1/0 on the other. A bucket key is a column value, so there is one right answer for what it looks like: whatever that column looks like on a find() row.

What was wrong — and it was not only the String()

Three paths produce a group key. Measured on better-sqlite3 before this change:

qty (number) won (boolean)
find() 3 number true boolean
aggregate() pushed down 3 number 0 / 1 number
in-memory fallback '3' string 'false' / 'true' string

Two independent causes, and the issue only named the first:

  1. applyInMemoryAggregation ran every key through String(). The pushed-down path never did.
  2. The pushed-down path returns raw builder output. aggregate()/distinct() 不过 formatOutput —— SQLite 上 Field.datetime 的裸 epoch 从聚合出口漏出去(#3773 同根不同出口) #3797 taught it to present temporal columns the way formatOutput does on a find() row — but not the boolean and numeric repairs. A SQLite boolean has no native type and is stored 0/1, so it surfaced as an integer from aggregate() and a real boolean from find().

So deleting the String() alone would not have closed this: it would have left false/true against 0/1, which is still a mismatch under === and under Map lookup. Both halves are needed, and both are gated separately below.

engine.aggregate chooses between the two aggregate paths per query — native aggregate? advertised granularity? UTC? — so the same column changed shape with no change to the data or the query.

Why it mattered

Every measure stayed correct, which is why this survived. What broke is downstream code that probes a raw Map keyed by the value's own type. Map lookup is SameValueZero, so '1' never finds 1:

  • Select-option labels (dimension-labels.ts) — table keyed by the option's own value; a numeric option value never matched a stringified key, so the chart rendered the raw stored value.
  • Lookup / master-detail labels — the id → record-name table is built by an inner query that always pushes down (raw ids), then probed with the outer query's keys, which may be in-memory (stringified). With a numeric primary key — routine for external/federated objects — every label missed.
  • Cross-object rebucketing (cross-object-rebucket.ts) — same build-and-probe shape, but a miss is not a fallback, it is RESTRICTED_BUCKET. A numeric FK filed every row under '(restricted)': one bar, correct grand total, no error anywhere.
  • Drill-through — the raw dimension value goes into the drill filter verbatim, so a boolean dimension drilled from the in-memory path sent { won: 'true' } to SQLite, whose INTEGER column can never equal the text 'true'. Zero rows.

Direction

Converge on raw, i.e. on what find() presents. Every driver that aggregates natively already returns raw values (driver-sql, sqlite-wasm, memory, mongodb, and cloud's Turso remote transport); framework's own third in-memory grouper (preview-evaluator.ts) returns raw; memory-driver and cross-object-rebucket already stringify the internal key while emitting the value raw. applyInMemoryAggregation was the sole outlier.

Coercing in SQL instead would mean a CAST(… AS TEXT) in four driver implementations plus hand-written analytics SQL, one of them in a different repo and release train — and dialects disagree on the text form of a boolean ('t' on Postgres, 1 on MySQL/SQLite) and of a float, so it would trade a JS-vs-SQL divergence for a worse Postgres-vs-MySQL-vs-SQLite one. It would also universalize the label bugs above instead of fixing them.

No zod schema, TS type, or JSON schema constrains the shape either way. DatasetDimensionSchema.type explicitly admits number and boolean.

Changes

  • applyInMemoryAggregation (objectql) emits the value verbatim.
  • The internal composite bucket id is type-preserving, so 1 and '1', true and 'true' stay distinct groups rather than merging on the way in. BigInt is encoded explicitly — JSON.stringify throws on it, and a value that used to bucket under String() must not start crashing the aggregate.
  • SqlDriver.aggregate / .distinct present group keys and min/max results with the same rules formatOutput applies, generalizing aggregate()/distinct() 不过 formatOutput —— SQLite 上 Field.datetime 的裸 epoch 从聚合出口漏出去(#3773 同根不同出口) #3797 to boolean and numeric columns. The protected helpers are renamed to match (temporalFieldKindreadPresentationKind, presentTemporalValuepresentReadValue, presentTemporalColumnspresentReadColumns); the kind union is exported as ReadPresentationKind. Nothing outside sql-driver.ts referenced the old names, including cloud's TursoDriver.

Date-bucketed groupBy is unaffected — both sides produce canonical string labels, and #3839 pinned their empty bucket.

Gate

packages/qa/dogfood/test/group-key-read-shape-parity.test.ts measures both aggregate paths against find() for a number, boolean and text column, on driver-sql and driver-sqlite-wasm. It asserts the runtime type, not just the value — folding both sides through String() is the reflex that hid this for so long and would make the check pass against the bug it exists to catch.

Each half proven to redden it independently:

  • revert only the in-memory change → 6/8 red (number and boolean, both drivers)
  • revert only the driver change → 4/8 red, expected [ '0<number>', '1<number>' ] to deeply equal [ 'false<boolean>', 'true<boolean>' ]

Plus objectql unit pins for type preservation, for 1/'1'/true/'true'/null/'null' landing in six distinct buckets, and for a BigInt key not throwing.

Verification

  • Suites green: objectql 1154, driver-sql 401, driver-sqlite-wasm 126, service-analytics 292, runtime 694, dogfood 395.
  • Notably nothing existing broke — the string shape was essentially untested, which is consistent with it never having been a deliberate contract.
  • pnpm build clean, no generated-artifact drift; pnpm lint clean; check:nul-bytes / check:doc-authoring / check:role-word OK.

UI impact

Not browser-verified — this is a server-side change; I traced the render path in objectui source instead. formatDimensionValue sends integers through String(v) (identical output to before) and booleans through String(v)"true"/"false" (also identical). The one visible delta: a non-integer dimension value now goes through toLocaleString(maximumFractionDigits: 2), so grouping by a float with more than two decimals rounds in the label — the same rounding numbers already get everywhere else in the UI. Grouping by a float is unusual; flagging it rather than burying it.

🤖 Generated with Claude Code

…shape `find()` presents it (#3849)

Three code paths produce a `groupBy` key and no two agreed:

                          qty (number)   won (boolean)
  find()                  3  number      true   boolean
  aggregate() pushed down 3  number      0/1    number
  in-memory fallback      '3' string     'false'/'true' string

Two independent causes. `applyInMemoryAggregation` ran every key through
`String()`, which the pushed-down path never did. And the pushed-down path
returns raw builder output: #3797 taught it to present temporal columns the way
`formatOutput` does on a `find()` row, but not the boolean and numeric repairs,
so a SQLite boolean — no native type, stored 0/1 — surfaced as an integer from
`aggregate()` and a real boolean from `find()`.

`engine.aggregate` picks between the two aggregate paths per query, so the same
column changed shape with no change to the data or the query.

The measures were always right, which is why nobody noticed. What broke was
downstream code probing a raw `Map` keyed by the value's own type — `Map` lookup
is SameValueZero, so '1' never finds 1: select-option labels fell back to raw
values, lookup labels missed whenever the referenced object had a numeric id
(routine for external objects), and cross-object rebucketing filed every row
under RESTRICTED_BUCKET — one bar, correct grand total, no error. A boolean
dimension drilled from the in-memory path also sent { won: 'true' } to SQLite,
which an INTEGER column can never match.

- `applyInMemoryAggregation` emits the value verbatim; its rows are find()
  output, so passing through is what makes the key equal the column's read shape.
- The internal bucket id is type-preserving, so 1 and '1', true and 'true' stay
  distinct groups. BigInt is encoded explicitly — JSON.stringify throws on it,
  and a value that bucketed fine under String() must not start crashing.
- `SqlDriver.aggregate`/`.distinct` present group keys and min/max with the same
  rules `formatOutput` applies, generalizing #3797 to boolean and numeric
  columns. The protected helpers are renamed to match (temporalFieldKind →
  readPresentationKind, presentTemporalValue → presentReadValue,
  presentTemporalColumns → presentReadColumns) and the kind union is exported.

Date-bucketed groupBy is unaffected: both sides already produce canonical string
labels, and #3839 pinned their empty bucket.

Gate: group-key-read-shape-parity.test.ts measures both aggregate paths against
find() for number, boolean and text columns on driver-sql and
driver-sqlite-wasm, asserting the runtime TYPE — folding through String() is the
reflex that hid this and would pass against the bug. Each half was confirmed to
redden the gate on its own.

Co-Authored-By: Claude <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
objectstack Ignored Ignored Jul 28, 2026 12:07pm

Request Review

@github-actions github-actions Bot added documentation Improvements or additions to documentation tests tooling size/m labels Jul 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 3 package(s): @objectstack/objectql, @objectstack/driver-sql, packages/qa.

20 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

  • content/docs/concepts/metadata-lifecycle.mdx (via @objectstack/objectql)
  • content/docs/data-modeling/drivers.mdx (via @objectstack/driver-sql)
  • content/docs/data-modeling/formulas.mdx (via packages/objectql)
  • content/docs/deployment/migration-from-objectql.mdx (via @objectstack/objectql)
  • content/docs/deployment/vercel.mdx (via @objectstack/objectql)
  • content/docs/getting-started/glossary.mdx (via @objectstack/driver-sql)
  • content/docs/kernel/services-checklist.mdx (via @objectstack/objectql)
  • content/docs/kernel/services.mdx (via @objectstack/objectql)
  • content/docs/permissions/authentication.mdx (via @objectstack/objectql)
  • content/docs/permissions/authorization.mdx (via packages/qa)
  • content/docs/permissions/delegated-administration.mdx (via packages/qa)
  • content/docs/plugins/anatomy.mdx (via @objectstack/driver-sql)
  • content/docs/plugins/index.mdx (via @objectstack/objectql)
  • content/docs/plugins/packages.mdx (via @objectstack/objectql, @objectstack/driver-sql)
  • content/docs/protocol/kernel/index.mdx (via @objectstack/objectql, @objectstack/driver-sql)
  • content/docs/protocol/kernel/lifecycle.mdx (via @objectstack/driver-sql)
  • content/docs/protocol/objectql/query-syntax.mdx (via @objectstack/driver-sql)
  • content/docs/protocol/objectql/state-machine.mdx (via @objectstack/objectql)
  • content/docs/releases/implementation-status.mdx (via @objectstack/objectql, @objectstack/driver-sql)
  • content/docs/releases/v9.mdx (via @objectstack/objectql)

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.

@os-zhuang
os-zhuang merged commit 5d4de37 into main Jul 28, 2026
17 checks passed
@os-zhuang
os-zhuang deleted the fix/3849-group-key-coercion branch July 28, 2026 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/m tests tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

非空分组桶的键两条路也不一致:下推 SQL 保留原类型,内存兜底 String() 强转(布尔是语义分歧,不只是类型)

1 participant