Skip to content

feat: JSON-document aggregation + keyset pagination pushdown - #15

Merged
aryasaatvik merged 4 commits into
devfrom
feat/fumadb-pushdown
Jun 14, 2026
Merged

feat: JSON-document aggregation + keyset pagination pushdown#15
aryasaatvik merged 4 commits into
devfrom
feat/fumadb-pushdown

Conversation

@aryasaatvik

@aryasaatvik aryasaatvik commented Jun 14, 2026

Copy link
Copy Markdown
Owner

Pushes JSON-document aggregation and keyset pagination down to SQL.

FumaDB (packages/core/fumadb): adds jsonCount, jsonGroupCount, jsonTimeBuckets, jsonStats (min/max + continuous percentiles), and jsonPage (cursor/keyset) over a JSON document column, on the memory and Drizzle (SQLite/Postgres) adapters; unsupported adapters fail loudly. A shared JS evaluator keeps memory + the SQLite percentile fallback aligned with Postgres percentile_cont.

Plugin storage (packages/core/sdk): the facade exposes collection.aggregate.{count,groupCount,timeBuckets,stats} and collection.queryKeyset(...), translating a collection's indexed-field where into JSON-path predicates while owner/tenant scoping stays enforced by the storage policy.

Tests: fumadb 44 (incl. a 12-test suite run against both memory and real SQLite); sdk 341 (incl. a suite driving aggregate/keyset through the real Drizzle+SQLite executor). Repo-wide typecheck clean.

Not included: per-collection expression indexes (declared indexes still full-scan the partition — a perf follow-up); the Postgres path uses standard SQL but isn't unit-tested locally (no pglite).

First of a 3-PR stack: pushdown -> runs API -> runs UI.

Stack

  1. feat: JSON-document aggregation + keyset pagination pushdown #15 👈 current
  2. feat(execution-history): keyset runs API with aggregate meta #16
  3. feat(execution-history): openstatus-style runs UI #17

Add group-by counts, time buckets, min/max + continuous percentiles, and
cursor (keyset) pagination over values addressed inside a JSON document
column. The query layer gains jsonCount/jsonGroupCount/jsonTimeBuckets/
jsonStats/jsonPage on AbstractQuery, wired through toORM so the real-column
where still gets read-policy enforcement while JSON-path predicates conjoin
in. Implemented natively for the memory and Drizzle (SQLite via json_extract,
Postgres via #>> with casts and percentile_cont) adapters; adapters without
support throw a clear error. Shared JS evaluation (aggregate-eval) keeps the
memory adapter and the SQLite percentile fallback agreeing with Postgres
percentile_cont. Covered by a suite that runs identical assertions against
both memory and real SQLite.
Surface the new FumaDB JSON-document operations on the plugin storage facade:
collection.aggregate.{count,groupCount,timeBuckets,stats} and
collection.queryKeyset(...). The facade translates a collection's
indexed-field where into JSON-path predicates and forwards a plugin_id +
collection scoping condition; owner/tenant scoping stays enforced by the
existing storage owner policy, so results match the prior in-JS path. Wires
the five methods through makeSafeFumaQuery, CoreDb, and LooseStorageDb, and
adds them to the test executor's lazy-db method allowlist. Covered against the
real Drizzle+SQLite executor.
@greptile-apps

greptile-apps Bot commented Jun 14, 2026

Copy link
Copy Markdown

Greptile Summary

This PR pushes JSON-document aggregation and keyset pagination down to SQL, replacing in-memory full-collection scans with database-native COUNT, GROUP BY, time-bucket expressions, percentile_cont (Postgres) / sorted-array fallback (SQLite), and a null-aware multi-column keyset cursor — all plumbed through a new aggregate-eval.ts shared evaluator that keeps the memory and SQL adapters semantically aligned.

  • fumadb layer (aggregate.ts, aggregate-eval.ts, both adapters, orm/index.ts): five new ORMAdapter hooks (jsonCount, jsonGroupCount, jsonTimeBuckets, jsonStats, jsonPage) with a pure-JS evaluator that the memory adapter and the SQLite percentile fallback both use; adapters that don't implement the hooks throw loudly.
  • SDK layer (executor.ts, plugin-storage.ts): collection.aggregate.{count,groupCount,timeBuckets,stats} and collection.queryKeyset(...) translate indexed-field where objects into JsonFilter predicates, enforce index membership, and pass owner/tenant scoping through the existing table policy.
  • Tests: a 12-test harness run against both memory and real SQLite (covering null three-valued logic, LIKE-wildcard parity, and nullable-column keyset pagination), plus an SDK-level suite exercising the full Drizzle+SQLite pushdown path.

Confidence Score: 5/5

Safe to merge; all previously flagged blocking issues (null semantics for !=, null-cursor keyset truncation, Postgres path injection, LIKE wildcard mismatch) are correctly addressed in this revision.

The null-handling regressions and adapter divergences called out in the previous review round have been resolved with proper SQL three-valued logic throughout aggregate-eval.ts and the null-aware strictTerm keyset implementation in the Drizzle adapter. The two remaining open items (bucketMs=0 silent NaN and non-atomic two-query Postgres stats) are quality-of-life concerns rather than correctness blockers. Tests exercise both adapters, nullable columns, and LIKE-wildcard edge cases.

packages/core/fumadb/src/adapters/drizzle/query.ts — the jsonStats Postgres path still issues two independent queries without a transaction, and jsonTimeBuckets does not validate bucketMs > 0.

Important Files Changed

Filename Overview
packages/core/fumadb/src/adapters/drizzle/query.ts Drizzle adapter gains jsonCount/jsonGroupCount/jsonTimeBuckets/jsonStats/jsonPage; previously-flagged LIKE-escaping and Postgres path injection are fixed; null-aware keyset cursor (strictTerm) is implemented. The two-query jsonStats path (separate count+min+max and percentile_cont queries) is still non-atomic on Postgres, and bucketMs=0 still silently produces NULL/NaN.
packages/core/fumadb/src/query/aggregate-eval.ts New pure-JS evaluation layer shared by the memory adapter and the SQLite percentile fallback: extractJsonPath, coerceJsonValue, matchesJsonFilter, computePercentiles, compareNullableAscending. Previously flagged != null issue is now correctly handled with SQL three-valued logic.
packages/core/fumadb/src/query/aggregate.ts New type definitions for JSON-document aggregation and keyset pagination: JsonFilter discriminated union, JsonKeysetOrder/Cursor, adapter-facing and public-facing option types.
packages/core/sdk/src/executor.ts Adds pluginStorageWhereToJsonFilter (correctly maps eq/gt/gte/lt/lte/in operators), the queryKeyset handler with null-aware cursor generation, and four aggregate handlers. Indexed-field validation and limit guard are applied consistently before any pushdown call.
packages/core/fumadb/src/query/aggregate.test.ts 12-test suite run against both memory and real SQLite; covers count, groupCount, timeBuckets, stats, keyset paging, null-column paging, null three-valued logic, and LIKE-wildcard parity.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["collection.aggregate.count / groupCount / timeBuckets / stats\ncollection.queryKeyset(...)"] --> B["pluginStorageWhereToJsonFilter\n(indexed-field where to JsonFilter)"]
    B --> C["core.jsonCount / jsonGroupCount /\njsonTimeBuckets / jsonStats / jsonPage"]
    C --> D["toORM compileScopedWhere\n(applies table read policies)"]
    D --> E{Adapter?}
    E -->|Drizzle SQLite| F["jsonExtractSql to json_extract\nbuildJsonFilter WHERE clause\nnull-aware strictTerm keyset cursor"]
    E -->|Drizzle Postgres| G["jsonExtractSql col #>> type\npercentile_cont WITHIN GROUP\nnull-aware strictTerm keyset cursor"]
    E -->|Memory| H["extractJsonPath + matchesJsonFilter\ncomputePercentiles linear interpolation\ncompareNullableAscending sort+filter"]
    E -->|Other| I["throw FumaDB op not supported"]
    F --> J["aggregate-eval.ts\ncoerceJsonValue / computePercentiles\nSQLite percentile fallback"]
    H --> J
Loading

Reviews (3): Last reviewed commit: "fix(fumadb): match SQL null semantics in..." | Re-trigger Greptile

Comment thread packages/core/fumadb/src/query/aggregate-eval.ts Outdated
Comment thread packages/core/fumadb/src/adapters/drizzle/query.ts Outdated
…gination

The memory adapter's `=`/`!=`/`in`/`not in` now exclude rows whose JSON path is
null (SQL three-valued logic: a NULL comparison is unknown and dropped from a
WHERE), so it agrees with the Drizzle adapters. The Drizzle keyset cursor now
handles a null sort value: order-by emits explicit NULLS FIRST (asc) / NULLS
LAST (desc) on both dialects, and the boundary predicate mirrors
compareNullableAscending instead of emitting `col > NULL` (which silently
truncated every page once the cursor row's sort value was null). Adds
memory-vs-sqlite parity tests for null `!=`/`=` and nullable-column keyset paging.
@aryasaatvik
aryasaatvik merged commit e78112a into dev Jun 14, 2026
8 of 13 checks passed
aryasaatvik added a commit that referenced this pull request Jun 14, 2026
Builds on the JSON pushdown. Reworks the runs list into a
keyset-paginated, aggregate-aware read surface.

`list` now takes an opaque keyset cursor, a sort field
(`startedAt`/`durationMs`) + direction, and a live-tail `after` floor,
and returns `{ runs, nextCursor, meta }`. The `meta` block (computed
once per filter set on the first page) carries status/trigger facet
counts (each ignoring its own filter), interaction counts, a
stacked-by-status timeline with a server-chosen bucket width,
total/filtered row counts, and duration min/max + P50-P99. All counts,
buckets, percentiles, and pages are pushed to SQL.

Tests: 9 plugin tests on real SQLite (facets, cursor pages, live
`after`).

Not included: tool-path facets + full-text code search (need a
`contains` facade op).

Stacked on the FumaDB pushdown PR.

<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)

1. #15
2. **#16** 👈 current
3. #17
<!-- stack:links:end -->
aryasaatvik added a commit that referenced this pull request Jun 14, 2026
Rebuilds the runs page as an openstatus-style observability table on the
keyset + aggregate API, using the plugin's Effect-atoms idiom.

- Faceted filter rail (status/trigger/interaction/time with live
counts), cmdk filter palette, recharts stacked timeline with
drag-to-zoom, live tailing, keyset infinite scroll, sortable columns,
column visibility, keyboard shortcuts, and the existing 4-tab detail
drawer.
- List data flows through a `useRunsList` hook that accumulates cursor
pages and polls the head for live mode; the accumulation/dedup logic is
a pure, unit-tested reducer.
- Adds `recharts` as an optional peer dependency (provided by
`@executor-js/react`).

Verified: lint/format/typecheck clean, 15 plugin tests, build success,
repo-wide typecheck. An adversarial review pass caught and fixed 2 bugs
(a refresh that blanked page 1; a recharts axis misconfig) plus 4
correctness issues (live-divider off-by-one, live-anchor timing,
drag-cancel-on-leave, filter-merge clobbering).

Coverage gap: no browser/render/e2e test this round — the reducer is
unit-tested and the components were adversarially reviewed, but the
rendered page has not been driven live. Worth exercising by hand: live
divider, timeline drag-to-zoom, the cmdk parser, infinite scroll. Live
mode is append-only (existing rows' status changes are not re-pulled;
meta updates live). Filters are local state (not URL-persisted yet).

Stacked on the runs API PR.

<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)

1. #15
2. #16
3. **#17** 👈 current
<!-- stack:links:end -->
@aryasaatvik
aryasaatvik deleted the feat/fumadb-pushdown branch June 14, 2026 15:03
aryasaatvik added a commit that referenced this pull request Jun 23, 2026
Pushes JSON-document aggregation and keyset pagination down to SQL.

**FumaDB** (`packages/core/fumadb`): adds `jsonCount`, `jsonGroupCount`,
`jsonTimeBuckets`, `jsonStats` (min/max + continuous percentiles), and
`jsonPage` (cursor/keyset) over a JSON document column, on the memory
and Drizzle (SQLite/Postgres) adapters; unsupported adapters fail
loudly. A shared JS evaluator keeps memory + the SQLite percentile
fallback aligned with Postgres `percentile_cont`.

**Plugin storage** (`packages/core/sdk`): the facade exposes
`collection.aggregate.{count,groupCount,timeBuckets,stats}` and
`collection.queryKeyset(...)`, translating a collection's indexed-field
`where` into JSON-path predicates while owner/tenant scoping stays
enforced by the storage policy.

Tests: fumadb 44 (incl. a 12-test suite run against both memory and real
SQLite); sdk 341 (incl. a suite driving aggregate/keyset through the
real Drizzle+SQLite executor). Repo-wide typecheck clean.

Not included: per-collection expression indexes (declared indexes still
full-scan the partition — a perf follow-up); the Postgres path uses
standard SQL but isn't unit-tested locally (no pglite).

First of a 3-PR stack: pushdown -> runs API -> runs UI.

<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)

1. **#15** 👈 current
2. #16
3. #17
<!-- stack:links:end -->
aryasaatvik added a commit that referenced this pull request Jun 23, 2026
Builds on the JSON pushdown. Reworks the runs list into a
keyset-paginated, aggregate-aware read surface.

`list` now takes an opaque keyset cursor, a sort field
(`startedAt`/`durationMs`) + direction, and a live-tail `after` floor,
and returns `{ runs, nextCursor, meta }`. The `meta` block (computed
once per filter set on the first page) carries status/trigger facet
counts (each ignoring its own filter), interaction counts, a
stacked-by-status timeline with a server-chosen bucket width,
total/filtered row counts, and duration min/max + P50-P99. All counts,
buckets, percentiles, and pages are pushed to SQL.

Tests: 9 plugin tests on real SQLite (facets, cursor pages, live
`after`).

Not included: tool-path facets + full-text code search (need a
`contains` facade op).

Stacked on the FumaDB pushdown PR.

<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)

1. #15
2. **#16** 👈 current
3. #17
<!-- stack:links:end -->
aryasaatvik added a commit that referenced this pull request Jun 23, 2026
Rebuilds the runs page as an openstatus-style observability table on the
keyset + aggregate API, using the plugin's Effect-atoms idiom.

- Faceted filter rail (status/trigger/interaction/time with live
counts), cmdk filter palette, recharts stacked timeline with
drag-to-zoom, live tailing, keyset infinite scroll, sortable columns,
column visibility, keyboard shortcuts, and the existing 4-tab detail
drawer.
- List data flows through a `useRunsList` hook that accumulates cursor
pages and polls the head for live mode; the accumulation/dedup logic is
a pure, unit-tested reducer.
- Adds `recharts` as an optional peer dependency (provided by
`@executor-js/react`).

Verified: lint/format/typecheck clean, 15 plugin tests, build success,
repo-wide typecheck. An adversarial review pass caught and fixed 2 bugs
(a refresh that blanked page 1; a recharts axis misconfig) plus 4
correctness issues (live-divider off-by-one, live-anchor timing,
drag-cancel-on-leave, filter-merge clobbering).

Coverage gap: no browser/render/e2e test this round — the reducer is
unit-tested and the components were adversarially reviewed, but the
rendered page has not been driven live. Worth exercising by hand: live
divider, timeline drag-to-zoom, the cmdk parser, infinite scroll. Live
mode is append-only (existing rows' status changes are not re-pulled;
meta updates live). Filters are local state (not URL-persisted yet).

Stacked on the runs API PR.

<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)

1. #15
2. #16
3. **#17** 👈 current
<!-- stack:links:end -->
aryasaatvik added a commit that referenced this pull request Jun 23, 2026
Pushes JSON-document aggregation and keyset pagination down to SQL.

**FumaDB** (`packages/core/fumadb`): adds `jsonCount`, `jsonGroupCount`,
`jsonTimeBuckets`, `jsonStats` (min/max + continuous percentiles), and
`jsonPage` (cursor/keyset) over a JSON document column, on the memory
and Drizzle (SQLite/Postgres) adapters; unsupported adapters fail
loudly. A shared JS evaluator keeps memory + the SQLite percentile
fallback aligned with Postgres `percentile_cont`.

**Plugin storage** (`packages/core/sdk`): the facade exposes
`collection.aggregate.{count,groupCount,timeBuckets,stats}` and
`collection.queryKeyset(...)`, translating a collection's indexed-field
`where` into JSON-path predicates while owner/tenant scoping stays
enforced by the storage policy.

Tests: fumadb 44 (incl. a 12-test suite run against both memory and real
SQLite); sdk 341 (incl. a suite driving aggregate/keyset through the
real Drizzle+SQLite executor). Repo-wide typecheck clean.

Not included: per-collection expression indexes (declared indexes still
full-scan the partition — a perf follow-up); the Postgres path uses
standard SQL but isn't unit-tested locally (no pglite).

First of a 3-PR stack: pushdown -> runs API -> runs UI.

<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)

1. **#15** 👈 current
2. #16
3. #17
<!-- stack:links:end -->
aryasaatvik added a commit that referenced this pull request Jun 23, 2026
Builds on the JSON pushdown. Reworks the runs list into a
keyset-paginated, aggregate-aware read surface.

`list` now takes an opaque keyset cursor, a sort field
(`startedAt`/`durationMs`) + direction, and a live-tail `after` floor,
and returns `{ runs, nextCursor, meta }`. The `meta` block (computed
once per filter set on the first page) carries status/trigger facet
counts (each ignoring its own filter), interaction counts, a
stacked-by-status timeline with a server-chosen bucket width,
total/filtered row counts, and duration min/max + P50-P99. All counts,
buckets, percentiles, and pages are pushed to SQL.

Tests: 9 plugin tests on real SQLite (facets, cursor pages, live
`after`).

Not included: tool-path facets + full-text code search (need a
`contains` facade op).

Stacked on the FumaDB pushdown PR.

<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)

1. #15
2. **#16** 👈 current
3. #17
<!-- stack:links:end -->
aryasaatvik added a commit that referenced this pull request Jun 23, 2026
Rebuilds the runs page as an openstatus-style observability table on the
keyset + aggregate API, using the plugin's Effect-atoms idiom.

- Faceted filter rail (status/trigger/interaction/time with live
counts), cmdk filter palette, recharts stacked timeline with
drag-to-zoom, live tailing, keyset infinite scroll, sortable columns,
column visibility, keyboard shortcuts, and the existing 4-tab detail
drawer.
- List data flows through a `useRunsList` hook that accumulates cursor
pages and polls the head for live mode; the accumulation/dedup logic is
a pure, unit-tested reducer.
- Adds `recharts` as an optional peer dependency (provided by
`@executor-js/react`).

Verified: lint/format/typecheck clean, 15 plugin tests, build success,
repo-wide typecheck. An adversarial review pass caught and fixed 2 bugs
(a refresh that blanked page 1; a recharts axis misconfig) plus 4
correctness issues (live-divider off-by-one, live-anchor timing,
drag-cancel-on-leave, filter-merge clobbering).

Coverage gap: no browser/render/e2e test this round — the reducer is
unit-tested and the components were adversarially reviewed, but the
rendered page has not been driven live. Worth exercising by hand: live
divider, timeline drag-to-zoom, the cmdk parser, infinite scroll. Live
mode is append-only (existing rows' status changes are not re-pulled;
meta updates live). Filters are local state (not URL-persisted yet).

Stacked on the runs API PR.

<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)

1. #15
2. #16
3. **#17** 👈 current
<!-- stack:links:end -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant