implement a `reduce` operator & ZQL group-by on top of it #45

tantaman · 2024-03-20T05:26:18Z

All aggregate functions can be modeled as reduce

This adds a reduce operator so we can do:

group by
avg, sum, count, min, max, etc.
group_array

This targets SQL behavior for group-by.

src/zql/ast-to-ivm/pipeline-builder.ts

tantaman · 2024-03-20T06:27:56Z

note to self: pullHistory messages can stop at a reduce operator if the operator has already been seeded with history.

arv · 2024-03-22T10:23:34Z

src/zql/ivm/graph/operators/operator-index.ts

+      if (entries === undefined) {
+        continue;
+      }
+      this.#index.delete(key);


This changes the iteration order of the keys. Is that intentional? I think it would be simpler to only delete if length === 0.

not intentional but also shouldn't matter?

Updating to only delete on length === 0

Also, slightly more efficient because less tombstones on the map storage ;-)

tantaman · 2024-03-25T22:26:50Z

Not sure that I like the aggregation API:

https://github.com/rocicorp/rails/pull/45/files#diff-c88168576b354654e2ec061b5bbab48370b2e04ff6aeec9f2a52d741558d68daR449-R454

q
    .select('status')
    .groupBy('status')
    .array('assignee')
    .min('created');

maybe it should be:

q
    .select('status', agg.array('assignee'), agg.min('created'))
    .groupBy('status')

arv · 2024-03-25T22:36:03Z

Another strawman:

q
  .select('status').
  .agg('array', 'assignee')
  .agg('min', 'created')

But I think I prefer the first suggested API.

arv · 2024-03-25T22:38:37Z

src/zql/query/entity-query.ts

+  constructor(context: Context, tableName: string, ast?: AST) {
+    this.#ast = ast ?? {
+      table: tableName,
+      alias: aliasCount++,


What is the alias used for?

it'll be used whenever we compile the query to SQL.

I think I mentioned this before but since the same table could be present many times (through joins and sub-queries) it'll be simplest to give each occurrence a unique alias.

Maybe it is irrelevant to do at this step and the AST -> SQL compiler can do this itself.

new API: ```ts q.select('foo', agg.min('bar'), agg.array('baz')).groupBy('foo') ```

tantaman · 2024-03-27T14:51:33Z

updated the API to allow putting aggregate calls in select:

q
    .select(
      'status',
      agg.array('assignee'),
      agg.min('created', 'minCreated'),
      agg.max('created', 'maxCreated'),
    )

we could eventually support this.

src/zql/ast-to-ivm/pipeline-builder.test.ts

src/zql/ast-to-ivm/pipeline-builder.ts

arv · 2024-03-28T07:56:51Z

src/zql/ast/ast.ts

@@ -5,6 +5,18 @@
 // input to the query builder.
 export type Ordering = readonly [readonly string[], 'asc' | 'desc'];
 export type Primitive = string | number | boolean | null;
+
+// I think letting users provide their own lambda functions


That introduces the possibilities of exceptions in the pipeline

It also makes converting to SQL impossible.

Yeah, the lambdas would only be available client side. What is run server side would have to return a superset of the data.

src/zql/integration.test.ts

arv · 2024-03-28T08:32:44Z

src/zql/query/entity-query.ts

+    });
+  }
+
+  groupBy<K extends keyof S['fields']>(...x: K[]) {


Suggested change

groupBy<K extends keyof S['fields']>(...x: K[]) {

groupBy<K extends Selectable<S>>(...x: K[]) {

src/zql/query/entity-query.ts

src/zql/query/agg.ts

src/zql/query/entity-query.test.ts

src/zql/ivm/graph/operators/operator-index.ts

- type alias on `Aggregate< AsString<keyof S['fields']>, string >;` - aggrable - aggregable - remove operator-index - test flatMapIter - tree-shakable `agg` exports - remove unused await

arv · 2024-03-29T09:23:11Z

src/zql/util/iterables.ts

+ * Flat maps the items returned from the iterable.
+ *
+ * `iter` is a lambda that returns an iterable
+ * so this function can return an `IterableIterator`


Maybe it should take an IterableIterator then?

Also, These should be compatible with the standard methods to simplify transition to them later:

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Iterator/flatMap

going to iterate on this separately. IterableIterator is also getting exhausted on me :/

E.g.,

x = flatMapIter(iter, fn); console.log([...x]) console.log([...x])

second spread is empty :/

Or maybe a better question to answer for myself is why would we need to pull the iterable twice. Seems like it should only ever happen once.

To get the second log to be non empty x needs to be something that has [Symbol.iterator](): Iterator which returns a new iterator every time.

This API was something that I felt we didn't get quite right for ES6. It is always confusing.

tantaman commented Mar 20, 2024

View reviewed changes

src/zql/ast-to-ivm/pipeline-builder.ts Outdated Show resolved Hide resolved

tantaman force-pushed the mlaw/zql-hoist branch from 4ad041e to de48ae2 Compare March 20, 2024 21:27

tantaman force-pushed the mlaw/group-by branch from 58699cf to 9948e36 Compare March 20, 2024 21:27

arv reviewed Mar 22, 2024

View reviewed changes

tantaman force-pushed the mlaw/group-by branch from 7c429e5 to ca63558 Compare March 25, 2024 22:23

tantaman force-pushed the mlaw/zql-hoist branch from de48ae2 to 98a1c34 Compare March 25, 2024 22:24

tantaman changed the title ~~implement a reduce operator to support group-by~~ implement a reduce & group-by Mar 25, 2024

tantaman changed the title ~~implement a reduce & group-by~~ implement a reduce operator & ZQL group-by on top of it Mar 25, 2024

arv reviewed Mar 25, 2024

View reviewed changes

tantaman force-pushed the mlaw/zql-hoist branch from 98a1c34 to 58c54f6 Compare March 27, 2024 11:29

Base automatically changed from mlaw/zql-hoist to main March 27, 2024 11:34

tantaman and others added 6 commits March 27, 2024 07:48

implement a basic reduce operator to support group-by

64aed19

nits while stepping through tests

c6f0a11

apply group-by before select

d798d21

add group-by to EntityQuery

96b63ef

test group-by

22def11

remove from index only when conlsidated.length === 0

c288564

tantaman force-pushed the mlaw/group-by branch from ca63558 to c288564 Compare March 27, 2024 11:55

tantaman added 2 commits March 27, 2024 07:58

fix missed merges (removal of bigint and date)

28d90ab

add type signatures for revised aggregate API

f9fd1c0

new API: ```ts q.select('foo', agg.min('bar'), agg.array('baz')).groupBy('foo') ```

tantaman marked this pull request as draft March 27, 2024 14:32

tantaman added 2 commits March 27, 2024 10:43

update tests and implementation to use the new agg.fn api

d23b2cd

test aliased aggregates

b50bee2

tantaman marked this pull request as ready for review March 27, 2024 14:51

throw on aggregation without group-by

61c7714

we could eventually support this.

arv reviewed Mar 28, 2024

View reviewed changes

address review comments

dd09785

- type alias on `Aggregate< AsString<keyof S['fields']>, string >;` - aggrable - aggregable - remove operator-index - test flatMapIter - tree-shakable `agg` exports - remove unused await

arv approved these changes Mar 29, 2024

View reviewed changes

tantaman merged commit 6f1ba16 into main Mar 29, 2024
4 checks passed

tantaman deleted the mlaw/group-by branch March 29, 2024 19:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement a `reduce` operator & ZQL group-by on top of it #45

implement a `reduce` operator & ZQL group-by on top of it #45

tantaman commented Mar 20, 2024

tantaman commented Mar 20, 2024

arv Mar 22, 2024

tantaman Mar 27, 2024

arv Mar 28, 2024

tantaman commented Mar 25, 2024 •

edited

Loading

arv commented Mar 25, 2024

arv Mar 25, 2024

tantaman Mar 27, 2024

tantaman commented Mar 27, 2024

arv Mar 28, 2024

tantaman Mar 28, 2024

arv Mar 28, 2024

arv Mar 29, 2024

tantaman Mar 29, 2024

arv Apr 1, 2024

	groupBy<K extends keyof S['fields']>(...x: K[]) {
	groupBy<K extends Selectable<S>>(...x: K[]) {

implement a reduce operator & ZQL group-by on top of it #45

implement a reduce operator & ZQL group-by on top of it #45

Conversation

tantaman commented Mar 20, 2024

tantaman commented Mar 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tantaman commented Mar 25, 2024 • edited Loading

arv commented Mar 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tantaman commented Mar 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

implement a `reduce` operator & ZQL group-by on top of it #45

implement a `reduce` operator & ZQL group-by on top of it #45

tantaman commented Mar 25, 2024 •

edited

Loading