Multicorn aggregation/grouping pushdown support #1

gruuya · 2021-12-13T13:16:34Z

Enable aggregation/grouping support offered in Multicorn through the accompanying PR splitgraph/Multicorn#1.

Provides a response to can_pushdown_upperrel with relevant details so that Multicorn can decide whether and what to push to the Python side.
Takes care of differences in the query and responses for the cases with and without aggregations.

Here are two instructive examples of the translated aggregation queries:

With GROUP BY

sgr@localhost:splitgraph> explain select column5, column4, min(column2) from es.iris group by column4, column5
+--------------------------------------------------------------------------------------------+
| QUERY PLAN                                                                                 |
|--------------------------------------------------------------------------------------------|
| Foreign Scan  (cost=1.00..1.00 rows=1 width=1)                                             |
|   Multicorn: Elasticsearch query to <Elasticsearch([{'host': 'es01-test', 'port': 9200}])> |
|   Multicorn: Query: {                                                                      |
|     "aggs": {                                                                              |
|         "group_buckets": {                                                                 |
|             "composite": {                                                                 |
|                 "sources": [                                                               |
|                     {                                                                      |
|                         "column5": {                                                       |
|                             "terms": {                                                     |
|                                 "field": "column5"                                         |
|                             }                                                              |
|                         }                                                                  |
|                     },                                                                     |
|                     {                                                                      |
|                         "column4": {                                                       |
|                             "terms": {                                                     |
|                                 "field": "column4"                                         |
|                             }                                                              |
|                         }                                                                  |
|                     }                                                                      |
|                 ],                                                                         |
|                 "size": 1000                                                               |
|             },                                                                             |
|             "aggregations": {                                                              |
|                 "min.column2": {                                                           |
|                     "min": {                                                               |
|                         "field": "column2"                                                 |
|                     }                                                                      |
|                 }                                                                          |
|             }                                                                              |
|         }                                                                                  |
|     }                                                                                      |
| }                                                                                          |
+--------------------------------------------------------------------------------------------+

Without GROUP BY

sgr@localhost:splitgraph> explain analyze select sum(column4), max(column4), sum(column4), min(column2), avg(column3)  from es.iris
+--------------------------------------------------------------------------------------------+
| QUERY PLAN                                                                                 |
|--------------------------------------------------------------------------------------------|
| Foreign Scan  (cost=1.00..1.00 rows=1 width=1) (actual time=2.344..2.350 rows=1 loops=1)   |
|   Multicorn: Elasticsearch query to <Elasticsearch([{'host': 'es01-test', 'port': 9200}])> |
|   Multicorn: Query: {                                                                      |
|     "aggs": {                                                                              |
|         "sum.column4": {                                                                   |
|             "sum": {                                                                       |
|                 "field": "column4"                                                         |
|             }                                                                              |
|         },                                                                                 |
|         "max.column4": {                                                                   |
|             "max": {                                                                       |
|                 "field": "column4"                                                         |
|             }                                                                              |
|         },                                                                                 |
|         "min.column2": {                                                                   |
|             "min": {                                                                       |
|                 "field": "column2"                                                         |
|             }                                                                              |
|         },                                                                                 |
|         "avg.column3": {                                                                   |
|             "avg": {                                                                       |
|                 "field": "column3"                                                         |
|             }                                                                              |
|         }                                                                                  |
|     }                                                                                      |
| }                                                                                          |
| Planning Time: 3.123 ms                                                                    |
| Execution Time: 2.467 ms                                                                   |
+--------------------------------------------------------------------------------------------+

CU-1t1wycg

mildbyte · 2021-12-14T20:00:13Z

pg_es_fdw/_es_query.py

    """Convert a list of Multicorn quals to an ElasticSearch query"""
    ignore_columns = ignore_columns or []
+
+    # Aggreagtion/grouping queries


typo: aggregation

mildbyte · 2021-12-14T20:06:36Z

pg_es_fdw/_es_query.py

+            }
+        }
+
+        if aggs is not None:


Can we be in a situation where aggs is None and group_clauses isn't? Is it basically something like SELECT a, b, c FROM T GROUP BY a, b, c which is the same as SELECT DISTINCT a, b, c FROM T?

Yes we can, that is a good example. Here's a concrete one:

sgr@localhost:splitgraph> explain select column5, column4 from es.iris group by column4, column5 +--------------------------------------------------------------------------------------------+ | QUERY PLAN | |--------------------------------------------------------------------------------------------| | Foreign Scan (cost=1.00..1.00 rows=1 width=1) | | Multicorn: Elasticsearch query to <Elasticsearch([{'host': 'es01-test', 'port': 9200}])> | | Multicorn: Query: { | | "aggs": { | | "group_buckets": { | | "composite": { | | "sources": [ | | { | | "column5": { | | "terms": { | | "field": "column5" | | } | | } | | }, | | { | | "column4": { | | "terms": { | | "field": "column4" | | } | | } | | } | | ], | | "size": 1000 | | } | | } | | } | | } | +--------------------------------------------------------------------------------------------+ EXPLAIN Time: 0.012s sgr@localhost:splitgraph> select column5, column4 from es.iris group by column4, column5 +-----------------+---------+ | column5 | column4 | |-----------------+---------| | Iris-setosa | 0.1 | | Iris-setosa | 0.2 | | Iris-setosa | 0.3 | | Iris-setosa | 0.4 | | Iris-setosa | 0.5 | | Iris-setosa | 0.6 | | Iris-versicolor | 1.0 | | Iris-versicolor | 1.1 | | Iris-versicolor | 1.2 | | Iris-versicolor | 1.3 | | Iris-versicolor | 1.4 | | Iris-versicolor | 1.5 | | Iris-versicolor | 1.6 | | Iris-versicolor | 1.7 | | Iris-versicolor | 1.8 | | Iris-virginica | 1.4 | | Iris-virginica | 1.5 | | Iris-virginica | 1.6 | | Iris-virginica | 1.7 | | Iris-virginica | 1.8 | | Iris-virginica | 1.9 | | Iris-virginica | 2.0 | | Iris-virginica | 2.1 | | Iris-virginica | 2.2 | | Iris-virginica | 2.3 | | Iris-virginica | 2.4 | | Iris-virginica | 2.5 | +-----------------+---------+

mildbyte · 2021-12-14T20:08:18Z

Would be nice to have some unit tests for this ES query handling (converting aggs/group_clauses into ES queries and back) + ~~handling the EXPLAIN/get_rel_size in Multicorn as you mentioned~~ (I see EXPLAIN works fine here?). LGTM otherwise.

UPD Dec 21 -- the tests will live in the splitgraph repo (including checking the ES queries) so this is fine

gruuya added 5 commits December 2, 2021 13:55

Enable aggregation pushdown minimal PoC working

3f59b82

Enable multi-aggregation pushdown

257bf01

Add support for pushdowns too

4de349f

Add upper rel pushdown details for Multicorn to parse

4c3fa96

Increase grouping page size and format code

afa2d3d

gruuya requested a review from mildbyte December 13, 2021 13:16

gruuya self-assigned this Dec 13, 2021

gruuya mentioned this pull request Dec 13, 2021

Implement aggregation and grouping pushdown splitgraph/Multicorn#1

Merged

gruuya added 2 commits December 14, 2021 10:26

Return adequate results for on aggregation queries

6ec877b

Enable COUNT() propagation to ES server

19ee117

mildbyte reviewed Dec 14, 2021

View reviewed changes

mildbyte approved these changes Dec 14, 2021

View reviewed changes

Fix typo in comment

1aecbde

gruuya mentioned this pull request Dec 17, 2021

Enable aggregation and grouping pushdown in the engine splitgraph/sgr#581

Merged

gruuya merged commit c47afb3 into master Dec 27, 2021

gruuya deleted the es-multicorn-agg-pushdown-poc-cu-1t1wycg branch December 27, 2021 15:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multicorn aggregation/grouping pushdown support #1

Multicorn aggregation/grouping pushdown support #1

gruuya commented Dec 13, 2021 •

edited

mildbyte Dec 14, 2021

mildbyte Dec 14, 2021

gruuya Dec 15, 2021

mildbyte commented Dec 14, 2021 •

edited

Multicorn aggregation/grouping pushdown support #1

Multicorn aggregation/grouping pushdown support #1

Conversation

gruuya commented Dec 13, 2021 • edited

mildbyte Dec 14, 2021

Choose a reason for hiding this comment

mildbyte Dec 14, 2021

Choose a reason for hiding this comment

gruuya Dec 15, 2021

Choose a reason for hiding this comment

mildbyte commented Dec 14, 2021 • edited

gruuya commented Dec 13, 2021 •

edited

mildbyte commented Dec 14, 2021 •

edited