-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding array functions pushdown to pinot #15260
Adding array functions pushdown to pinot #15260
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am having second thoughts about ARRAY_AGGREGATION_ALLOWED_PUSHDOWN_MAP: I am wondering if we should ditch the map and just have a case statement to handle the 4 or 5 of those keys separately ?
They are all somewhat custom anyway ... sometimes we special case on things like array_min/array_max and sometimes we look up the map.
I was wondering if the code would be simpler by having a couple of helper methods with case expressions instead of a map, to account for the customness.
...-toolkit/src/main/java/com/facebook/presto/pinot/query/PinotAggregationProjectConverter.java
Outdated
Show resolved
Hide resolved
...-toolkit/src/main/java/com/facebook/presto/pinot/query/PinotAggregationProjectConverter.java
Outdated
Show resolved
Hide resolved
...-toolkit/src/main/java/com/facebook/presto/pinot/query/PinotAggregationProjectConverter.java
Show resolved
Hide resolved
492b7b6
to
0301b8e
Compare
I overall feel, it would be simpler to have both map and switch case so we can have some common code logics to handle the function name conventions. This is the first PR of pushing down array functions and I only implemented 4 functions. There would be more array functions to be pushed down. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay. I am fine with leaving the map there.
...-toolkit/src/main/java/com/facebook/presto/pinot/query/PinotAggregationProjectConverter.java
Show resolved
Hide resolved
0301b8e
to
10b39e8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Please make sure to update the PR description with a test plan that enumerates some examples of what all presto array expressions can now be pushed into Pinot.
2fc2ef1
to
83f91ee
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was wondering how type checking happens for these new array functions ? How do we check that the arguments to array_sum and contains for example are indeed arrays ?
What happens if I invoke those functions on scalar columns ?
Should we add some negative tests ?
...ot-toolkit/src/main/java/com/facebook/presto/pinot/query/PinotFilterExpressionConverter.java
Outdated
Show resolved
Hide resolved
...ot-toolkit/src/main/java/com/facebook/presto/pinot/query/PinotFilterExpressionConverter.java
Outdated
Show resolved
Hide resolved
462536f
to
81ca2d2
Compare
...ot-toolkit/src/main/java/com/facebook/presto/pinot/query/PinotFilterExpressionConverter.java
Outdated
Show resolved
Hide resolved
81ca2d2
to
442e0bc
Compare
...ot-toolkit/src/main/java/com/facebook/presto/pinot/query/PinotFilterExpressionConverter.java
Outdated
Show resolved
Hide resolved
...-toolkit/src/main/java/com/facebook/presto/pinot/query/PinotAggregationProjectConverter.java
Outdated
Show resolved
Hide resolved
442e0bc
to
52f19fb
Compare
Support pushing down Array functions to Pinot.
array_sum
,array_min
,array_max
,array_average
) inside aggregation functions or group by clause to Pinot.contains
E.g. For Presto query:
SELECT sum(array_max(col)) FROM myTable WHERE contains(col, 1)
, the pushdown pinot query is:SELECT sum(arrayMax(col)) FROM myTable WHERE col = 1
Test plan:
Unit tests and tested with local setup for array functions(
array_sum
,array_min
,array_max
,array_average
,contains
) pushdown to Pinot: