[CALCITE-3679] Allow lambda expressions in SQL queries #3502

macroguo-ghy · 2023-11-04T07:03:31Z

This PR is about three things:

Implementing parsing, validating, and converting SqlNode to RelNode, and RelNode to SqlNode for lambda expressions.
Implementing the conversion of lambda expression RexNode to Linq4j Expression to execute higher order functions.
Adding the EXISTS function, which is enabled in the Spark library.

Parse

Lambda expression is only accepted as a parameter of function call.

select func(1, (x, y) -> x + y); // accepted
select (x,y) -> x + y; // illegal for now

Validate

I add a new type LAMBDA_EXPRESSION to describe type of lambda expression.
For example:

  public static final SqlFunction EXISTS =
      SqlBasicFunction.create("EXISTS",
          ReturnTypes.BOOLEAN_NULLABLE,
          OperandTypes.sequence("EXISTS(<ARRAY>, <FUNCTION(ANY)->BOOLEAN>)",
              OperandTypes.ARRAY, OperandTypes.lambda(SqlTypeFamily.BOOLEAN, SqlTypeFamily.ANY)));

The type of lambda in exists function is a function type where the input parameter is of type ANY and the return type is of type BOOLEAN.
The lambda expression will be validated twice. For the first time, calcite will validate the lambda expression itself, which means all the types of parameters of lambda expression are considered as ANY. For example, (x, y) -> x + y, both x and y are ANY type, we can deduce that x + y is alsoANY, so the type of (x, y) -> x + y is FUNCTION(ANY, ANY) -> ANY. And then we need to know if the type of lambda is legal in a function, this is what LambdaExpressionOperandTypeChecker do. It will reset the type of parameters, and infer the type of lambda expression, and check if the derived type satisfy this checker. After the checker, we can infer the real type of lamda expression.

RelNode

In this part, I create a class RexLambdaRef to reference the parameter of lambda expression. From another perspective, we can view lambda expressions as "constants" rather than tables, so we cannot directly use 'RexInputRef' and 'RexLocalRef'`.

I referred to #1733, so I listed Ritesh as a co-author.

mihaibudiu · 2023-11-05T19:52:02Z

Is there a design document for this feature?
This looks extremely useful, but before I review it I would like to have a big picture understanding of how it's supposed to work.
For example, is there type inference for lambdas?

Moreover, if EXISTS is independent on the lambdas perhaps it should be in a separate PR.

mihaibudiu · 2023-11-05T19:53:14Z

Leaving aside the design document, I don't see any changes to the documentation files either.

mihaibudiu · 2023-11-07T00:32:28Z

The real tests of whether the design works properly is to allow a lambda expression wherever a function name is allowed.
So you should be allowed to say SELECT (x -> x + 1)(age) FROM Person or
SELECT ":" || ((x -> x || x)(SUBSTRING(name, 3, 3))) FROM Person WHERE (x -> x > 5)(age).
I think there is limited value to having a lambda expression that only works in some very restricted contexts.

macroguo-ghy · 2023-11-13T10:53:52Z

Hi @mihaibudiu, thanks for your review. I have made some commits to address your feedback:

Add docs about lambda expressions and higher-order functions.
Add some negative tests in SqlParserTest.
Log a new jira case CALCITE-6116 to track the EXISTS function.

julianhyde

This looks very good.

I've suggested quite a few changes to names and scope (e.g. removing public) to make this easier to maintain.

Can you add a quidem test? I would like to see this working end-to-end.

julianhyde · 2023-11-14T19:25:21Z

core/src/main/java/org/apache/calcite/rex/RexLambdaRef.java

+ */
+public class RexLambdaRef extends RexInputRef {
+
+  private final String paramName;


could this be an ordinal?

After making RexLambdaRef extend RexSlot, we can now use the name, so there is no need for the paramName anymore.

julianhyde · 2023-11-14T19:25:32Z

core/src/main/java/org/apache/calcite/rex/RexLambdaRef.java

+import org.apache.calcite.sql.SqlKind;
+
+/**
+ * Variable which references a field of a lambda expression.


s/which/that/

julianhyde · 2023-11-14T19:27:11Z

core/src/main/java/org/apache/calcite/rex/RexLambdaExpression.java

+  }
+
+  @Override public boolean equals(@Nullable Object o) {
+    if (this == o) {


can you use the compact form 'this == o || o instanceof RexLambdaExpression & ...'?

no need for Objects.equals. use expression.equals because it's not-null.

julianhyde · 2023-11-14T19:27:40Z

core/src/main/java/org/apache/calcite/rex/RexLambdaExpression.java

+  //~ Constructors -----------------------------------------------------------
+
+  RexLambdaExpression(List<RexLambdaRef> parameters, RexNode expression) {
+    this.expression = expression;


ImmutbleList.copyOf(parameters)
requireNonNull(expression)

julianhyde · 2023-11-14T19:28:33Z

core/src/main/java/org/apache/calcite/rex/RexLambdaExpression.java

+/**
+ * Represents a lambda expression.
+ */
+public class RexLambdaExpression extends RexNode {


rename RexLambdaExpression to RexLambda. give example.

julianhyde · 2023-11-14T20:07:40Z

core/src/main/codegen/templates/Parser.jj

@@ -1060,6 +1064,9 @@ void AddArg(List<SqlNode> list, ExprContext exprContext) :
    )
    (
        e = Default()
+    |
+        LOOKAHEAD((SimpleIdentifierOrList() | <LPAREN> <RPAREN>) <LAMBDA_OPERATOR>)
+        e = LambdaExpression()


How expensive is this LOOKAHEAD?

julianhyde · 2023-11-14T20:09:10Z

core/src/main/codegen/templates/Parser.jj

+        LOOKAHEAD(2)
+        <LPAREN> <RPAREN> { parameters = SqlNodeList.EMPTY; }
+    |
+        parameters = SimpleIdentifierOrList()


should there be a rule that allows (), x and (x [, y]*)?

I extract this rule to SimpleIdentifierOrListOrEmpty.

julianhyde · 2023-11-14T20:09:31Z

core/src/main/codegen/templates/Parser.jj

@@ -8787,6 +8817,7 @@ void NonReservedKeyWord2of3() :
 |   < NE2: "!=" >
 |   < PLUS: "+" >
 |   < MINUS: "-" >
+|   < LAMBDA_OPERATOR: "->" >


just LAMBDA

julianhyde · 2023-11-14T20:10:06Z

core/src/main/java/org/apache/calcite/rel/rel2sql/SqlImplementor.java

+        }
+        final SqlNode expression = toSql(program, lambda.getExpression());
+        return new SqlLambdaExpression(POS, parameters, expression);
+      case LAMBDA_REF:


add blank line before

julianhyde · 2023-11-14T20:13:12Z

core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java

+        .fails("Cannot apply '(?s).*HIGHER_ORDER_FUNCTION' to arguments of type "
+            + "'HIGHER_ORDER_FUNCTION\\(<INTEGER>, <FUNCTION\\(ANY, ANY, ANY\\) -> ANY>\\)'.*");
+  }
+


need to see more tests for EXISTS. including negative tests. wrong type of arguments, wrong number of arguments, arguments of wrong name, arguments of wrong case.

check type derivation (if you didn't do it in SqlOperatorTest)

julianhyde · 2023-11-14T20:16:08Z

core/src/main/java/org/apache/calcite/rel/type/RelDataTypeFactory.java

@@ -134,6 +134,18 @@ RelDataType createMapType(
      RelDataType keyType,
      RelDataType valueType);

+  /**
+   * Create a lambda expression type. Lambda expressions are functions that


s/Create/Creates/

julianhyde · 2023-11-14T20:19:32Z

core/src/main/java/org/apache/calcite/rex/RexLambdaExpression.java

+    return Objects.hash(expression, parameters);
+  }
+
+  @Override public String toString() {


why override toString rather than computeDigest?

I think it would be clearer and more concise if you use a for loop rather than functional style.

Best thing is to do what RexMRAggCall does - make the class final and initialize digest in the constructor

Fixed, I use for loop instead.
RexLambdaExpression extends RexNode, the computeDigest is defined in RexCall.

macroguo-ghy

Thanks for your review and encouragement, @julianhyde. I have made some changes based on your comments. And, I will be adding an end-to-end test later.

macroguo-ghy · 2023-11-15T05:54:28Z

core/src/main/codegen/templates/Parser.jj

@@ -1033,6 +1034,9 @@ void AddArg0(List<SqlNode> list, ExprContext exprContext) :
    )
    (
        e = Default()
+    |
+        LOOKAHEAD((SimpleIdentifierOrList() | <LPAREN> <RPAREN>) <LAMBDA_OPERATOR>)
+        e = LambdaExpression()


The most important token is ->, so the expense depends on how many tokens come before ->.
For example:

() -> true // equals LOOKAHEAD(3) (a, b) -> a + b // equals LOOKAHED(6)

macroguo-ghy · 2023-11-15T06:25:14Z

core/src/main/codegen/templates/Parser.jj

+        LOOKAHEAD(2)
+        <LPAREN> <RPAREN> { parameters = SqlNodeList.EMPTY; }
+    |
+        parameters = SimpleIdentifierOrList()


I extract this rule to SimpleIdentifierOrListOrEmpty.

macroguo-ghy · 2023-11-15T06:25:48Z

core/src/main/codegen/templates/Parser.jj

@@ -8787,6 +8817,7 @@ void NonReservedKeyWord2of3() :
 |   < NE2: "!=" >
 |   < PLUS: "+" >
 |   < MINUS: "-" >
+|   < LAMBDA_OPERATOR: "->" >


macroguo-ghy · 2023-11-15T06:26:26Z

core/src/main/java/org/apache/calcite/rel/rel2sql/SqlImplementor.java

+        }
+        final SqlNode expression = toSql(program, lambda.getExpression());
+        return new SqlLambdaExpression(POS, parameters, expression);
+      case LAMBDA_REF:


macroguo-ghy · 2023-11-15T06:27:58Z

core/src/main/java/org/apache/calcite/rel/type/RelDataTypeFactory.java

@@ -134,6 +134,18 @@ RelDataType createMapType(
      RelDataType keyType,
      RelDataType valueType);

+  /**
+   * Create a lambda expression type. Lambda expressions are functions that


macroguo-ghy · 2023-11-17T09:57:51Z

core/src/main/java/org/apache/calcite/sql/type/SqlTypeFamily.java

@@ -77,6 +77,7 @@ public enum SqlTypeFamily implements RelDataTypeFamily {
  CURSOR,
  COLUMN_LIST,
  GEO,
+  LAMBDA_EXPRESSION,


macroguo-ghy · 2023-11-17T10:04:02Z

core/src/main/java/org/apache/calcite/sql/validate/SqlLambdaExpressionScope.java

+    super(parent);
+    this.lambdaExpr = lambdaExpr;
+
+    // default parameter type is ANY


In fact, the lambda expression will be validated twice.

A simple lambda expression a -> a, we can not infer the type of lambda if we just know the lambda expression, this is why I say "default parameter type is ANY".

But if we see the lambda expression in a function, we will validate the lambda expression second time in LambdaExpressionOperandTypeChecker.

For example, give a function test_func1, whose type checker is

OperandTypes.sequence("TEST_FUNC1(INTEGER, FUNCTION(STRING) -> NUMERIC)", OperandTypes.family(SqlTypeFamily.INTEGER), OperandTypes.function(SqlTypeFamily.NUMERIC, SqlTypeFamily.STRING))

Assuming there is an SQL statement,

select test_func1(1, x -> x / 2);

The first validation:
we assume the type of x is ANY. We can know the x -> x / 2 is legal.
The second validation:
Based on the type checker, the type of x is string, so x / 2 is not a valid expression.

macroguo-ghy · 2023-11-17T10:47:16Z

core/src/main/java/org/apache/calcite/sql/validate/SqlLambdaExpressionScope.java

+ */
+public class SqlLambdaExpressionScope extends ListScope {
+  private final SqlLambdaExpression lambdaExpr;
+  private final Map<String, RelDataType> parameterTypes;


We need to register the type of parameter to SqlLambdaExpressionScope when validate lambda expression second time, so I use the map as cache.

macroguo-ghy · 2023-11-17T11:34:17Z

core/src/main/java/org/apache/calcite/sql/type/LambdaExpressionSqlType.java

+
+/**
+ * SQL lambda expression type.
+ */


I have renamed this class to FunctionSqlType. But I think all the RelDataType classes are public, FunctionSqlType should be consistent with them.

macroguo-ghy · 2023-11-17T11:41:48Z

testkit/src/main/java/org/apache/calcite/test/MockSqlOperatorTable.java

@@ -623,4 +625,24 @@ public CompositeFunction() {
      return typeFactory.createSqlType(SqlTypeName.BIGINT);
    }
  }
+
+  private static final SqlFunction HIGHER_ORDER_FUNCTION =
+      new SqlFunction("HIGHER_ORDER_FUNCTION",


mihaibudiu · 2023-11-21T18:07:47Z

Do the lambdas allow capturing values?
SELECT EXISTS(T.a, x -> x < T.VALUE) FROM T

macroguo-ghy · 2023-12-12T11:51:57Z

Thank you for your review. I apologize for the delayed response; I have been quite busy over the past few weeks.

Regarding this PR, I have made modifications to almost all of the reviews. And then, I will write more tests. Once that is done, I will split this PR into two parts: one for lambda and another for the exists function

Co-authored-by: Ritesh Kapoor <riteshkapoor.opensource@gmail.com>

sonarcloud · 2023-12-19T04:48:33Z

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

12 New issues
0 Security Hotspots
72.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

macroguo-ghy force-pushed the lambda branch from 132ea4c to f3abbd7 Compare November 5, 2023 05:46

macroguo-ghy force-pushed the lambda branch 2 times, most recently from 98e238c to 340a603 Compare November 6, 2023 09:00

libenchao force-pushed the main branch from 47db81a to 0be8eae Compare November 10, 2023 13:18

macroguo-ghy force-pushed the lambda branch from 340a603 to 0bd9b66 Compare November 13, 2023 05:53

julianhyde requested changes Nov 14, 2023

View reviewed changes

julianhyde reviewed Nov 14, 2023

View reviewed changes

macroguo-ghy commented Nov 17, 2023

View reviewed changes

macroguo-ghy force-pushed the lambda branch 4 times, most recently from eaacd78 to 7a2d043 Compare December 13, 2023 09:13

[CALCITE-3679] Allow lambda expressions in SQL queries

d3b0b4d

Co-authored-by: Ritesh Kapoor <riteshkapoor.opensource@gmail.com>

macroguo-ghy force-pushed the lambda branch from 7a2d043 to d3b0b4d Compare December 19, 2023 04:12

macroguo-ghy merged commit 6f64865 into apache:main Dec 19, 2023
17 checks passed

MasseGuillaume mentioned this pull request Jan 16, 2024

[CALCITE-3679] Allow lambda expressions in SQL queries #1733

Closed

[CALCITE-3679] Allow lambda expressions in SQL queries #3502

[CALCITE-3679] Allow lambda expressions in SQL queries #3502

Conversation

macroguo-ghy commented Nov 4, 2023 • edited

Parse

Validate

RelNode

mihaibudiu commented Nov 5, 2023

mihaibudiu commented Nov 5, 2023

mihaibudiu commented Nov 7, 2023 • edited

macroguo-ghy commented Nov 13, 2023

julianhyde left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

macroguo-ghy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mihaibudiu commented Nov 21, 2023

macroguo-ghy commented Dec 12, 2023

sonarcloud bot commented Dec 19, 2023

Quality Gate passed

macroguo-ghy commented Nov 4, 2023 •

edited

mihaibudiu commented Nov 7, 2023 •

edited