-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-21848][SQL] Add trait UserDefinedExpression to identify user-defined functions #19064
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why a trait? It doesn't seem to be used other than to mark.
@@ -22,6 +22,9 @@ import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow} | |||
import org.apache.spark.sql.catalyst.expressions.codegen._ | |||
import org.apache.spark.sql.types.DataType | |||
|
|||
// Trait for identifying user-defined functions. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/**
* Common base trait for user-defined functions, including ScalaUDF, ScalaUDAF, PythonUDF,
* HiveSimpleUDF, HiveGenericUDF, and HiveUDAFFunction.
*/
Yes. This is a potential optimization we should do, but it should be cost based. Let us improve the resolution logics using this common base trait in this PR first. |
* Common base trait for user-defined functions, including ScalaUDF, ScalaUDAF, PythonUDF, | ||
* HiveSimpleUDF, HiveGenericUDF, and HiveUDAFFunction. | ||
*/ | ||
trait UDFType |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe UserDefinedExpression?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also sort of weird to put this in ScalaUDF.scala. Either put it in Expression.scala, or create its own file ...
Test build #81174 has finished for PR 19064 at commit
|
Test build #81177 has finished for PR 19064 at commit
|
I'm +1 on this, it should be useful to pattern match the trait instead of handling ScalaUDF/ScalaUDAF/PythonUDF... in future optimize rules. |
Test build #81192 has finished for PR 19064 at commit
|
It is better to revise PR title and description to reflect the change from |
Test build #81202 has finished for PR 19064 at commit
|
retest this please. |
Test build #81204 has finished for PR 19064 at commit
|
LGTM |
Thanks! Merged to master. |
What changes were proposed in this pull request?
Add trait UserDefinedExpression to identify user-defined functions.
UDF can be expensive. In optimizer we may need to avoid executing UDF multiple times.
E.g.
If UDF is expensive in this case, optimizer should not collapse the project to
Currently UDF classes like PythonUDF, HiveGenericUDF are not defined in catalyst.
This PR is to add a new trait to make it easier to identify user-defined functions.
How was this patch tested?
Unit test