[SPARK-21848][SQL] Add trait UserDefinedExpression to identify user-defined functions

[gengliangwang]

What changes were proposed in this pull request?

Add trait UserDefinedExpression to identify user-defined functions.
UDF can be expensive. In optimizer we may need to avoid executing UDF multiple times.
E.g.

table.select(UDF as 'a).select('a, ('a + 1) as 'b)

If UDF is expensive in this case, optimizer should not collapse the project to

table.select(UDF as 'a, (UDF+1) as 'b)

Currently UDF classes like PythonUDF, HiveGenericUDF are not defined in catalyst.
This PR is to add a new trait to make it easier to identify user-defined functions.

How was this patch tested?

Unit test

[srowen]

Why a trait? It doesn't seem to be used other than to mark.

[gatorsmile]

/**
 * Common base trait for user-defined functions, including ScalaUDF, ScalaUDAF, PythonUDF, 
 * HiveSimpleUDF, HiveGenericUDF, and HiveUDAFFunction.
 */

[gatorsmile]

Yes. This is a potential optimization we should do, but it should be cost based. Let us improve the resolution logics using this common base trait in this PR first.

[rxin]

maybe UserDefinedExpression?

[rxin]

also sort of weird to put this in ScalaUDF.scala. Either put it in Expression.scala, or create its own file ...

[SparkQA]

Test build #81174 has finished for PR 19064 at commit 5f508ab.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #81177 has finished for PR 19064 at commit de642d9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

I'm +1 on this, it should be useful to pattern match the trait instead of handling ScalaUDF/ScalaUDAF/PythonUDF... in future optimize rules.

[SparkQA]

Test build #81192 has finished for PR 19064 at commit 7b67cd3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

It is better to revise PR title and description to reflect the change from UDFType to UserDefinedExpression.

[SparkQA]

Test build #81202 has finished for PR 19064 at commit be85df3.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

retest this please.

[SparkQA]

Test build #81204 has finished for PR 19064 at commit be85df3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

LGTM

[gatorsmile]

Thanks! Merged to master.