[FLINK-3097] [table] Add support for custom functions in Table API

[twalthr]

Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the How To Contribute guide.
In addition to going through the list, please provide a meaningful description of your changes.

• General
  • The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
  • The pull request addresses only one issue
  • Each commit in the PR has a meaningful commit message (including the JIRA id)
• Documentation
  • Documentation has been added for new functionality
  • Old documentation affected by the pull request has been updated
  • JavaDoc for public methods has been added
• Tests & Build
  • Functionality added by the pull request is covered by tests
  • mvn clean verify has been executed successfully locally or a Travis build has passed

This PR introduces user-defined scalar functions for the Table and SQL API.
I will add documentation soon, but this is the general syntax so far:

In Java:

public class HashCode extends ScalarFunction {
    public int eval(String s) {
        return s.hashCode();
    }
}

tableEnv.registerFunction("hashCode", new HashCode());
Table result = table.select("text.hashCode()");
Table result = tableEnv.sql("SELECT hashCode(text) FROM MyTable")

In Scala:

object hashCode extends ScalarFunction {
  def eval(s: String): Int = {
    s.hashCode()
  }
}

val result = table.select(hashCode(text));
val result = tableEnv.sql("SELECT hashCode(text) FROM MyTable")

[wuchong]

Do we have any google docs or FLIP talking about this design ?

I think the ScalarFunction has too many internal functions, and should not be exposed to users. Maybe we can create a new interface for custom functions, such as UDF or UserDefinedFunction.

[twalthr]

No, there is no FLIP about it. I think a discussion in JIRA or in this PR should be enough. That's why I haven't documented it yet, still open for changes. I was inspired by your document. You are right, ScalarFunction has many internal functions but they are not exposed to the user, only 2 methods can be overriden.
An interface is not enough as it might be sometimes necessary to override getReturnType and getParameterType.

[wuchong]

Yes, you are right. I'm just a little concerned about the class name of ScalarFunction, haha..

In addition, Java Table API should be table.select("hashCode(text)"); which is better I think. Assume that the eval function takes two or more parameters, "udf(a,b)" will be satisfied and be consistent with Scala Table API and SQL on syntax.

[aljoscha]

Yes, @wuchong's suggestion for the Java Table API seems more extensible.

[twalthr]

I was also thinking a lot about the names, because we have currently many Functions in Flink. I chose UserDefinedFunction as the top-level function for all user-defined functions such as ScalarFunction, TableFunction, AggregateFunction, or what ever will come in future.

If you have a look into the tests you will see that the Java API supports both: postfix and infix notation. So you can also call functions hashCode(text) if you like.

[aljoscha]

Ah ok, thats perfect! (about infix and postfix)

[wuchong]

Yes, I see. That's great!

[twalthr]

Merging...