[SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference #28087

huaxingao · 2020-04-01T01:43:35Z

What changes were proposed in this pull request?

Document UDF in SQL Reference

Why are the changes needed?

To make SQL Reference complete.

Does this PR introduce any user-facing change?

Yes. Here are the new pages:

How was this patch tested?

Manually build and check

huaxingao · 2020-04-01T01:45:10Z

cc @gatorsmile @maropu

SparkQA · 2020-04-01T01:58:40Z

Test build #120660 has finished for PR 28087 at commit df36893.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

docs/sql-ref-functions-udf-aggregate.md

maropu · 2020-04-01T02:23:39Z

docs/sql-ref-functions-udf-aggregate.md

@@ -1,22 +1,65 @@
 ---
 layout: global
-title: User defined Aggregate Functions (UDAF)
-displayTitle: User defined Aggregate Functions (UDAF)
+title: User Defined Aggregate Functions (UDAF)


nit: UDAF -> UDAFs?

docs/sql-ref-functions-udf-aggregate.md

maropu · 2020-04-01T02:33:24Z

docs/sql-ref-functions-udf-aggregate.md

+
+// Define, register a UDAF to calculate the sum of product of two columns
+// Scala
+import functions.udf


import functions.udf -> import org.apache.spark.sql.functions.udf?

Also, we need to import import org.apache.spark.sql.expressions.Aggregator and import org.apache.spark.sql.Encoder?

maropu · 2020-04-01T02:37:41Z

docs/sql-ref-functions-udf-aggregate.md

+SELECT agg(a, b) FROM testUDAF;
+
+---------------------------------------------+
+|$anon$2(CAST(a AS BIGINT), CAST(b AS BIGINT))|


btw, (this is not related to this pr though), the "agg" name defined in spark.udf.register("agg", agg) is not used in the name of the plan...

docs/sql-ref-functions-udf-scalar.md

maropu · 2020-04-01T02:41:32Z

docs/sql-ref-functions-udf-scalar.md

+
+{% highlight sql %}
+
+// Define, register a zero argument non-deterministic UDF


Define, register -> Define and register

docs/sql-ref-functions-udf-scalar.md

maropu · 2020-04-01T02:44:04Z

docs/sql-ref-functions-udf-scalar.md

+
+// Define, register a zero argument non-deterministic UDF
+// Scala
+import functions.udf


import functions.udf -> import org.apache.spark.sql.functions.udf?

maropu · 2020-04-01T02:46:27Z

docs/sql-ref-functions-udf-scalar.md

+spark.udf.register("groupFunction", (n: Int) => { n > 10 })
+
+val df = Seq(("red", 1), ("red", 2), ("blue", 10),
+    ("green", 100), ("green", 200)).toDF("color", "value")


It is better to match either format: a single value per line or multiple values per line https://github.com/apache/spark/pull/28087/files#diff-97b11639e031df253c371da5298980d9R44

maropu · 2020-04-01T02:51:06Z

docs/sql-ref-functions-udf.md

-Spark. This document describes the SQL constructs supported by Spark in detail
-along with usage examples when applicable.
+User-Defined Functions (UDFs) are a feature of Spark SQL that allows users to define their own functions when the system's built-in functions are not enough to perform the desired task. To use UDFs in SPARK SQL, users must first define the function, then register the function with SPARK, and finally call the registered function. The User-Defined Functions can act on a single row or act on multiple rows at once.
+


We don't need to document Hive UDFs here?

I will think about this.

gatorsmile · 2020-04-01T03:01:54Z

Do we have a page for introducing the APIs for Scala, Java, Python UDFs/UDAFs/UDTF? If not, could we introduce it?

huaxingao · 2020-04-01T05:31:28Z

Do we have a page for introducing the APIs for Scala, Java, Python UDFs/UDAFs/UDTF? If not, could we introduce it?

I will think about where and how to document these.

Thanks for your review! @maropu @gatorsmile

SparkQA · 2020-04-01T05:50:47Z

Test build #120664 has finished for PR 28087 at commit df84414.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2020-04-01T15:40:25Z

cc @Ngone51 Please help review this?

huaxingao · 2020-04-01T23:47:40Z

I added Scala APIs. Java APIs and Python APIs are similar so I prefer not to add them here. I think the Scala APIs can already give the users a very good idea about what these UDF APIs are. If they need to check the Java or Python APIs, they can refer to the APIs doc. I don't want the majority of the page become API docs.

huaxingao · 2020-04-01T23:49:47Z

docs/sql-ref-functions-udf-aggregate.md


 ### Examples

 {% highlight sql %}

 // Define and register a UDAF to calculate the sum of product of two columns
 // Scala
-import org.apache.spark.sql.Encoder


I am confused with this import. My test works OK without it.

SparkQA · 2020-04-02T00:04:42Z

Test build #120700 has finished for PR 28087 at commit 62d2655.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-02T05:00:40Z

docs/sql-ref-functions-udf-aggregate.md

+df.createOrReplaceTempView("testUDAF")
+
+-- SQL
+SELECT agg(a, b) FROM testUDAF;


If this is a Scala case, sql("SELECT agg(a, b) FROM testUDAF").show() is better?

I will change all examples to this format

maropu · 2020-04-02T05:03:54Z

docs/sql-ref-functions-udf-aggregate.md

+<dl>
+  <dt><code><em>reduce(b: BUF, a: IN): BUF</em></code></dt>
+  <dd>
+    Combine two values to produce a new value. For performance, the function may modify b and return it instead of constructing new object for b.


a new value -> a new aggregated value?

better to wrap b with `, e.g., `b`?

changed to b because ` doesn't work inside html.

maropu · 2020-04-02T05:12:09Z

docs/sql-ref-functions-udf-aggregate.md

+
+User-Defined Aggregate Functions (UDAFs) are user-programmable routines that act on multiple rows at once and return a single aggregated value as a result. This documentation lists the classes that are required for creating and registering UDAFs. It also contains examples that demonstrate how to define and register UDAFs in Scala and invoke them in Spark SQL.
+
+### org.apache.spark.sql.expressions.Aggregator[-IN, BUF, OUT]


How about "Interface of Aggregator[-IN, BUF, OUT]"? We need the full-qualifier here?

I guess I will just use Aggregator[-IN, BUF, OUT]. Because UserDefinedFunctionis an abstract class , and UDFRegistration is a regular class

maropu · 2020-04-02T05:12:36Z

docs/sql-ref-functions-udf-aggregate.md

+  </dd>
+</dl>
+
+### org.apache.spark.sql.UDFRegistration


ditto: how about "Inerface of UDFRegistration"?

changed to UDFRegistration

maropu · 2020-04-02T05:18:56Z

docs/sql-ref-functions-udf-scalar.md

+|     6|
+------+
+
+// Define a two arguments UDF and register it with Spark in one step


Define a two arguments -> Define two arguments?

I guess I will change to a two-argument UDF?

docs/sql-ref-functions-udf-aggregate.md

viirya · 2020-04-02T06:03:31Z

docs/sql-ref-functions-udf-aggregate.md

+<dl>
+  <dt><code><em>zero: BUF</em></code></dt>
+  <dd>
+    A zero value for this aggregation.


The initial value of the intermediate result for this aggregation.

docs/sql-ref-functions-udf-aggregate.md

viirya · 2020-04-02T06:11:11Z

docs/sql-ref-functions-udf-scalar.md

+|300       |
+----------+
+
+# Define and register a UDF using Python


Do we need this section? I think there is doc for Python UDFs. No?

Just want to give an example using Python. There is API doc for Python UDFs, but I won't have a separate doc for Python. I am OK to keep or delete it.

SparkQA · 2020-04-02T06:14:28Z

Test build #120710 has finished for PR 28087 at commit 3bc5074.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-03T07:52:01Z

docs/sql-ref-functions-udf-scalar.md

+</dl>
+
+<dl>
+  <dt><code><em>nullable: Boolean</em></code></dt>


we need deterministic and nullable here? I personally think asNonNullable, asNondeterministic, and withName is enough for users.

maropu · 2020-04-03T07:57:47Z

docs/sql-ref-functions-udf-scalar.md

+
+### UserDefinedFunction
+
+A user-defined function. To create one, use the `udf` functions in `functions`.


How about saying here To define the properties of a user-defined function, you can use the some methods defined in 'UserDefinedFunction';?

SparkQA · 2020-04-03T19:18:01Z

Test build #120789 has finished for PR 28087 at commit 6e254aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-03T21:01:29Z

Test build #120790 has finished for PR 28087 at commit e0b77ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-05T23:52:36Z

docs/sql-ref-functions-udf-aggregate.md

+
+### UDFRegistration
+
+Functions for registering user-defined functions. Use `SparkSession.udf` to access this.


Users need to know this class? I personally think not, so how about removing this section?

user needs to call UDFRegistration.register to register the udaf
spark.udf.register("agg", agg)
I guess we keep this section?

I still think its ok for users to just know how to register udf/udaf in the document, so user don't need to know what UDFRegistration is.

I will remove

maropu · 2020-04-06T00:00:34Z

docs/sql-ref-functions-udf-aggregate.md

+### Aggregator[-IN, BUF, OUT]
+
+A base class for user-defined aggregations, which can be used in Dataset operations to take all of the elements of a group and reduce them to a single value.
+- IN The input type for the aggregation.


nit: need a single blank?

A base class for user-defined aggregations, which can be used in Dataset operations to take all of the elements of a group and reduce them to a single value. - IN The input type for the aggregation.

nit: also, how about putting a separator between IN and the statement? e.g., IN - The input...

btw, the contents are somewhat overlapped with https://spark.apache.org/docs/3.0.0-preview/sql-getting-started.html#aggregations ? Probably, we need to organize these aggregator contents.

I will think about this.

I personally think its better to move all the contents of sql-getting-started.html into the SQL doc. Since the user-defined aggregator topic is advanced one, the SQL doc is more suitable than the Getting Started doc. cc: @gatorsmile

@maropu sounds good to me.

SparkQA · 2020-04-06T00:58:28Z

Test build #120851 has finished for PR 28087 at commit 24d4975.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Ngone51 · 2020-04-07T08:38:42Z

docs/sql-ref-functions-udf-aggregate.md

+import org.apache.spark.sql.expressions.Aggregator
+import org.apache.spark.sql.functions.udaf
+
+val agg = udaf(new Aggregator[(Long, Long), Long, Long] {


Seems we already have an example at https://spark.apache.org/docs/latest/sql-getting-started.html#aggregations.

And do we still need to keep that Aggregations section or not? If not, I think we could reuse the previous examples here.

Ngone51 · 2020-04-07T08:48:30Z

docs/sql-ref-functions-udf-aggregate.md

+{% highlight sql %}
+
+// Define and register a UDAF to calculate the sum of product of two columns
+// Scala


Can we highlight Scala codes as other doc does? e.g,

<div data-lang="scala" markdown="1"> {% include_example generic_load_save_functions scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala %} </div>

SparkQA · 2020-04-10T00:23:51Z

Test build #121047 has finished for PR 28087 at commit adee919.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaUserDefinedScalar

maropu · 2020-04-10T07:57:32Z

examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedScalar.scala

+
+    spark.stop()
+  }
+


nit: remove the blank line.

maropu · 2020-04-10T07:58:01Z

docs/sql-ref-functions-udf-scalar.md

+</div>
+</div>
+
+


nit: remove the single blank line.

maropu · 2020-04-10T07:58:11Z

docs/sql-ref-functions-udf-scalar.md

+
+User-Defined Functions (UDFs) are user-programmable routines that act on one row. This documentation lists the classes that are required for creating and registering UDFs. It also contains examples that demonstrate how to define and register UDFs and invoke them in Spark SQL.
+
+


nit: remove the blank line.

maropu · 2020-04-10T07:58:25Z

docs/sql-ref-functions-udf-aggregate.md

+</div>
+
+### Related Statements
+- [Scalar User Defined Functions (UDFs)](sql-ref-functions-udf-scalar.html)


nit: - -> *

maropu

LGTM except for the existing comments.

SparkQA · 2020-04-11T00:20:08Z

Test build #121108 has finished for PR 28087 at commit 5b7c1c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-11T00:34:50Z

Test build #121109 has finished for PR 28087 at commit 8736f5c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2020-04-13T04:38:37Z

Merged to master/3.0

huaxingao · 2020-04-13T04:44:31Z

Thank you everyone!

HyukjinKwon · 2020-04-13T09:22:26Z

examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedScalar.java

+      (String s, Integer x) -> s.length() + x, DataTypes.IntegerType
+    );
+    spark.udf().register("strLen", strLen);
+    spark.sql("SELECT strLen('test', 1)").show();


It should have been better probably if we added SQL example with UDAF too, e.g.)

val agg = Aggregator(...) spark.udf.register("udaf_func", agg)

which was added in SPARK-27296.

The current example is only Scala/Java APIs there in UserDefinedTypedAggregation.scala and JavaUserDefinedTypedAggregation.java.

+1; @huaxingao Could you add it in follow-up? ~~The initali commit in this PR includes the example (b276e16#diff-97b11639e031df253c371da5298980d9R32-R54), but it seems to disappear during the reviews....~~

I just submitted a follow-up to add the SQL example.

maropu reviewed Apr 1, 2020

View reviewed changes

docs/sql-ref-functions-udf-scalar.md Outdated Show resolved Hide resolved

maropu reviewed Apr 1, 2020

View reviewed changes

docs/sql-ref-functions-udf-scalar.md Outdated Show resolved Hide resolved

maropu reviewed Apr 1, 2020

View reviewed changes

docs/sql-ref-functions-udf-scalar.md Outdated Show resolved Hide resolved

maropu reviewed Apr 1, 2020

View reviewed changes

maropu changed the title ~~[SPARK-31319][SQL][DOCS] Document UDF in SQL Reference~~ [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference Apr 1, 2020

maropu reviewed Apr 1, 2020

View reviewed changes

huaxingao commented Apr 1, 2020

View reviewed changes

maropu reviewed Apr 2, 2020

View reviewed changes

viirya reviewed Apr 2, 2020

View reviewed changes

docs/sql-ref-functions-udf-aggregate.md Outdated Show resolved Hide resolved

viirya reviewed Apr 2, 2020

View reviewed changes

docs/sql-ref-functions-udf-aggregate.md Outdated Show resolved Hide resolved

viirya reviewed Apr 2, 2020

View reviewed changes

docs/sql-ref-functions-udf-aggregate.md Outdated Show resolved Hide resolved

viirya reviewed Apr 2, 2020

View reviewed changes

maropu reviewed Apr 3, 2020

View reviewed changes

address comments

6e254aa

add links to cross reference related statements

e0b77ad

maropu reviewed Apr 5, 2020

View reviewed changes

maropu reviewed Apr 6, 2020

View reviewed changes

address comments

24d4975

Ngone51 reviewed Apr 7, 2020

View reviewed changes

move the UDAF examples from getting started to SQL Reference

adee919

maropu reviewed Apr 10, 2020

View reviewed changes

docs/sql-ref-functions-udf-scalar.md Outdated

</div>

</div>

Copy link

Member

maropu Apr 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove the single blank line.

huaxingao reacted with thumbs up emoji

maropu reviewed Apr 10, 2020

View reviewed changes

maropu approved these changes Apr 10, 2020

View reviewed changes

huaxingao added 2 commits April 10, 2020 17:03

address comments

5b7c1c0

remove UDFRegistration section

8736f5c

srowen closed this in 3bbd80d Apr 13, 2020

huaxingao deleted the udf branch April 13, 2020 04:44

HyukjinKwon reviewed Apr 13, 2020

View reviewed changes

maropu mentioned this pull request Apr 14, 2020

[SPARK-31319][SQL][FOLLOW-UP] Add a SQL example for UDAF #28209

Closed


		{% highlight sql %}

		// Define, register a zero argument non-deterministic UDF


		User-Defined Aggregate Functions (UDAFs) are user-programmable routines that act on multiple rows at once and return a single aggregated value as a result. This documentation lists the classes that are required for creating and registering UDAFs. It also contains examples that demonstrate how to define and register UDAFs in Scala and invoke them in Spark SQL.

		### org.apache.spark.sql.expressions.Aggregator[-IN, BUF, OUT]


		### UserDefinedFunction

		A user-defined function. To create one, use the `udf` functions in `functions`.


		### UDFRegistration

		Functions for registering user-defined functions. Use `SparkSession.udf` to access this.


		User-Defined Functions (UDFs) are user-programmable routines that act on one row. This documentation lists the classes that are required for creating and registering UDFs. It also contains examples that demonstrate how to define and register UDFs and invoke them in Spark SQL.

[SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference #28087

[SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference #28087

Conversation

huaxingao commented Apr 1, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

huaxingao commented Apr 1, 2020

SparkQA commented Apr 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gatorsmile commented Apr 1, 2020

huaxingao commented Apr 1, 2020

SparkQA commented Apr 1, 2020

gatorsmile commented Apr 1, 2020

huaxingao commented Apr 1, 2020

Choose a reason for hiding this comment

SparkQA commented Apr 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 3, 2020

SparkQA commented Apr 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 10, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maropu left a comment

Choose a reason for hiding this comment

SparkQA commented Apr 11, 2020

SparkQA commented Apr 11, 2020

srowen commented Apr 13, 2020

huaxingao commented Apr 13, 2020

Choose a reason for hiding this comment

maropu Apr 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huaxingao commented Apr 1, 2020 •

edited

Loading

maropu Apr 13, 2020 •

edited

Loading