[SPARK-32552][SQL][DOCS]Complete the documentation for Table-valued Function #29355

huaxingao · 2020-08-04T23:02:08Z

What changes were proposed in this pull request?

There are two types of TVF. We only documented one type. Adding the doc for the 2nd type.

Why are the changes needed?

complete Table-valued Function doc

Does this PR introduce any user-facing change?

How was this patch tested?

Manually build and check

huaxingao · 2020-08-04T23:09:10Z

docs/sql-ref-syntax-qry-select-tvf.md

-
-```sql
-function_name ( expression [ , ... ] ) [ table_alias ]
-```


[ table_alias ] doesn't work with the 2nd type of TVF. Seems it doesn't have much value to have a syntax that only has function_name ( expression [ , ... ] ), so I deleted this section.

huaxingao · 2020-08-04T23:09:35Z

cc @maropu

SparkQA · 2020-08-04T23:15:59Z

Test build #127069 has finished for PR 29355 at commit fda6ecf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-08-06T00:31:06Z

Thanks! I'll check it in hours.

maropu · 2020-08-06T00:33:18Z

Ah, could you file a new JIRA ticket just in case?

huaxingao · 2020-08-06T01:00:20Z

https://github.com/apache/spark/pull/28185/files#r464239662

huaxingao · 2020-08-06T01:11:44Z

Here is the link for HIVE Built-in Table-Generating Functions (UDTF). Should we also include json_tuple and parse_url?

maropu · 2020-08-06T05:23:26Z

docs/sql-ref-syntax-qry-select-tvf.md

+|   1|   2|
+|   3|null|
+----+----+
+


nit: remove the blank.

maropu · 2020-08-06T05:25:21Z

docs/sql-ref-syntax-qry-select-tvf.md

@@ -49,6 +31,13 @@ function_name ( expression [ , ... ] ) [ table_alias ]
 |**range** ( *start, end* )|Long, Long|Creates a table with a single *LongType* column named *id*, <br/> containing rows in a range from *start* to *end* (exclusive) with step value 1.|
 |**range** ( *start, end, step* )|Long, Long, Long|Creates a table with a single *LongType* column named *id*, <br/> containing rows in a range from *start* to *end* (exclusive) with *step* value.|
 |**range** ( *start, end, step, numPartitions* )|Long, Long, Long, Int|Creates a table with a single *LongType* column named *id*, <br/> containing rows in a range from *start* to *end* (exclusive) with *step* value, with partition number *numPartitions* specified.|
+|**explode** ( *expr* )|Expression|Separates the elements of array *expr* into multiple rows, or the elements of map *expr* into multiple rows and columns. Unless specified otherwise, uses the default column name col for elements of the array or key and value for the elements of the map.|


How about making a new table for these new entries?

maropu · 2020-08-06T05:34:06Z

docs/sql-ref-syntax-qry-select-tvf.md

+|   1|   2|
+|   3|null|
+----+----+
+
 ```

 ### Related Statements


How about adding a link to sql-ref-syntax-qry-select-lateral-view.md?

maropu · 2020-08-06T05:36:43Z

docs/sql-ref-syntax-qry-select-tvf.md

@@ -98,6 +87,39 @@ SELECT * FROM range(5, 8) AS test;
 |  6|
 |  7|
 +---+
+
+SELECT explode(array(10, 20));


How about adding examples with LATERAL VIEW?

maropu · 2020-08-06T05:39:38Z

docs/sql-ref-syntax-qry-select-tvf.md

-    Specifies a temporary name with an optional column name list.
-
-    **Syntax:** `[ AS ] table_name [ ( column_name [ , ... ] ) ]`
+A table-valued function (TVF) is a function that returns a relation or a set of rows. There are two types of TVFs in Spark SQL: 1) A TVF that can be specified in a FROM clause, e.g. range; 2) A TVF that can be specified in a SELECT clause, e.g. explode.


nit: A TVF -> a TVF

nit: a SELECT clause -> SELECT/LATERAL VIEW clauses?

maropu · 2020-08-06T05:42:07Z

docs/sql-ref-syntax-qry-select-tvf.md

@@ -49,6 +31,13 @@ function_name ( expression [ , ... ] ) [ table_alias ]
 |**range** ( *start, end* )|Long, Long|Creates a table with a single *LongType* column named *id*, <br/> containing rows in a range from *start* to *end* (exclusive) with step value 1.|
 |**range** ( *start, end, step* )|Long, Long, Long|Creates a table with a single *LongType* column named *id*, <br/> containing rows in a range from *start* to *end* (exclusive) with *step* value.|
 |**range** ( *start, end, step, numPartitions* )|Long, Long, Long, Int|Creates a table with a single *LongType* column named *id*, <br/> containing rows in a range from *start* to *end* (exclusive) with *step* value, with partition number *numPartitions* specified.|
+|**explode** ( *expr* )|Expression|Separates the elements of array *expr* into multiple rows, or the elements of map *expr* into multiple rows and columns. Unless specified otherwise, uses the default column name col for elements of the array or key and value for the elements of the map.|


Expression -> array or map?

maropu · 2020-08-06T05:43:03Z

Here is the link for HIVE Built-in Table-Generating Functions (UDTF). Should we also include json_tuple and parse_url?

yea, I think we should.

…le-valued Function

huaxingao · 2020-08-07T00:35:10Z

@maropu Thanks for your review! I have addressed the comments. Could you please take another look?

SparkQA · 2020-08-07T00:46:14Z

Test build #127161 has finished for PR 29355 at commit 292eb02.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-08-07T00:58:11Z

@maropu Thanks for your review! I have addressed the comments. Could you please take another look?

Thanks for the update, @huaxingao ! Yea, I'l check it later (in hours).

maropu · 2020-08-07T01:32:30Z

docs/sql-ref-syntax-qry-select-tvf.md

-
-    **Syntax:** `[ AS ] table_name [ ( column_name [ , ... ] ) ]`
+A table-valued function (TVF) is a function that returns a relation or a set of rows. There are two types of TVFs in Spark SQL:
+1. a TVF that can be specified in a FROM clause, e.g. range;


How about simply saying 1. A function that can be ...

maropu · 2020-08-07T01:34:33Z

docs/sql-ref-syntax-qry-select-tvf.md

 ```

 ### Related Statements

 * [SELECT](sql-ref-syntax-qry-select.html)
+* [LATERAL VIEW Clause](sql-ref-syntax-qry-select-lateral-view.html)


Could you add a link in the LATERAL VIEW side to this page?
https://github.com/apache/spark/blame/master/docs/sql-ref-syntax-qry-select-lateral-view.md#L114

maropu · 2020-08-07T01:35:24Z

docs/sql-ref-syntax-qry-select-tvf.md

 |Function|Argument Type(s)|Description|
 |--------|----------------|-----------|
 |**range** ( *end* )|Long|Creates a table with a single *LongType* column named *id*, <br/> containing rows in a range from 0 to *end* (exclusive) with step value 1.|
 |**range** ( *start, end* )|Long, Long|Creates a table with a single *LongType* column named *id*, <br/> containing rows in a range from *start* to *end* (exclusive) with step value 1.|
 |**range** ( *start, end, step* )|Long, Long, Long|Creates a table with a single *LongType* column named *id*, <br/> containing rows in a range from *start* to *end* (exclusive) with *step* value.|
 |**range** ( *start, end, step, numPartitions* )|Long, Long, Long, Int|Creates a table with a single *LongType* column named *id*, <br/> containing rows in a range from *start* to *end* (exclusive) with *step* value, with partition number *numPartitions* specified.|

+#### TVFs that can be specified in SELECT/LATERAL VIEW clauses:


Could you add a link to this section in https://github.com/apache/spark/blame/master/docs/sql-ref-syntax-qry-select-lateral-view.md#L40 ?

maropu · 2020-08-07T01:40:15Z

docs/sql-ref-syntax-qry-select-tvf.md

+|--------|----------------|-----------|
+|**explode** ( *expr* )|Array/Map|Separates the elements of array *expr* into multiple rows, or the elements of map *expr* into multiple rows and columns. Unless specified otherwise, uses the default column name col for elements of the array or key and value for the elements of the map.|
+|**explode_outer** <br> ( *expr* )|Array/Map|Separates the elements of array *expr* into multiple rows, or the elements of map *expr* into multiple rows and columns. Unless specified otherwise, uses the default column name col for elements of the array or key and value for the elements of the map.|
+|**inline** ( *expr* )|Expression|Explodes an array of structs into a table. Uses column names col1, col2, etc. by default unless specified otherwise.|


I think Expression is not a type, so we should put a suitable type name here, struct? https://github.com/apache/spark/blame/master/docs/sql-ref-datatypes.md#L59

maropu · 2020-08-07T01:43:12Z

docs/sql-ref-syntax-qry-select-tvf.md

+
+|Function|Argument Type(s)|Description|
+|--------|----------------|-----------|
+|**explode** ( *expr* )|Array/Map|Separates the elements of array *expr* into multiple rows, or the elements of map *expr* into multiple rows and columns. Unless specified otherwise, uses the default column name col for elements of the array or key and value for the elements of the map.|


Btw, did you copy&pasge it from ExpressionDescription? To be honest, its the best to automatically generate this table from it though...

maybe delete this page and add table-valued functions under scalar functions in sql-ref-functions page?

Ah, I see. The suggestion sounds reasonable. Could you try it? cc: @gatorsmile

@maropu
I tried to generate table for tvf, but have two questions:

range is not in expressions package, I can't generate table for it, right? I think I will still keep this page and keep the current range table, but generate table for explode.

json_tuple also belongs to JSONFunctions. Is there a way to put this function in two groups?

Ah, I see. Thanks for the check. I think it is difficult to automatically generate it now. So, its okay as it is.

maropu · 2020-08-24T00:41:29Z

retest this please

…Function # What changes were proposed in this pull request? There are two types of TVF. We only documented one type. Adding the doc for the 2nd type. ### Why are the changes needed? complete Table-valued Function doc ### Does this PR introduce _any_ user-facing change? <img width="1099" alt="Screen Shot 2020-08-06 at 5 30 25 PM" src="https://user-images.githubusercontent.com/13592258/89595926-c5eae680-d80a-11ea-918b-0c3646f9930e.png"> <img width="1100" alt="Screen Shot 2020-08-06 at 5 30 49 PM" src="https://user-images.githubusercontent.com/13592258/89595929-c84d4080-d80a-11ea-9803-30eb502ccd05.png"> <img width="1101" alt="Screen Shot 2020-08-06 at 5 31 19 PM" src="https://user-images.githubusercontent.com/13592258/89595931-ca170400-d80a-11ea-8812-2f009746edac.png"> <img width="1100" alt="Screen Shot 2020-08-06 at 5 31 40 PM" src="https://user-images.githubusercontent.com/13592258/89595934-cb483100-d80a-11ea-9e18-9357aa9f2c5c.png"> ### How was this patch tested? Manually build and check Closes #29355 from huaxingao/tvf. Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org> (cherry picked from commit db74fd0) Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>

maropu · 2020-08-24T00:45:30Z

Thanks, @huaxingao ! Merged to master/branch-3.0. cc: @gatorsmile

SparkQA · 2020-08-24T00:53:26Z

Test build #127822 has finished for PR 29355 at commit 292eb02.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-08-24T03:43:16Z

Thanks a lot! @maropu

probot-autolabeler bot added the DOCS label Aug 4, 2020

huaxingao commented Aug 4, 2020

View reviewed changes

huaxingao changed the title ~~[SPARK-31419][SQL][DOCS][FOLLOW-UP]Complete the documentation for Table-valued Function~~ [SPARK-32552][SQL][DOCS]Complete the documentation for Table-valued Function Aug 6, 2020

maropu reviewed Aug 6, 2020

View reviewed changes

huaxingao added 2 commits August 6, 2020 14:16

[SPARK-31419][SQL][DOCS][FOLLOW-UP]Complete the documentation for Tab…

24615f8

…le-valued Function

address comments

292eb02

huaxingao force-pushed the tvf branch from fda6ecf to 292eb02 Compare August 7, 2020 00:34

maropu reviewed Aug 7, 2020

View reviewed changes

maropu approved these changes Aug 24, 2020

View reviewed changes

maropu closed this in db74fd0 Aug 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32552][SQL][DOCS]Complete the documentation for Table-valued Function #29355

[SPARK-32552][SQL][DOCS]Complete the documentation for Table-valued Function #29355

huaxingao commented Aug 4, 2020 •

edited

huaxingao Aug 4, 2020

huaxingao commented Aug 4, 2020

SparkQA commented Aug 4, 2020

maropu commented Aug 6, 2020

maropu commented Aug 6, 2020

huaxingao commented Aug 6, 2020

huaxingao commented Aug 6, 2020

maropu Aug 6, 2020

maropu Aug 6, 2020

maropu Aug 6, 2020

maropu Aug 6, 2020

maropu Aug 6, 2020

maropu Aug 6, 2020

maropu Aug 6, 2020

maropu commented Aug 6, 2020

huaxingao commented Aug 7, 2020

SparkQA commented Aug 7, 2020

maropu commented Aug 7, 2020

maropu Aug 7, 2020

maropu Aug 7, 2020

maropu Aug 7, 2020

maropu Aug 7, 2020

maropu Aug 7, 2020

huaxingao Aug 7, 2020

maropu Aug 7, 2020

huaxingao Aug 23, 2020

maropu Aug 24, 2020

maropu commented Aug 24, 2020

maropu commented Aug 24, 2020

SparkQA commented Aug 24, 2020

huaxingao commented Aug 24, 2020

[SPARK-32552][SQL][DOCS]Complete the documentation for Table-valued Function #29355

[SPARK-32552][SQL][DOCS]Complete the documentation for Table-valued Function #29355

Conversation

huaxingao commented Aug 4, 2020 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Choose a reason for hiding this comment

huaxingao commented Aug 4, 2020

SparkQA commented Aug 4, 2020

maropu commented Aug 6, 2020

maropu commented Aug 6, 2020

huaxingao commented Aug 6, 2020

huaxingao commented Aug 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maropu commented Aug 6, 2020

huaxingao commented Aug 7, 2020

SparkQA commented Aug 7, 2020

maropu commented Aug 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maropu commented Aug 24, 2020

maropu commented Aug 24, 2020

SparkQA commented Aug 24, 2020

huaxingao commented Aug 24, 2020

huaxingao commented Aug 4, 2020 •

edited