[SPARK-52497][DOCS] Add docs for SQL user-defined functions #51281
Conversation
|
|
||
| When `TEMPORARY` is specified, the function is only available for the current session. Otherwise, it is persisted in the catalog and available across sessions. The `OR REPLACE` option allows updating an existing function definition, while `IF NOT EXISTS` prevents errors when creating a function that already exists. | ||
|
|
||
| The function parameters must be specified with their data types. The return type can be either a scalar data type or a table with an optional schema definition. |
There was a problem hiding this comment.
oh, the return table schema definition is optional?
There was a problem hiding this comment.
actually, the entire RETURN clause is optional, right?
There was a problem hiding this comment.
The RETURNS is optional, RETURN is not
There was a problem hiding this comment.
Yes the RETURNS clause is optional for scalar UDFs, and the RETURNS TABLE schema is optional for TVFs.
There was a problem hiding this comment.
So it should be The return type can be either a scalar data type or a table with an schema definition. If not specified the return type will be inferred from the function body?
There was a problem hiding this comment.
Yes it is mentioned below (in the RETURNS clause section). Here it simply means the function return type can be a scalar or a table value.
- **RETURNS [data_type](sql-ref-datatypes.md)**
The return data type of the scalar function. This clause is optional. The data type will be derived from the SQL function body if it is not provided.
|
|
||
| ### Syntax | ||
|
|
||
| ```sql |
There was a problem hiding this comment.
I know in Spark we use sql, but it makes no sense to do that for the syntax.. I wish there were a BNF ....
Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>
|
LGTM! |
|
The CI issue is unrelated as it's a doc only PR, thanks, merging to master/4.0! |
### What changes were proposed in this pull request? This PR adds docs for SQL UDFs. ### Why are the changes needed? Add documentation for a new Spark 4 feature. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually verify the documentation build ### Was this patch authored or co-authored using generative AI tooling? No Closes #51281 from allisonwang-db/spark-52497-sql-udf-docs. Lead-authored-by: Allison Wang <allison.wang@databricks.com> Co-authored-by: Allison Wang <allisonwang@apache.org> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
What changes were proposed in this pull request?
This PR adds docs for SQL UDFs.
Why are the changes needed?
Add documentation for a new Spark 4 feature.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Manually verify the documentation build
Was this patch authored or co-authored using generative AI tooling?
No