[SPARK-45232][SQL][DOCS] Add missing function groups to SQL references

[zhengruifeng]

What changes were proposed in this pull request?

Add missing function groups to SQL references:

• xml_funcs
• lambda_funcs
• collection_funcs
• url_funcs
• hash_funcs
• struct_funcs

Note that this PR doesn't fix table_funcs:
1, gen-sql-functions-docs.py doesn't work properly with TableFunctionRegistry, I took a cursory look but fail to fix it;
2, table functions except range (e.g. explode) were already contained in Generator Functions, not sure we need to show them twice.

Why are the changes needed?

when referring to the SQL references, I find many functions are missing https://spark.apache.org/docs/latest/sql-ref-functions.html.

Does this PR introduce any user-facing change?

yes

How was this patch tested?

manually check

Was this patch authored or co-authored using generative AI tooling?

no

[zhengruifeng]

check against

spark/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java

Lines 43 to 48 in 37ab190

	private static final Set<String> validGroups =
	new HashSet<>(Arrays.asList("agg_funcs", "array_funcs", "binary_funcs", "bitwise_funcs",
	"collection_funcs", "predicate_funcs", "conditional_funcs", "conversion_funcs",
	"csv_funcs", "datetime_funcs", "generator_funcs", "hash_funcs", "json_funcs",
	"lambda_funcs", "map_funcs", "math_funcs", "misc_funcs", "string_funcs", "struct_funcs",
	"window_funcs", "xml_funcs", "table_funcs", "url_funcs"));

two difference:
1, table_funcs: not support in gen-sql-functions-docs.py;
2, binary_funcs: I can not find any function using this group;

[allisonwang-db]

QQ: For generator_funcs, do we have documentation for them when used in the FROM clause of a query? Functions like explode are typically considered table-valued generator functions.

[allisonwang-db]

QQ: For generator_funcs, do we have documentation for them when used in the FROM clause of a query? Functions like explode are typically considered table-valued generator functions.

[zhengruifeng]

@allisonwang-db I am not sure, I don't see document for FROM clause, you may check 3 places:

• https://spark.apache.org/docs/latest/api/sql/index.html#explode
• https://spark.apache.org/docs/latest/sql-ref-functions-builtin.html#generator-functions
• https://spark.apache.org/docs/latest/sql-ref-syntax.html

[gatorsmile]

cc @srielau

[zhengruifeng]

[zhengruifeng]

we can check the documents built in the GA of this PR, https://github.com/zhengruifeng/spark/actions/runs/6249096629

however, it expires after one day

[srielau]

What is the purpose of "lambda function"? All others are type-specific or "functionality"-specific.
But lambda is "technology". What is the user journey that would drive one to browse lambda functions?

[zhengruifeng]

What is the purpose of "lambda function"? All others are type-specific or "functionality"-specific. But lambda is "technology".

lambda functions were already exposed to end users (e.g. https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.array_sort.html#pyspark.sql.functions.array_sort).

I think if we document other functions here, it is better to add lambda functions as well.

What is the user journey that would drive one to browse lambda functions?

I think this could be an example: when a user try to sort array of structs by a specific order, he may refer to the document of array_sort

[HyukjinKwon]

Just to be clear, this is automatic documentation based on the current documentation. If the grouping is wrong, or to be fixed, we should fix functions.scala main documentation.

[zhengruifeng]

@HyukjinKwon this page is not built from functions.scala, but from the groups specified in expression definitions, like

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala

Line 1313 in 6b747ab

group = "bitwise_funcs")

[HyukjinKwon]

yeah, I mean individual ExpressionDescription.

[srielau]

What is the purpose of "lambda function"? All others are type-specific or "functionality"-specific. But lambda is "technology".

lambda functions were already exposed to end users (e.g. https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.array_sort.html#pyspark.sql.functions.array_sort).

I think if we document other functions here, it is better to add lambda functions as well.

What is the user journey that would drive one to browse lambda functions?

I think this could be an example: when a user try to sort array of structs by a specific order, he may refer to the document of array_sort

If I try to find a function that sorts arrays I will try to find that function under collection functions.
Just like substr() is not a math function, even though most of its arguments are integers. Substr operates on strings...
array_sort operates on arrays.

[zhengruifeng]

What is the purpose of "lambda function"? All others are type-specific or "functionality"-specific. But lambda is "technology".

lambda functions were already exposed to end users (e.g. https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.array_sort.html#pyspark.sql.functions.array_sort).
I think if we document other functions here, it is better to add lambda functions as well.

What is the user journey that would drive one to browse lambda functions?

I think this could be an example: when a user try to sort array of structs by a specific order, he may refer to the document of array_sort

If I try to find a function that sorts arrays I will try to find that function under collection functions. Just like substr() is not a math function, even though most of its arguments are integers. Substr operates on strings... array_sort operates on arrays.

Got it.
I think we can rename the group name in .md files, but need to be different from others. What about Advanced Collection Functions? @srielau

[srielau]

What is the purpose of "lambda function"? All others are type-specific or "functionality"-specific. But lambda is "technology".

lambda functions were already exposed to end users (e.g. https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.array_sort.html#pyspark.sql.functions.array_sort).
I think if we document other functions here, it is better to add lambda functions as well.

What is the user journey that would drive one to browse lambda functions?

I think this could be an example: when a user try to sort array of structs by a specific order, he may refer to the document of array_sort

If I try to find a function that sorts arrays I will try to find that function under collection functions. Just like substr() is not a math function, even though most of its arguments are integers. Substr operates on strings... array_sort operates on arrays.

Got it. I think we can rename the group name in .md files, but need to be different from others. What about Advanced Collection Functions? @srielau

How about having our cake and eat it to? Can a function be in more than one group?

[zhengruifeng]

What is the purpose of "lambda function"? All others are type-specific or "functionality"-specific. But lambda is "technology".

lambda functions were already exposed to end users (e.g. https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.array_sort.html#pyspark.sql.functions.array_sort).
I think if we document other functions here, it is better to add lambda functions as well.

What is the user journey that would drive one to browse lambda functions?

I think this could be an example: when a user try to sort array of structs by a specific order, he may refer to the document of array_sort

If I try to find a function that sorts arrays I will try to find that function under collection functions. Just like substr() is not a math function, even though most of its arguments are integers. Substr operates on strings... array_sort operates on arrays.

Got it. I think we can rename the group name in .md files, but need to be different from others. What about Advanced Collection Functions? @srielau

How about having our cake and eat it to? Can a function be in more than one group?

probably we can. I will try to map lambda to collection just in the doc build.

I think making a function in more than one group would be much complex.

[zhengruifeng]

now we put lambda functions into collection functions

[zhengruifeng]

@srielau I have put lambda functions in collection functions, I think this PR is ready to merge?

[zhengruifeng]

thanks @srielau @allisonwang-db @HyukjinKwon

merged to master