[SPARK-44918][SQL][PYTHON] Support named arguments in scalar Python/Pandas UDFs

[ueshin]

What changes were proposed in this pull request?

Supports named arguments in scalar Python/Pandas UDF.

For example:

>>> @udf("int")
... def test_udf(a, b):
...     return a + 10 * b
...
>>> spark.udf.register("test_udf", test_udf)

>>> spark.range(2).select(test_udf(b=col("id") * 10, a=col("id"))).show()
+---------------------------------+
|test_udf(b => (id * 10), a => id)|
+---------------------------------+
|                                0|
|                              101|
+---------------------------------+

>>> spark.sql("SELECT test_udf(b => id * 10, a => id) FROM range(2)").show()
+---------------------------------+
|test_udf(b => (id * 10), a => id)|
+---------------------------------+
|                                0|
|                              101|
+---------------------------------+

or:

>>> @pandas_udf("int")
... def test_udf(a, b):
...     return a + 10 * b
...
>>> spark.udf.register("test_udf", test_udf)

>>> spark.range(2).select(test_udf(b=col("id") * 10, a=col("id"))).show()
+---------------------------------+
|test_udf(b => (id * 10), a => id)|
+---------------------------------+
|                                0|
|                              101|
+---------------------------------+

>>> spark.sql("SELECT test_udf(b => id * 10, a => id) FROM range(2)").show()
+---------------------------------+
|test_udf(b => (id * 10), a => id)|
+---------------------------------+
|                                0|
|                              101|
+---------------------------------+

Why are the changes needed?

Now that named arguments support was added (#41796, #42020).

Scalar Python/Pandas UDFs can support it.

Does this PR introduce any user-facing change?

Yes, named arguments will be available for scalar Python/Pandas UDFs.

How was this patch tested?

Added related tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[ueshin]

cc @dtenedor @HyukjinKwon @xinrong-meng

[dtenedor]

Generally LGTM. I left one suggestion for a testing idea. Thanks a lot for adding this feature!!

[dtenedor]

Suggested change

	* - don't contain positional arguments
	* - don't contain positional arguments after named arguments
	* - all map to valid argument names from the function declaration

[ueshin]

The third item is not right for Python UDF/UDTF. Currently it relies on Python.

[allisonwang-db]

Good catch!

[ueshin]

Thanks! merging to master.