[SPARK-26979][PYTHON][FOLLOW-UP] Make binary math/string functions take string as columns as well

[HyukjinKwon]

What changes were proposed in this pull request?

This is a followup of #23882 to handle binary math/string functions. For instance, see the cases below:

Before:

>>> from pyspark.sql.functions import lit, ascii
>>> spark.range(1).select(lit('a').alias("value")).select(ascii("value"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../spark/python/pyspark/sql/functions.py", line 51, in _
    jc = getattr(sc._jvm.functions, name)(col._jc if isinstance(col, Column) else col)
  File "/.../spark/python/lib/py4j-0.10.8.1-src.zip/py4j/java_gateway.py", line 1286, in __call__
  File "/.../spark/python/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/.../spark/python/lib/py4j-0.10.8.1-src.zip/py4j/protocol.py", line 332, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling z:org.apache.spark.sql.functions.ascii. Trace:
py4j.Py4JException: Method ascii([class java.lang.String]) does not exist
	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:339)
	at py4j.Gateway.invoke(Gateway.java:276)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)

>>> from pyspark.sql.functions import atan2
>>> spark.range(1).select(atan2("id", "id"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../spark/python/pyspark/sql/functions.py", line 78, in _
    jc = getattr(sc._jvm.functions, name)(col1._jc if isinstance(col1, Column) else float(col1),
ValueError: could not convert string to float: id

After:

>>> from pyspark.sql.functions import lit, ascii
>>> spark.range(1).select(lit('a').alias("value")).select(ascii("value"))
DataFrame[ascii(value): int]

>>> from pyspark.sql.functions import atan2
>>> spark.range(1).select(atan2("id", "id"))
DataFrame[ATAN2(id, id): double]

Note that,

• This PR causes a slight behaviour changes for math functions. For instance, numbers as strings (e.g., "1") were supported as arguments of binary math functions before. After this PR, it recognises it as column names.

• I also intentionally didn't document this behaviour changes since we're going ahead for Spark 3.0 and I don't think numbers as strings make much sense in math functions.

• There is another exception when, which takes string as literal values as below. This PR doeesn't fix this ambiguity.

  >>> spark.range(1).select(when(lit(True), col("id"))).show()

  +--------------------------+
  |CASE WHEN true THEN id END|
  +--------------------------+
  |                         0|
  +--------------------------+
  

  >>> spark.range(1).select(when(lit(True), "id")).show()

  +--------------------------+
  |CASE WHEN true THEN id END|
  +--------------------------+
  |                        id|
  +--------------------------+
  

This PR also fixes as below:

#23882 fixed it to:

• Rename _create_function to _create_name_function
• Define new _create_function to take strings as column names.

This PR, I proposes to:

• Revert _create_name_function name to _create_function.
• Define new _create_function_over_column to take strings as column names.

How was this patch tested?

Some unit tests were added for binary math / string functions.

[HyukjinKwon]

@asmello, @rxin, @srowen, @cloud-fan, this is the followup of #23882 to address my own comments.

If the discussion is going to be longer in this PR, I would like to revert #23882

[SparkQA]

Test build #103599 has finished for PR 24121 at commit a0574d6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #103604 has finished for PR 24121 at commit 9f65807.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[asmello]

I'm definitely in favour of the change in how math functions work. Somehow I missed that cast to float, so I thought they just passed the argument directly when given a string (which would've worked, since the Scala API has string overloads for all math functions). So this is something I should've included in my PR.

I'll add some inline comments for stuff I disagree with.

[SparkQA]

Test build #103612 has finished for PR 24121 at commit 7f78cc4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

I'll merge this one assuming we want this way if there are no more comments this week.

adding @holdenk, @felixcheung, @BryanCutler, @ueshin, @viirya for visibility.

[SparkQA]

Test build #103670 has finished for PR 24121 at commit ccbe51e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Just add a quick release-notes tag and Docs text to the JIRA for the one potential behavior change. This looks good.

[asmello]

Looks good!

[HyukjinKwon]

Merged to master.