[SPARK-35211][PYTHON] Proper NumericType conversion for applySchemaToPythonRDD

[da-liii]

What changes were proposed in this pull request?

spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "false")
from pyspark.testing.sqlutils import ExamplePoint
import pandas as pd
pdf = pd.DataFrame({'point': pd.Series([ExamplePoint(1, 1), ExamplePoint(2, 2)])})
df = spark.createDataFrame(pdf, verifySchema=False)
df.show()

Before:

+----------+
|     point|
+----------+
|(0, 0)|
|(0, 0)|
+----------+

After

+----------+
|     point|
+----------+
|(1.0, 1.0)|
|(2.0, 2.0)|
+----------+

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

[HyukjinKwon]

@sadhen, SPARK-35211 is not merged yet, and it makes less sense to call it a followup. Let's make a separate JIRA for this PR.

[HyukjinKwon]

You should also probably update https://github.com/apache/spark/blob/master/python/pyspark/sql/functions.py#L4922-L4940 table.

The problem is that:

• the type coercion isn't really consistent with what we have in SQL
• it has a different type coercion rule with pandas: https://github.com/apache/spark/blob/master/python/pyspark/sql/pandas/functions.py#L305-L323

We should probably define one standard to follow.

[HyukjinKwon]

ok to test

[HyukjinKwon]

I guess this change is okay though ..

[SparkQA]

Test build #137904 has finished for PR 32327 at commit 829dda9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42433/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42433/

[SparkQA]

Test build #137912 has finished for PR 32327 at commit 829dda9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[da-liii]

We should probably define one standard to follow.

I don't know about standard. That's why I only do conversion for numeric type: int/long/byte/short/float/double.
How about fix for numeric type first?

Could you describe the problem in detail?

[HyukjinKwon]

I am okay with this change but you'll have to update https://github.com/apache/spark/blob/master/python/pyspark/sql/functions.py#L4922-L4940 as well by using the codes here: 9b9d81b

[HyukjinKwon]

cc @ueshin @BryanCutler @viirya too FYI

[viirya]

Hmm, first I am confused by why there are many PRs for the same JIRA? Are they for the same issue?

[BryanCutler]

Most of these statements seem to follow a pattern of only converting similar types. What exactly was causing the problem from the example in the description? Was it trying to convert an integer to a double?

[BryanCutler]

It seems odd that we would want to silently convert a double to a short. Are we sure that is the correct behavior?

[da-liii]

Are they for the same issue?

Yes. They are for the same issues but from different part of the code path.

Well, now I've created another JIRA ticket (https://issues.apache.org/jira/browse/SPARK-36283) to track this issue.