[SPARK-29188][PYTHON] toPandas (without Arrow) gets wrong dtypes when applied on empty DF

[dlindelof]

What changes were proposed in this pull request?

An empty Spark DataFrame converted to a Pandas DataFrame wouldn't have the right column types. Several type mappings were missing.

Why are the changes needed?

Empty Spark DataFrames can be used to write unit tests, and verified by converting them to Pandas first. But this can fail when the column types are wrong.

Does this PR introduce any user-facing change?

Yes; the error reported in the JIRA issue should not happen anymore.

How was this patch tested?

Through unit tests in pyspark.sql.tests.test_dataframe.DataFrameTests#test_to_pandas_from_empty_dataframe

[srowen]

Out of curiosity, what was the previous behavior? was it just missing Long / Double mappings?

[HyukjinKwon]

We should list up all the data types here. Initially it was in order to correct pandas's inferred type.
Now, in case of empty data, pandas always infers it as object and you should rely on this type mapping unlike the intended case before.

See to_arrow_type as an example for complete type mapping. You might need to check what Spark -> Python -> pandas type conversion combinations and whitelist it here.

[HyukjinKwon]

I would also just keep the if-elif logic. It might be more efficient by using a map but it needs to create a map everytime this function is called. More importantly, this code path isn't supposed to be performance sensitive as it's called per a column. So, I would just keep the logic as was.

[HyukjinKwon]

ok to test

[HyukjinKwon]

cc @BryanCutler

[HyukjinKwon]

Also, seems we should handle the case when Arrow optimization is enabled as well (spark.sql.execution.arrow.pyspark.enabled set to true). But I suspect this can be done separately.

[SparkQA]

Test build #114809 has finished for PR 26747 at commit 916e19d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dlindelof]

@HyukjinKwon I've reverted back to an if-else chain instead of a dict. Was there anything else you think I should change?

[dlindelof]

@srowen This illustrates the current behaviour, where an empty Spark Dataframe with a column of type LongType becomes a Pandas Dataframe with a column of type object, i.e. string:

In [62]: foo = spark.sql("SELECT CAST(1 AS LONG) AS bar WHERE 1 = 0")

In [63]: foo
Out[63]: DataFrame[bar: bigint]

In [64]: foo.toPandas().dtypes
Out[64]:
bar    object
dtype: object

When the dataframe is not empty, this is what you see:

In [65]: foo = spark.sql("SELECT CAST(1 AS LONG) AS bar WHERE 1 = 1")

In [66]: foo.toPandas().dtypes
Out[66]:
bar    int64
dtype: object

[SparkQA]

Test build #114838 has finished for PR 26747 at commit f25827c.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[dlindelof]

I'm seeing a failed build, but it doesn't look like it has anything to do with this patch, does it?

[HyukjinKwon]

retest this please

[HyukjinKwon]

@dlindelof How does it work for decimal and other types? You're fixing a fundamental problem (see #26747 (comment))

Can you test other type combinations, and make sure non-empty and empty types are same?

[dlindelof]

Hi,

I've added some more types, I think we have the most important ones now. I've also checked how this behaves in the presence of nulls.

Let me know if you think I'm missing something or if I should have done something differently.

[SparkQA]

Test build #114846 has finished for PR 26747 at commit f25827c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #114906 has finished for PR 26747 at commit 150ecf7.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #114907 has finished for PR 26747 at commit bc95e27.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dlindelof]

Hi,

Just wanted to check what you think of the patch now, is there anything else you think that should be changed? Just let me know and I'll be happy to implement it.

[srowen]

Looks reasonable to me, but @HyukjinKwon and @BryanCutler are the experts -- other thoughts?

[HyukjinKwon]

Merged to master.

@dlindelof, thanks for addressing my comments and welcome to Apache Spark contributors :-).

[HyukjinKwon]

@dlindelof, what's your JIRA id? I need to assign you a Contributor role to assign you to the JIRA https://issues.apache.org/jira/browse/SPARK-29188

[dlindelof]

@HyukjinKwon my JIRA id is dlindelof. Thanks for approving this PR, happy to help.