-
Notifications
You must be signed in to change notification settings - Fork 28.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-43473][PYTHON] Support struct type in createDataFrame from pan…
…das DataFrame ### What changes were proposed in this pull request? Supports struct type in `createDataFrame` from pandas DataFrame. With Arrow optimization, it works without the fallback: ```py >>> import pandas as pd >>> from pyspark.sql.types import Row >>> >>> pdf = pd.DataFrame( ... {"a": [Row(1, "a"), Row(2, "b")], "b": [{"s": 3, "t": "x"}, {"s": 4, "t": "y"}]} ... ) >>> schema = "a struct<x int, y string>, b struct<s int, t string>" >>> >>> df = spark.createDataFrame(pdf, schema) >>> df.show() +------+------+ | a| b| +------+------+ |{1, a}|{3, x}| |{2, b}|{4, y}| +------+------+ ``` and Spark Connect also works. ### Why are the changes needed? In vanilla PySpark without Arrow optimization, `Row` object or `dict` can be handled as struct type if the schema is provided: ```py >>> import pandas as pd >>> from pyspark.sql.types import * >>> >>> pdf = pd.DataFrame( ... {"a": [Row(1, "a"), Row(2, "b")], "b": [{"s": 3, "t": "x"}, {"s": 4, "t": "y"}]} ... ) >>> schema = "a struct<x int, y string>, b struct<s int, t string>" >>> >>> df = spark.createDataFrame(pdf, schema) >>> df.show() +------+------+ | a| b| +------+------+ |{1, a}|{3, x}| |{2, b}|{4, y}| +------+------+ ``` Whereas with Arrow, it uses a fallback to make it: ```py >>> spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', True) >>> spark.createDataFrame(pdf, schema).show() /.../pyspark/sql/pandas/conversion.py:329: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true; however, failed by the reason below: A field of type StructType expects a pandas.DataFrame, but got: <class 'pandas.core.series.Series'> Attempting non-optimization as 'spark.sql.execution.arrow.pyspark.fallback.enabled' is set to true. warn(msg) +------+------+ | a| b| +------+------+ |{1, a}|{3, x}| |{2, b}|{4, y}| +------+------+ ``` and Spark Connect fails: ```py >>> df = spark.createDataFrame(pdf, schema) Traceback (most recent call last): ... ValueError: A field of type StructType expects a pandas.DataFrame, but got: <class 'pandas.core.series.Series'> ``` ### Does this PR introduce _any_ user-facing change? `Row` object or `dict` in pandas DataFrame works as struct type when `createDataFrame` if the schema is provided. ### How was this patch tested? Added the related test. Closes #41149 from ueshin/issues/SPARK-43473/rows. Authored-by: Takuya UESHIN <ueshin@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
- Loading branch information
1 parent
d53ddbe
commit 6221995
Showing
4 changed files
with
154 additions
and
82 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters