From e9c3db5629162ec3f4e01b83a89b25a223aa3d2d Mon Sep 17 00:00:00 2001 From: Amanda Liu Date: Tue, 2 Dec 2025 17:36:11 -0800 Subject: [PATCH 1/3] doc build --- python/docs/source/tutorial/sql/arrow_pandas.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/python/docs/source/tutorial/sql/arrow_pandas.rst b/python/docs/source/tutorial/sql/arrow_pandas.rst index 386fe83b4821..4bea057fb175 100644 --- a/python/docs/source/tutorial/sql/arrow_pandas.rst +++ b/python/docs/source/tutorial/sql/arrow_pandas.rst @@ -379,6 +379,7 @@ and tuples to strings can yield ambiguous results. Arrow Python UDFs, on the oth capabilities to standardize type coercion and address these issues effectively. Type coercion differences are introduced by the following changes: + * Since Spark 4.2, Arrow optimization is enabled by default for regular Python UDFs. The full type coercion difference is summarized in the tables `here `__. To disable Arrow optimization, set ``spark.sql.execution.pythonUDF.arrow.enabled`` to false. From 34851554906f65237459efc1fa40a0579d08f4a1 Mon Sep 17 00:00:00 2001 From: Amanda Liu Date: Tue, 2 Dec 2025 18:31:00 -0800 Subject: [PATCH 2/3] space --- python/docs/source/tutorial/sql/arrow_pandas.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/python/docs/source/tutorial/sql/arrow_pandas.rst b/python/docs/source/tutorial/sql/arrow_pandas.rst index 4bea057fb175..8913ccad34c9 100644 --- a/python/docs/source/tutorial/sql/arrow_pandas.rst +++ b/python/docs/source/tutorial/sql/arrow_pandas.rst @@ -386,7 +386,6 @@ To disable Arrow optimization, set ``spark.sql.execution.pythonUDF.arrow.enabled * Since Spark 4.1, unnecessary conversion to pandas instances in Arrow-optimized Python UDF is removed in the serializer when ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled`` is disabled. - The behavior difference is summarized in the tables `here `__. To restore the legacy behavior, set ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled`` to true. From f7a664e77f077cb10ef644d540f8e0060126ff11 Mon Sep 17 00:00:00 2001 From: Amanda Liu Date: Tue, 2 Dec 2025 20:16:51 -0800 Subject: [PATCH 3/3] fmt --- python/docs/source/tutorial/sql/arrow_pandas.rst | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/python/docs/source/tutorial/sql/arrow_pandas.rst b/python/docs/source/tutorial/sql/arrow_pandas.rst index 8913ccad34c9..608307266f1f 100644 --- a/python/docs/source/tutorial/sql/arrow_pandas.rst +++ b/python/docs/source/tutorial/sql/arrow_pandas.rst @@ -380,14 +380,9 @@ capabilities to standardize type coercion and address these issues effectively. Type coercion differences are introduced by the following changes: -* Since Spark 4.2, Arrow optimization is enabled by default for regular Python UDFs. -The full type coercion difference is summarized in the tables `here `__. -To disable Arrow optimization, set ``spark.sql.execution.pythonUDF.arrow.enabled`` to false. - -* Since Spark 4.1, unnecessary conversion to pandas instances in Arrow-optimized Python UDF is removed in the serializer -when ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled`` is disabled. -The behavior difference is summarized in the tables `here `__. -To restore the legacy behavior, set ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled`` to true. +* Since Spark 4.2, Arrow optimization is enabled by default for regular Python UDFs. The full type coercion difference is summarized in the tables `here `__. To disable Arrow optimization, set ``spark.sql.execution.pythonUDF.arrow.enabled`` to false. + +* Since Spark 4.1, unnecessary conversion to pandas instances in Arrow-optimized Python UDF is removed in the serializer when ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled`` is disabled. The behavior difference is summarized in the tables `here `__. To restore the legacy behavior, set ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled`` to true. Usage Notes -----------