Skip to content

[SPARK-54555][PYTHON][TESTS][FOLLOW-UP] Doc build #53298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
+2 −7
Closed

[SPARK-54555][PYTHON][TESTS][FOLLOW-UP] Doc build #53298

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e9c3db5
doc build
asl3 Dec 3, 2025
3485155
space
asl3 Dec 3, 2025
f7a664e
fmt
asl3 Dec 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 2 additions & 7 deletions python/docs/source/tutorial/sql/arrow_pandas.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -379,15 +379,10 @@ and tuples to strings can yield ambiguous results. Arrow Python UDFs, on the oth
capabilities to standardize type coercion and address these issues effectively.

Type coercion differences are introduced by the following changes:
* Since Spark 4.2, Arrow optimization is enabled by default for regular Python UDFs.
The full type coercion difference is summarized in the tables `here <https://github.com/apache/spark/pull/41706>`__.
To disable Arrow optimization, set ``spark.sql.execution.pythonUDF.arrow.enabled`` to false.

* Since Spark 4.1, unnecessary conversion to pandas instances in Arrow-optimized Python UDF is removed in the serializer
when ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled`` is disabled.
* Since Spark 4.2, Arrow optimization is enabled by default for regular Python UDFs. The full type coercion difference is summarized in the tables `here <https://github.com/apache/spark/pull/41706>`__. To disable Arrow optimization, set ``spark.sql.execution.pythonUDF.arrow.enabled`` to false.

The behavior difference is summarized in the tables `here <https://github.com/apache/spark/pull/51225>`__.
To restore the legacy behavior, set ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled`` to true.
* Since Spark 4.1, unnecessary conversion to pandas instances in Arrow-optimized Python UDF is removed in the serializer when ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled`` is disabled. The behavior difference is summarized in the tables `here <https://github.com/apache/spark/pull/51225>`__. To restore the legacy behavior, set ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled`` to true.

Usage Notes
-----------
Expand Down