[SPARK-54555][PYTHON] Enable Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default

[asl3]

What changes were proposed in this pull request?

Enable PySpark Arrow-based optimizations by default in Spark 4.2.0, updating default conf values:

• Set spark.sql.execution.pythonUDF.arrow.enabled and spark.sql.execution.pythonUDTF.arrow.enabled totrue by default to enable Arrow-optimized execution for regular Python UDFs and UDTFs.
• Set spark.sql.execution.arrow.pyspark.enabled to true by default to enable Arrow-based columnar data exchange for PySpark APIs such as DataFrame.toPandas and SparkSession.createDataFrame when the input is a pandas DataFrame or NumPy array.

Update user-facing docs and migration guides to reflect the change.

Why are the changes needed?

Arrow’s columnar IPC significantly improves JVM↔Python throughput and reduces serialization/deserialization overhead, speeding up Python UDFs and DataFrame conversions. Additionally, Arrow provides consistent, well-defined rules for type coercion when Python return values differ from declared UDF return types, reducing ambiguous behavior.

Enabling arrow by default brings performance and correctness improvements to the majority of PySpark users with minimal configuration. Users who depend on the previous (non-Arrow) implementation can opt out by explicitly setting spark.sql.execution.pythonUDF.arrow.enabled, spark.sql.execution.pythonUTF.arrow.enabled, and spark.sql.execution.arrow.pyspark.enabled to false.

Does this PR introduce any user-facing change?

Yes, changes the default configuration of spark.sql.execution.pythonUDF.arrow.enabled, spark.sql.execution.pythonUDTF.arrow.enabled, and spark.sql.execution.arrow.pyspark.enabled to true and updates user-facing docs.

How was this patch tested?

Existing PySpark test suites are run with enabling and disabling the arrow conf.

Was this patch authored or co-authored using generative AI tooling?

No

[dongjoon-hyun]

Sorry, but -1 for Apache Spark 4.1.0, @asl3 .

• https://github.com/apache/spark/releases/tag/v4.1.0-rc2

We are already in RC2 status. It's really too late.

[HyukjinKwon]

Yeah let's do this in master

[dongjoon-hyun]

If you re-target this to 4.2.0, we can merge this and backport this to your company, @asl3 .

[asl3]

Agreed, retargeting to 4.2.0 / master @HyukjinKwon @dongjoon-hyun

[dongjoon-hyun]

Thank you, @asl3 and @HyukjinKwon .

[allisonwang-db]

cc @ueshin @shujingyang-db

[shujingyang-db]

Will be great if we can document the exact type coercion difference introduced by this change

[dbtsai]

LGTM.

[dbtsai]

Will be great if we can document the exact type coercion difference introduced by this change

+1 on the doc. We can create a separate PR for it as followup.

[zhengruifeng]

regarding the failure in pyspark.sql.tests.connect.test_connect_creation.SparkConnectCreationTests.test_with_none_and_nan, you can just skip it for now, I will take a look

[dbtsai]

Merged into master as the error is unrelated to this PR. Thanks.

[zhengruifeng]

@dbtsai @asl3 the failure is related, I mean we can use unittest.skip to skip it for now

[zhengruifeng]

#53296 to restore the CI

[zhengruifeng]

besides test_with_none_and_nan, the doc build is also failed by this PR

https://github.com/apache/spark/actions/runs/19878465653/job/56971154150

@HyukjinKwon @dbtsai @asl3 shall we revert it for now?

[asl3]

@zhengruifeng Thanks! There was a doc fix I had not pushed - I can push a follow-up to add the whitespace: #53298