[SPARK-58016][PYTHON] Revert Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default

[Yicong-Huang]

What changes were proposed in this pull request?

This PR reverts SPARK-54555 (commit ea0a35e, "Enable Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default"), restoring the following config defaults to false:

• spark.sql.execution.pythonUDF.arrow.enabled
• spark.sql.execution.pythonUDTF.arrow.enabled
• spark.sql.execution.arrow.pyspark.enabled (via its fallback spark.sql.execution.arrow.enabled)

The associated user-facing documentation and migration-guide entries added by SPARK-54555 are reverted as well, and the test_unified_udf test is restored to no longer force the conf off.

The revert was applied with git revert; the only manual conflict resolution was in the migration guide and the Arrow/pandas tutorial, where subsequent PRs had touched adjacent lines. Only the entries introduced by SPARK-54555 were removed; unrelated 4.1-to-4.2 migration notes are preserved.

Why are the changes needed?

Enabling these Arrow optimizations by default in 4.2.0 is being reverted so the change does not ship on by default in the 4.2.0 release. Users who want the Arrow-optimized behavior can still opt in explicitly via the same configs.

Does this PR introduce any user-facing change?

Yes. It restores the previous default configuration: regular Python UDFs, Python UDTFs, and PySpark Arrow-based columnar data exchange (DataFrame.toPandas / SparkSession.createDataFrame) are once again non-Arrow by default. This is a revert of an unreleased change (SPARK-54555 targeted 4.2.0), so there is no behavior change relative to released Spark versions.

How was this patch tested?

Reverting existing changes; relies on existing PySpark test coverage that runs with the Arrow conf both enabled and disabled. sql/catalyst compiles.

Was this patch authored or co-authored using generative AI tooling?

No

[Yicong-Huang]

Closing this PR. Instead of reverting, we will fix forward to reduce the regression via:

• [SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk #57099
• [SPARK-58023][PYTHON] Fast paths for converting Arrow string, binary, numeric and boolean columns to Python rows #57104
• [SPARK-58024][PYTHON] Convert Arrow struct and map columns to Python rows in bulk #57105

Note: the fix spans two parts. In addition to the PySpark-side changes above, a complementary fix is also needed in the Apache Arrow (PyArrow) community itself.