[SPARK-54182][SQL][PYTHON] Optimize non-arrow conversion of df.toPandas

[Yicong-Huang]

What changes were proposed in this pull request?

Following up with #52680, this PR optimizes the non-Arrow path of toPandas() to eliminate intermediate DataFrame creation.

Key optimizations:

1. Avoid intermediate DataFrame copy

   • pd.DataFrame.from_records(rows) → Direct column extraction via zip(*rows)
   • 2 DataFrame creations → 1 DataFrame creation

2. Optimize column-by-column conversion (especially for wide tables)

   • Tuples → Lists for faster Series construction
   • Implicit dtype inference → Explicit dtype=object
   • pd.concat(axis="columns") + column rename → pd.concat(axis=1, keys=columns)
   • Result: 43-67% speedup for 50-100 columns

Why are the changes needed?

Problem: Current flow creates DataFrame twice:

• rows → pd.DataFrame.from_records() → temporary DataFrame → pd.concat() → final DataFrame

The intermediate DataFrame is immediately discarded, wasting memory. This is especially inefficient for wide tables where column-by-column overhead is significant.

Does this PR introduce any user-facing change?

No. This is a pure performance optimization with no API or behavior changes.

How was this patch tested?

• Existing unit tests.
• Benchmark

Benchmark setup:

• Hardware: Driver memory 4GB, Executor memory 4GB
• Configuration: spark.sql.execution.arrow.pyspark.enabled=false (testing non-Arrow path)
• Iterations: 10 iterations per test case for statistical reliability
• Test cases:
  • Simple (numeric columns)
  • Mixed (int, string, double, boolean)
  • Timestamp (date and timestamp types)
  • Nested (struct and array types)
  • Wide (5, 10, 50, 100 column counts)

Performance Results

General Benchmark (10 iterations):

Test Case	Rows	OLD → NEW	Speedup
simple	1M	1.376s → 1.383s	≈ Tied
mixed	1M	2.396s → 2.553s	6% slower
timestamp	500K	4.323s → 4.392s	≈ Tied
nested	100K	0.558s → 0.580s	4% slower
wide (50)	100K	1.458s → 1.141s	28% faster 🚀

Column Width Benchmark (100K rows, 10 iterations):

Columns	OLD → NEW	Speedup
5	0.188s → 0.179s	5% faster
10	0.262s → 0.270s	≈ Tied
50	1.430s → 0.998s	43% faster 🚀
100	3.320s → 1.988s	67% faster 🚀

Was this patch authored or co-authored using generative AI tooling?

Yes. Co-Generated-by Cursor

[zhengruifeng]

why is dtype=object necessary?

[Yicong-Huang]

here is building a series to pass to _create_converter_to_pandas, which will convert the series to the declared type in field.dataType. So I think the dtype here is optional and unnecessary. If we do not supply object, then when creating series it will start to infer the type, which will be quite slow.

[Yicong-Huang]

I also tried to use field.dataType explicitly but it would need some conversion to pandas data type, which is the purpose of _create_converter_to_pandas. So I suggest we keep use object for disabling type inference.

[zhengruifeng]

can we avoid creating columns_data: list[list] ?

[Yicong-Huang]

good point, although I think it is a list of list references so memory won't be too large of a difference. I changed it to iterator.

[dongjoon-hyun]

Thank you, @Yicong-Huang and @zhengruifeng . I updated the JIRA information to target to 4.2.0 according to the PR description.

Following up with #52680, this PR optimizes the non-Arrow path of toPandas() to eliminate intermediate DataFrame creation.

[zhengruifeng]

merged to master