[SPARK-58013][PS][PYTHON] Support pandas 3 in pandas-on-Spark tests

[HyukjinKwon]

What changes were proposed in this pull request?

Make the pandas-on-Spark test suite (doctests + a few pyarrow conversion tests) pass under pandas 3.0, fixing main code where pandas-on-Spark diverged from pandas rather than skipping tests.

Main-code fixes

• internal.py: pandas 3 collapses a contiguous integer index column into a RangeIndex via set_index. Reconstruct as a materialized Index when the result is not the canonical default (start=0, step=1), matching plain pandas behavior.
• sql/pandas/types.py, typedef/typehints.py: gate StringType → pd.StringDtype on future.infer_string so the option is respected.

Test harness

• Each module's _test() sets pd.set_option("future.infer_string", False) before running doctests, preserving the pandas < 3 string dtype spelling. Scoped to doctest entry points only — unit tests keep native pandas 3 behavior. (TODO: remove with SPARK-58014 when min pandas >= 3.)
• test_describe.py: normalize empty-timestamp describe() stats (NaT/nan → None) to match pandas-on-Spark's object-column representation.

Doctest updates (genuine pandas 3 differences only)

• datetime/timedelta unit: [ns] → [...] via ELLIPSIS (pandas 3 defaults to microsecond).
• factorize uniques: None → ... (nan under pandas 3).
• # doctest: +SKIP with reason for bool() (removed), apply(sum) (TypeError), and two default-RangeIndex cases indistinguishable at conversion.

pyarrow tests: version-aware for ns→us default, new string dtype, and zoneinfo timezones.

Why are the changes needed?

Build / Python-only (Python 3.12, Pandas 3) has been failing. This makes it pass.

Does this PR introduce any user-facing change?

No. The only behavioral change is in internal.py: an explicitly materialized integer index now round-trips as Index([...]) instead of RangeIndex(...) under pandas 3, matching plain pandas.

How was this patch tested?

All 19 pandas-on-Spark doctest modules verified locally against pandas 3.0.3 (0 real failures). Verified on CI with the pandas 3 image.

• Passing pandas 3 run: https://github.com/HyukjinKwon/spark/actions/runs/28891453096
• Previously failing run: https://github.com/apache/spark/actions/runs/28755311863

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

Co-authored-by: Ruifeng Zheng 7322292+zhengruifeng@users.noreply.github.com
Co-authored-by: Fangchen Li 7614606+fangchenli@users.noreply.github.com

[HyukjinKwon]

cc @zhengruifeng @fangchenli this should make pandas 3 CI green. Can you take a look? I am adding both of you are co-author of this PR.

[HyukjinKwon]

Code review summary

Reviewed the full diff. No blocking findings — this is a well-scoped, low-risk pandas 3 compatibility change. The main-code fixes are correct and localized, and the doctest/test adjustments are limited to genuine, semantically-equivalent pandas 3 behavior changes.

Verified correct

• window.py (sem) — std(ddof) = sqrt(var_samp·(count-1)/(count-ddof)), then / sqrt(count). Since Spark's var_samp = SS/(count-1), the (count-1) cancels to SS/(count-ddof) = variance with the ddof divisor, so any ddof is honored. The count > ddof guard (unchanged from the pandas<3 branch) protects the count-ddof denominator.
• internal.py (restore_index) — is_non_default_range = start != 0 or step != 1 correctly separates a genuine default RangeIndex from a materialized integer index. The ps.Index([0, 1, 2]) (contiguous-from-zero) case that's indistinguishable from a default index is acknowledged in the PR description and handled via +SKIP.
• types.py / typehints.py — the two future.infer_string gates are logically consistent (both require pandas≥3 and the option enabled), and each short-circuits so pd.get_option("future.infer_string") is never evaluated on pandas<3 — no KeyError on older pandas.
• test_pyarrow_arrow_to_pandas_default.py — the overrides= kwarg is pre-existing infrastructure on master (signature + consumer both present), so passing it here is safe.
• Doctest-heavy files (frame/generic/series/namespace/datetimes/indexes/*) contain no hidden logic changes — all additions are [...] / +ELLIPSIS / +SKIP output edits.

Minor / optional (non-blocking)

• window.py — the pandas≥3 and pandas<3 branches now duplicate the sem structure. Fine as-is given the version divergence; if the pandas<3 support window is short, a follow-up could collapse them once the minimum is bumped (same lifecycle as the __init__.py TODO).
• internal.py — for an empty explicit index (ps.Index([])), pdf.index is RangeIndex(0, 0, 1) → not reconstructed → stays an empty RangeIndex rather than an empty Index. Edge case, indistinguishable at conversion time, and consistent with the documented default-vs-materialized limitation — noting only for completeness.

Housekeeping (already tracked by the author)

• [SPARK-XXXXX] placeholders in the title and TODO(SPARK-XXXXX) need a real ticket before finalizing.
• The [TEMP][DO-NOT-MERGE] fork-only workflow (.github/workflows/zz_fork_pandas3_verify.yml) is to be dropped before review, per the PR note. It contains no secrets or risky patterns.

LGTM once the draft/DO-NOT-MERGE scaffolding is removed and the JIRA ID is filled in.

[gaogaotiantian]

I have doubts about this.. We are literally disabling tests for pandas 3 and that's not how we support pandas 3.. The users will still run into issues.

And from my understanding when I was working on this issue. I don't believe this PR can fix all the issues for pandas 3. I also don't believe we can have a single PR that's simple enough to support pandas 3 for spark. BTW I don't believe we should solve this by asking claude to "support pandas 3 on spark" either.

[HyukjinKwon]

This PR only disables doctest by disabling panda 3 output. They are still tested through unittests. For doctests, we can enable it back when we bump up minimum pandas to 3.

In any event, things have to be tested through unittests ad doctests are more for showing examples in any event.

[HyukjinKwon]

They are being set under _test for doctests