[SPARK-58657][PYTHON][TESTS] Add tests for pa.Array.from_pandas with the mask argument

[Spenserrrr]

What changes were proposed in this pull request?

This PR adds a golden-file test pinning the behavior of pa.Array.from_pandas() under the mask argument, part of the umbrella effort (SPARK-54936) to monitor upstream PyArrow/pandas behavior that PySpark's pandas <-> Arrow conversion layer depends on. These tests run without a Spark session and act as a drift canary: if a library upgrade changes the primitive's behavior, the golden comparison fails loudly at the exact cell.

The changes:

• New test_pyarrow_array_from_pandas_non_default.py with PyArrowArrayFromPandasMaskTests, which subclasses _PyArrowFromPandasTestBase (from the default test) to reuse its source Series inventory, and records golden_pyarrow_array_from_pandas_mask.{csv,md} (98 rows x 3 columns: pandas series | mask=None | mask=isnull()).
• A small behavior-preserving refactor of the merged default test, test_pyarrow_array_from_pandas_default.py:
  • the source-Series inventory and repr_from_pandas_result were already lifted into a test-free base _PyArrowFromPandasTestBase, with PyArrowArrayFromPandasDefaultTests as a thin subclass, so the non-default tests can subclass the base directly (no unbound-call trick, and the base carries no test_* to be re-collected);
  • the shared _from_pandas_cell(series, **kwargs) cell formatter is now a method on that base, used by both the default and the new mask test.
• Registered the new module in dev/sparktestsupport/modules.py.

PySpark does not pass mask freely; it derives it from how the Series is stored (python/pyspark/sql/conversion.py:435, python/pyspark/sql/pandas/conversion.py:113):

mask = None if hasattr(series.array, "__arrow_array__") else series.isnull()

The golden records both mask=None and mask=isnull() so the two regimes stay observable:

• numpy-backed dtypes do not implement __arrow_array__, so PySpark passes mask=series.isnull(). from_pandas also infers nulls from the Series at mask=None, so the two columns agree; the test pins that agreement.
• protocol dtypes (implementing __arrow_array__ -- the nullable extension, string[python], and [pyarrow] dtypes) return a finished Arrow array with their own validity bitmap, so PyArrow rejects a caller-supplied mask with ValueError on the argument's mere presence (even an all-False no-op mask raises). PySpark passes mask=None for them, and mask=isnull() records ERR@ValueError.

Because mask is fixed by the input row's dtype, it is a column pair rather than an independent matrix dimension.

Why are the changes needed?

PySpark calls pa.Array.from_pandas(series, mask=mask, ...) on the pandas -> Arrow path (createDataFrame(pandas_df) and every pandas UDF's return value). The mask argument's two-regime behavior -- silently accepted on numpy-backed input, rejected on protocol input -- is upstream behavior PySpark relies on but does not itself test. A golden test makes any future drift (for example, protocol dtypes beginning to accept a mask, or mask=None no longer inferring nulls) fail visibly in CI instead of silently changing conversion results.

Does this PR introduce any user-facing change?

No. This adds tests only.

How was this patch tested?

New golden-file test, run with and without SPARK_GENERATE_GOLDEN_FILES=1 (regeneration is byte-identical). Validated across the full support matrix -- PyArrow 18, 19, 20, 21, 22, 23, 24, 25 x pandas 2 and 3 (16/16 combinations pass), each in a fresh virtualenv running the committed test against the committed golden. ruff check and ruff format --check are clean. The pandas-3 differences are recorded as version-guarded overrides (notably string:inferred, whose dtype becomes the dedicated str type on pandas 3 and therefore moves from a mask-accepting row to a protocol row).

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Anthropic), model Claude Opus 4.8

[Spenserrrr]

Hi @Yicong-Huang @zhengruifeng! This is a PR adding from_pandas monitoring for upstream behavior monitoring. Could you take a look when you have a moment? Thank you!

[uros-b]

@Spenserrrr Please rebase in accordance with #57864.

[Spenserrrr]

Thanks for your reminder! I just rebased it.