[data] add read_parquet metadata fetch memory regression test

[justinvyu]

Description

• Add regression test for parquet schema inference memory scaling
• Verifies that read_parquet memory usage doesn't grow linearly with file count when the first file has a pa.null() column (which triggers the _infer_schema fallback path on PyArrow < 22.0)

Test plan

python -m pytest python/ray/data/tests/datasource/test_parquet.py::test_read_parquet_memory_growth -xvs

[gemini-code-assist]

Code Review

This pull request introduces a regression test, test_read_parquet_memory_growth, to ensure that memory usage during Parquet schema inference does not grow linearly with the number of files. The test compares memory deltas between reading small and large sets of files. Feedback was provided regarding the potential for a ZeroDivisionError in the ratio calculation if the memory delta for the small file set is zero, as well as the risk of misleading results if the delta is negative due to system memory management.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit adf870f. Configure here.}

[justinvyu]

Sanity check ✅

Fails on premerge before merging in datasource v2 from master:

[2026-05-18T22:42:30Z] >       assert ratio < 2, (
[2026-05-18T22:42:30Z]             f"Memory grew too much with more files: ratio={ratio:.1f}\n"
[2026-05-18T22:42:30Z]             f"delta_small={delta_small / 1024 / 1024:.1f} MiB, "
[2026-05-18T22:42:30Z]             f"delta_large={delta_large / 1024 / 1024:.1f} MiB"
[2026-05-18T22:42:30Z]         )
[2026-05-18T22:42:30Z] E       AssertionError: Memory grew too much with more files: ratio=12.6
[2026-05-18T22:42:30Z] E         delta_small=9.1 MiB, delta_large=114.8 MiB
[2026-05-18T22:42:30Z] E       assert 12.581335616438356 < 2
[2026-05-18T22:42:30Z]
[2026-05-18T22:42:30Z] python/ray/data/tests/datasource/test_parquet.py:1741: AssertionError

Passes on premerge after merging in datasource v2
Ran with a print statement on my local to produce this result:

  - delta_small: 2.58 MiB (100 files)
  - delta_large: 0.72 MiB (1000 files)
  - ratio: 0.28