[Python] New row_group_size default of 1 Mi not working

[jonashaag]

Describe the bug, including details regarding any error messages, version, and platform.

In the release notes of Arrow 12 say that the default row group size has been lowered from 64 Mi to 1 Mi. But here's what's happening in practice (PyArrow from conda-forge):

In [1]: import pyarrow.parquet as pq

In [2]: import pyarrow as pa

In [3]: pa.__version__
Out[3]: '12.0.0'

In [12]: pq.write_table(pa.Table.from_pydict({"x": [42] * (64 * 1024 * 1024 + 1)}), "/tmp/x")

In [13]: pq.read_metadata("/tmp/x")
Out[13]:
<pyarrow._parquet.FileMetaData object at 0x7fa6b81eaa70>
  created_by: parquet-cpp-arrow version 12.0.0
  num_columns: 1
  num_rows: 67108865
  num_row_groups: 2  # <=============
  format_version: 2.6
  serialized_size: 485

In [14]: pq.write_table(pa.Table.from_pydict({"x": [42] * (64 * 1024 * 1024 + 0)}), "/tmp/x")

In [15]: pq.read_metadata("/tmp/x")
Out[15]:
<pyarrow._parquet.FileMetaData object at 0x7fa6b823d490>
  created_by: parquet-cpp-arrow version 12.0.0
  num_columns: 1
  num_rows: 67108864
  num_row_groups: 1  # <=============
  format_version: 2.6
  serialized_size: 376

Component(s)

Python

[westonpace]

This is embarrassing, and shame on me for not writing better regression tests.

#34281 changed the default for C++ and python but it was too strict and it wasn't possible (via python) to go past the default.

#34435 restored the ability to go past the default but it looks like it changed the default for pyarrow in the process.

I'll put in a fix soon.