[Python][C++] No longer possible to specify higher chunksize than the default for Parquet writing

[jorisvandenbossche]

Describe the bug, including details regarding any error messages, version, and platform.

See #34374 (comment) for context

#34281 changed the default row group size (chunksize in the C++ WriteTable API). However, that PR changed DEFAULT_MAX_ROW_GROUP_LENGTH, which doesn't set the default chunksize, but actually caps the max chunk (row group) size regardless of the user-specified chunksize.

It seems this constant is used both for setting this max upper cap, and as the default values for chunk_size in WriteTable. I assume we will have to distinguish those two meanings.

Component(s)

C++, Parquet, Python

[westonpace]

Ah, I think the problem is that parquet::arrow::FileWriter::WriteTable has a chunk_size argument (which pyarrow sets) and it also has a parquet::WriterProperties which has max_row_group_length which pyarrow does not set (and this latter property overrides the former). So either we need to change pyarrow to set parquet::WriterProperties::max_chunk_size or only change the default for chunk_size (this might be preferred).

[jorisvandenbossche]

or only change the default for chunk_size (this might be preferred).

I don't know what the typical usage is from C++? For that, it might be more useful to actually change the max_row_group_length? (since not everyone will write through WriteTable)

I misinterpreted max_row_group_length, thinking it was meant as some global max size we still set regardless of what the user provides, but through WriterProperties, it's basically the chunk size argument users can set using the properties interface. It's also only WriteTable that gives the additional chunk_size keyword, while other write methods like WriteRecordBatch only use the properties max_row_group_length.

Naively, I would expect that specifying chunk_size in WriteTable would overrule the max_row_group_size, instead of being overwritten by that.
Changing that to give priority to chunk_size would also solve the issue, but that's a breaking change?

So either we need to change pyarrow to set parquet::WriterProperties::max_row_group_length

That's also not that simple, since it has similar logic as in C++: the ParquetWriter class is created with properties, and then afterwards the write_table method has this chunk_size keyword, but at that point the Writer object was already created (and in theory you can also call that method multiple times with different chunk_size values for writing to the same file).

[wjones127]

For Python, could we just set max_row_group_length to a very high value (knowing that it will always be overridden with chunk_size)? That seems to be the status quo right now, right? (That is, you can't from Python write more than 64 million rows per group)

[westonpace]

For Python, could we just set max_row_group_length to a very high value (knowing that it will always be overridden with chunk_size)? That seems to be the status quo right now, right? (That is, you can't from Python write more than 64 million rows per group)

That seems like a reasonable compromise. I think python will always use write_table and so it should work.

[westonpace]

I created #34435 with @wjones127 's suggestion

[jorisvandenbossche]

@wjones127 good idea!