define minimum row group

[djouallah]

I am using delta_rs ( which is based on datafusion) to write delta table, but by default it produce parquet files with 1 M row group, in my case, I want the minimum row group size to be 8 M rows, does datafusion support that functionality ?

[devinjdangelo]

Yes! See the configuration variable datafusion.execution.parquet.max_row_group_size https://arrow.apache.org/datafusion/user-guide/configs.html

I am not familiar with delta-rs enough to know how you can access this configuration directly, but it likely is possible.

[alamb]

Note the parquet writer also attempts to limit the size of each page (in addition to the numbr of rows in each row group)

Perhaps you can try and increase the page size limits as well as the row count limits

https://arrow.apache.org/datafusion/user-guide/sql/write_options.html#parquet-format-specific-options

DATA_PAGE_ROW_COUNT_LIMIT
DATA_PAGESIZE_LIMIT

[djouallah]

Thanks, I am familiar with max_row_group_size, I wanted rather something like min_row_group_size, I want the minimum to be 8 M

[devinjdangelo]

Hm... perhaps the name "max_row_group_size" is confusing. It sort of means the same thing as "minimum row group size" depending on your perspective. The parquet writer will continue to write data into a row group until it reaches "max_row_group_size" rows. Then, it will open a new row group and start writing to that. As a result, all but the very last row group will have exactly "max_row_group_size" rows.

So, if you set max_row_group_size to 8M the row groups will have 8M rows. You could have the very last row group be smaller if the total number of rows is not divisible by 8M. I am not aware of a mechanism in arrow-rs or datafusion to force even the last row group to be above a minimum value. You could do a count of your data up front and then choose a max_row_group_size value the exactly divides the rows to ensure even the last row group is the same size.