[C++][Datasets] Improve memory usage of datasets API when scanning parquet #30892
Description
This is a more targeted fix to improve memory usage when scanning parquet files. It is related to broader issues like ARROW-14648 but those will likely take longer to fix. The goal here is to make it possible to scan large parquet datasets with many files where each file has reasonably sized row groups (e.g. 1 million rows). Currently we run out of memory scanning a configuration as simple as:
21 parquet files
Each parquet file has 10 million rows split into row groups of size 1 million
Reporter: Weston Pace / @westonpace
Assignee: Weston Pace / @westonpace
Related issues:
- [C++][Datasets] Improve memory usage of datasets (is depended upon by)
PRs and other links:
Note: This issue was originally created as ARROW-15410. Please see the migration documentation for further details.