Discussion: should set FileScanConfig's batch_size to min(batch_size, limit)

[niebayes]

If a limit is set for FileScanConfig, I wonder if it's reasonable to also set the batch_size to min(batch_size, limit), which might reduces the number of rows being scanned and decoded.

Originally posted apache/arrow-rs#8836

[zhuqi-lucas]

This is a good point, we may can add some unit test to confirm it will scan more data if we have limit.

[niebayes]

Generally, the default batch size is relatively large (8192) compared to the limit (typically, 1 ~ 10). I think this optimization would introduce considerable performance improvement.

[alamb]

I suspect for sources that already handle the limit specially (like Parquet reader) it won't change behavior very much

However, for sources that don't have special limit handling, this change would improve things