Fix: eliminate unnecessary repartitioning for small datasets

[ShashidharM0118]

Which issue does this PR close?

• Closes Eliminate Repartitioning for Small Datasets #18595

Rationale for this change

Small datasets were undergoing unnecessary repartitioning, causing overhead without performance benefit. This change ensures that small Parquet datasets use single-partition aggregation.

What changes are included in this PR?

• Added a condition in the physical planner to check Parquet row count statistics before repartitioning. If the dataset is smaller than the batch_size, the planner selects AggregateMode::Single.
• Updated the aggregate_repartition.slt test file to expect AggregateExec: mode=Single for small Parquet datasets.

Are these changes tested?

Yes

Are there any user-facing changes?

Yes

[martin-g]

How about:

Suggested change

	if let Precision::Exact(num_rows) = stats.num_rows {
	if let Some(num_rows) = stats.num_rows.get_value() {

This will cover both Exact and Inexact.

[martin-g]

Suggested change

	return Ok(num_rows > batch_size);
	return Ok(num_rows >= batch_size);

[martin-g]

Does it have to fail the aggregation if the statistics fail for any reason ?
IMO it would be better to return Ok(true) instead.

[ShashidharM0118]

@martin-g, Thanks for the review!

I made these changes:

• Switched to stats.num_rows.get_value() instead of Precision::Exact(num_rows)
• Added check for round_robin_repartition() to respect when users want extra parallelism
• Added logic to get statistics, defaulting to repartitioning when stats aren't available

I set the threshold to 10 * batch_size. IMO, if the dataset size is only "in and around" a single batch size, distributing it creates "micro-batches" and causes unnecessary overhead. I am not entirely sure if this is the best value, so let me know your thoughts.

---

[martin-g]

• Added check for round_robin_repartition() to respect when users want extra parallelism

Is this really needed ?
If there are not enough rows to partition then there is no need of extra parallelism.

[martin-g]

I set the threshold to 10 * batch_size

Why 10 times ?
When I suggested adding support for Inexact I also thought about adding 10%, not 1000%.