fix: Avoid panicing when stats are not available for a file group split

[mkleen]

Which issue does this PR close?

• Closes Panic in DataFusion 54.0.0 when ordering Parquet scan by computed projection alias #23219.

Rationale for this change

The query from the issue:

SELECT   (((Cast(id AS BIGINT) % 1024) + 1024) % 1024) AS computed_bucket
FROM     profile
ORDER BY computed_bucket,
         Cast(id AS BIGINT) limit 10;

panics:

thread 'main' panicked at .../datafusion-datasource-54.0.0/src/statistics.rs:100:48:
index out of bounds: the len is 0 but the index is 0

The underlying issue is that the current code panics when files are split by statistics and there are no statistics available for the column where the sort order is defined in this case computed_bucket.

What changes are included in this PR?

• Fix in MinMaxStatistics to check if there are stats available for a given column
• Test

Are these changes tested?

Yes

Are there any user-facing changes?

No

[comphead]

Thanks @mkleen WDYT if its possible to add the test to SLT test as well?

[notfilippo]

Thanks! We are seeing this issue as well in our DF54 upgrade. The fix matches our patch downstream.

[mkleen]

@comphead @geoffreyclaude @notfilippo Thanks for the review! I added all the feedback from you but i am having troubles to reproduce it in a plain slt. I will give it another try. The problem is the data creation.

[alamb]

Error rather than panic seems better to me.

It is not clear to me that this condition can happen during normal operation (e.g. it looks like maybe it happens due to a bug in partition reporting).

Thanks @mkleen

[alamb]

should this be an internal error (rather than a planning error)?

Also it might be nice to print out the requested statistics value (partition_value( and the toal number of columns in the error message to help debug issues

                            Err(plan_datafusion_err!("statistics not found for partition {partition_value}, expected at most {}", s.column_statistics.len() - 1))

[alamb]

Thanks! We are seeing this issue as well in our DF54 upgrade. The fix matches our patch downstream.

Added as candidate for backport to 54.1.0

@notfilippo do you have other issues you have seen during upgrade that you think we should backport?

[notfilippo]

have other issues

@alamb thanks for the question! As of now nothing else surfaced. If something else comes up I'll update the backport issue thread or I'll create a new issue.