Skip to content

Treat truncated parquet stats as inexact #15976

New issue
New issue
Open
#16412
Open
Treat truncated parquet stats as inexact#15976
#16412
Assignees
nssalianCookiePieWw
Labels
bugSomething isn't workinggood first issueGood for newcomers
@robert3005

Description

@robert3005
robert3005
opened on May 7, 2025

Describe the bug

When reading parquet files with truncated stats datafusion will report the min/max as exact even though metadata in the file indicates that min/max has been truncated

To Reproduce

Create a parquet file with truncated statistics, read the file, the statistics on the table are Exact

Expected behavior

The statistic should be Absent or Inexact

Additional context

No response

Activity

robert3005
added
bugSomething isn't working
on May 7, 2025
robert3005
mentioned this on May 7, 2025
alamb

alamb commented on May 16, 2025

@alamb
alamb
on May 16, 2025
Contributor

👍

alamb
added
good first issueGood for newcomers
on May 16, 2025
alamb

alamb commented on May 16, 2025

@alamb
alamb
on May 16, 2025
Contributor

I think this is a good first issue as there is a test case and clear explanation of the need

CookiePieWw

CookiePieWw commented on May 22, 2025

@CookiePieWw
CookiePieWw
on May 22, 2025

take

github-actions
assigned
CookiePieWw
on May 22, 2025
CookiePieWw

CookiePieWw commented on May 29, 2025

@CookiePieWw
CookiePieWw
on May 29, 2025 · edited by CookiePieWw

Hi :) I've spent some time on this and found the problem in get_col_stats

datafusion/datafusion/datasource-parquet/src/file_format.rs

Lines 1101 to 1107 in 2c2f225

ColumnStatistics {
null_count: null_counts[i],
max_value: max_value.map(Precision::Exact).unwrap_or(Precision::Absent),
min_value: min_value.map(Precision::Exact).unwrap_or(Precision::Absent),
sum_value: Precision::Absent,
distinct_count: Precision::Absent,
}

Here we always use Precision::Exact to wrap the stats, but actually we need to respect the is_max_value_exact and is_min_value_exact flags in the column metadata.

The max and min values are extracted at

datafusion/datafusion/datasource-parquet/src/file_format.rs

Lines 1112 to 1139 in 2c2f225

fn summarize_min_max_null_counts(
min_accs: &mut [Option<MinAccumulator>],
max_accs: &mut [Option<MaxAccumulator>],
null_counts_array: &mut [Precision<usize>],
arrow_schema_index: usize,
num_rows: usize,
stats_converter: &StatisticsConverter,
row_groups_metadata: &[RowGroupMetaData],
) -> Result<()> {
let max_values = stats_converter.row_group_maxes(row_groups_metadata)?;
let min_values = stats_converter.row_group_mins(row_groups_metadata)?;
let null_counts = stats_converter.row_group_null_counts(row_groups_metadata)?;
if let Some(max_acc) = &mut max_accs[arrow_schema_index] {
max_acc.update_batch(&[max_values])?;
}
if let Some(min_acc) = &mut min_accs[arrow_schema_index] {
min_acc.update_batch(&[min_values])?;
}
null_counts_array[arrow_schema_index] = Precision::Exact(match sum(&null_counts) {
Some(null_count) => null_count as usize,
None => num_rows,
});
Ok(())
}

But I didn't find a method to access the ..exact flags in StatisticsConverter, so my plan is to first add functions similar to row_group_mins to the converter to extract the flags, which requires a change to arrow-rs first, and then collect and pass the extracted boolean array of flags to get_col_stats to decide which one to use, Precision::Exact and Precision::InExact.

Please let me know if this direction makes sense.

alamb

alamb commented on May 29, 2025

@alamb
alamb
on May 29, 2025
Contributor

But I didn't find a method to access the ..exact flags in StatisticsConverter, so my plan is to first add functions similar to row_group_mins to the converter to extract the flags, which requires a change to arrow-rs first, and then collect and pass the extracted boolean array of flags to get_col_stats to decide which one to use, Precision::Exact and Precision::InExact.

I think the first thing that is needed in arrow-rs is to expose the is_max_value_exact and is_min_value_exact fields into the corresponding Rust structs (ValueStatistics):

https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html
https://docs.rs/parquet/latest/parquet/file/statistics/struct.ValueStatistics.html

CookiePieWw
mentioned this on May 31, 2025
CookiePieWw

CookiePieWw commented on May 31, 2025

@CookiePieWw
CookiePieWw
on May 31, 2025

Thanks for your feedback! I found ValueStatistics has already have max_is_exact and min_is_exact, seems we can directly make use of them. I've drafted a pr at apache/arrow-rs#7574 :)

alamb
added a commit that references this issue on Jun 6, 2025

feat: add `row_group_is_[max/min]_value_exact` to StatisticsConverter (…

9e575bd
CookiePieWw
linked a pull request that will close this issue on Jun 14, 2025
nssalian

nssalian commented on Jul 21, 2025

@nssalian
nssalian
on Jul 21, 2025

take

github-actions
assigned
nssalian
on Jul 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinggood first issueGood for newcomers

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @alamb@robert3005@nssalian@CookiePieWw

    Issue actions
      Treat truncated parquet stats as inexact · Issue #15976 · apache/datafusion