Skip to content

Conversation

@mapleFU
Copy link
Member

@mapleFU mapleFU commented Oct 26, 2025

Which issue does this PR close?

Rationale for this change

Add Index and Footer info in parquet-layout

What changes are included in this PR?

  1. Add Index and Footer info in parquet-layout
  2. Expose metadata_size

Are these changes tested?

It was testing by hand.

Are there any user-facing changes?

No

@mapleFU mapleFU requested review from alamb and etseidl October 26, 2025 14:31
@github-actions github-actions bot added the parquet Changes to the parquet crate label Oct 26, 2025

#[derive(Serialize, Debug)]
struct Footer {
metadata_size: Option<usize>,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is required to estimate a prefetch_hint for files.

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me -- thank you @mapleFU

It would be nice to have some integration tests for this function , but given they don't yet exist I don't think you need to add it in this PR

@alamb alamb merged commit 560596e into apache:main Oct 28, 2025
16 checks passed
@mapleFU mapleFU deleted the parquet-layout-add-index-and-footer branch October 29, 2025 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parquet-layout: More info about layout including footer size, page index, bloom filter?

2 participants