GH-46404: [C++][Python] add support for max_page_header_size for pyarrow#47758
Open
ayushbansal07 wants to merge 1 commit into
Open
GH-46404: [C++][Python] add support for max_page_header_size for pyarrow#47758ayushbansal07 wants to merge 1 commit into
ayushbansal07 wants to merge 1 commit into
Conversation
ayushbansal07
requested review from
AlenkaF,
raulcd,
rok and
wgtmac
as code owners
|
|
github-actions
Bot
added
Component: Parquet
Component: C++
Component: Python
awaiting review
wgtmac
reviewed
Oct 9, 2025
| @@ -260,7 +261,7 @@ class SerializedPageReader : public PageReader { | |||
| // Fills in data_page_statistics. | |||
| bool ShouldSkipPage(EncodedStatistics* data_page_statistics); | |||
|
|
|||
Member
There was a problem hiding this comment.
I'd suggest keeping it as const. We can change the function set_max_page_header_size to do nothing after this PR and mark it as deprecated.
| arrow::TimeUnit::type coerce_int96_timestamp_unit = arrow::TimeUnit::NANO; | ||
| Type::type binary_type = Type::BINARY; | ||
| Type::type list_type = Type::LIST; | ||
| uint32_t max_page_header_size = 0; |
Member
There was a problem hiding this comment.
Why 0 but not a meaningful default?
| }; | ||
|
|
||
| // 16 MB is the default maximum page header size | ||
| static constexpr uint32_t kDefaultMaxPageHeaderSize = 16 * 1024 * 1024; |
Member
There was a problem hiding this comment.
Suggested change
| static constexpr uint32_t kDefaultMaxPageHeaderSize = 16 * 1024 * 1024; | |
| constexpr uint32_t kDefaultMaxPageHeaderSize = 16 * 1024 * 1024; |
Member
There was a problem hiding this comment.
Also let's make this int32_t not uint32_t.
github-actions
Bot
added
awaiting committer review
pitrou
requested changes
Oct 9, 2025
| /// Set the size of the buffered stream buffer in bytes. | ||
| void set_buffer_size(int64_t size) { buffer_size_ = size; } | ||
|
|
||
| /// Return the size of the buffered stream buffer. 0 means default |
Member
There was a problem hiding this comment.
Please, can you make sure the docstrings actually describe the methods inside of blindly copy-pasting them?
Comment on lines
+108
to
+110
| uint32_t max_page_header_size() const { return max_page_header_size_; } | ||
| /// Set the size of the buffered stream buffer in bytes. 0 means default | ||
| void set_max_page_header_size(uint32_t size) { max_page_header_size_ = size; } |
Member
There was a problem hiding this comment.
Can we change these to take int32_t? We try to avoid unsigned integers in public API.
| }; | ||
|
|
||
| // 16 MB is the default maximum page header size | ||
| static constexpr uint32_t kDefaultMaxPageHeaderSize = 16 * 1024 * 1024; |
Member
There was a problem hiding this comment.
Also let's make this int32_t not uint32_t.
| the canonical `arrow.uuid` extension type). | ||
| max_page_header_size : int, default None | ||
| If not None, override the maximum size of a page header. | ||
| Deafults to 16MB, which should be sufficient for most Parquet files. |
Member
There was a problem hiding this comment.
Suggested change
| Deafults to 16MB, which should be sufficient for most Parquet files. | |
| Defaults to 16MB, which should be sufficient for most Parquet files. |
| thrift_container_size_limit=None, filesystem=None, | ||
| page_checksum_verification=False, arrow_extensions_enabled=True): | ||
| page_checksum_verification=False, arrow_extensions_enabled=True, | ||
| max_page_header_size=None): |
Member
There was a problem hiding this comment.
It's ok, but is the new argument also available on pq.read_table and friends?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Support in pyarrow to read parquet files with page header statistics which contain values larger than 8MiB.
What changes are included in this PR?
Are these changes tested?
Yes, tried with a sample python code.
Are there any user-facing changes?
Yes
pyarrow#46404