Populate partition column statistics for PartitionedFile #19284
Merged
+296
−311
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
core
adriangb
commented
Dec 11, 2025
Comment on lines
-669
to
-720
| let schema_adapter = self.create_schema_adapter(); | ||
| let (schema_mapper, _) = schema_adapter.map_schema(self.file_schema.as_ref())?; |
Contributor
Author
There was a problem hiding this comment.
This was mapping from file schema -> table schema by filling in unknown stats for partition columns. It did not do any per-file adaptation, it assumes all files already have the same schema.
adriangb
force-pushed
the
partition-statistics
branch
from
e1d1c91 to
26f07ab
Compare
mbutrovich
approved these changes
Dec 11, 2025
Contributor
mbutrovich
left a comment
mbutrovich
left a comment
There was a problem hiding this comment.
That's a nice net-negative line count! Thanks @adriangb!
adriangb
force-pushed
the
partition-statistics
branch
from
3ad1f1f to
e15dbe0
Compare
Contributor
Author
|
Thanks @mbutrovich ! I'll leave this open for another day or so for public comment. |
Contributor
Author
|
@xudong963 in particular I'm interested in your input since you authored #15865 |
martin-g
approved these changes
Dec 11, 2025
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
adriangb
force-pushed
the
partition-statistics
branch
from
c457754 to
2b569a9
Compare
xudong963
approved these changes
Dec 15, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Superseeds #15865
Part of #16800
The idea here was to remove usage of
SchemaAdapterand at the same time actually populate the partition column statistics.