Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet Writer Ignores Statistics specification in WriterProperties #2014

Closed
tustvold opened this issue Jul 6, 2022 · 3 comments
Closed

Parquet Writer Ignores Statistics specification in WriterProperties #2014

tustvold opened this issue Jul 6, 2022 · 3 comments
Labels
bug parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

tustvold commented Jul 6, 2022

Describe the bug

The parquet writer does not respect the settings in WriterProperties for whether to compute statistics, and the max size of the computed statistics.

To Reproduce

Inspect code

Expected behavior

We should not have configuration options that have no effect
@tustvold tustvold added the bug label Jul 6, 2022
@tustvold tustvold changed the title Parquet Writer Ignores WriterProperties Statistics Options Parquet Writer Ignores Statistics WriterProperties Jul 6, 2022
@tustvold tustvold mentioned this issue Jul 7, 2022
Merged
@alamb alamb changed the title Parquet Writer Ignores Statistics WriterProperties Parquet Writer Ignores Statistics specification in WriterProperties Jul 7, 2022
@alamb alamb added the parquet Changes to the parquet crate label Jul 7, 2022
@alamb
Copy link
Contributor

alamb commented Jul 8, 2022

Update here: #2022 fixes the writer to respect the "do/don't compute stats", however the "max size of computed statistics" setting is still ignored, which is presumably why @tustvold left it open.

@tustvold do you mind if I break out the "max size of stats setting is ignored" into a separate ticket to make the current state of affairs clearer?

@tustvold
Copy link
Contributor Author

tustvold commented Jul 8, 2022

SGTM or feel free to modify this ticket directly

@alamb
Copy link
Contributor

alamb commented Jul 8, 2022

Follow on ticket is #2033

@alamb alamb closed this as completed Jul 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug parquet Changes to the parquet crate
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alamb @tustvold