Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet Metadata Store deduplication on filter defaults to true instead of false #70

Closed
gallushi opened this issue May 12, 2021 · 1 comment
Closed

Parquet Metadata Store deduplication on filter defaults to true instead of false #70

gallushi opened this issue May 12, 2021 · 1 comment

Comments

@gallushi
Copy link
Contributor

Deduplication during filtering is documented as false by default, but accidentally configured as true by default.
This does not affect query results in any way, but may degrade performance in some cases, as it causes a distinct to be run on the metadata.
to disable dedup during filtering, set io.xskipper.parquet.filter.dedup to false

From the next release (the one following 1.2.3) will be disabled by default.
@gallushi gallushi changed the title Parquet Metadata Store deduplication on filter default to true instead of false Parquet Metadata Store deduplication on filter defaults to true instead of false May 12, 2021
@guykhazma
Copy link
Member

fixed in #72

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@guykhazma @gallushi