Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unnecessary Filter on Parquet datasources #5149

Closed
Dandandan opened this issue Feb 1, 2023 · 2 comments
Closed

Unnecessary Filter on Parquet datasources #5149

Dandandan opened this issue Feb 1, 2023 · 2 comments
Labels
enhancement New feature or request performance Make DataFusion faster

Comments

@Dandandan
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Now that Parquet supports pushing down predicates to the scanning phase, we could remove the additional filter, as we're no longer providing an inexact filter.

Describe the solution you'd like
Add a method to FileFormat that returns whether the format supports exact filter pushdown.
The supports_filter_pushdown function should be updated accordingly.

Describe alternatives you've considered

Additional context

@Dandandan Dandandan added enhancement New feature or request performance Make DataFusion faster labels Feb 1, 2023
@Dandandan Dandandan changed the title Unecessary Filter on Parquet datasources Unnecessary Filter on Parquet datasources Feb 1, 2023
@tustvold
Copy link
Contributor

tustvold commented Feb 2, 2023

I think this may be a duplicate of #4028

@Dandandan
Copy link
Contributor Author

Yeah, it is. Thanks @tustvold

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance Make DataFusion faster
Projects
None yet
Development

No branches or pull requests

2 participants