Skip to content

Conversation

@Dandandan
Copy link
Contributor

Which issue does this PR close?

Closes #220

Rationale for this change

DataFusion gives wrong results for filters that don't reference any column

What changes are included in this PR?

Workarounds the issue by not pushing down predicates that do not reference any column (1=2, FALSE, TRUE).
As we should be able to remove these anyway with constant folding and removing/replacing WHERE FALSE/ WHERE TRUE, I think this is a good option.

Are there any user-facing changes?

No, results should be correct now.

@Dandandan Dandandan changed the title Fix Filter /where clause without column names is removed in optimization pass #220 Fix Filter /where clause without column names is removed in optimization pass Apr 29, 2021
@Dandandan Dandandan changed the title Fix Filter /where clause without column names is removed in optimization pass Fix Filter / where clause without column names is removed in optimization pass Apr 29, 2021
@codecov-commenter
Copy link

codecov-commenter commented Apr 29, 2021

Codecov Report

Merging #225 (94c5b3a) into master (e86ad26) will increase coverage by 0.12%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #225      +/-   ##
==========================================
+ Coverage   76.30%   76.43%   +0.12%     
==========================================
  Files         134      135       +1     
  Lines       23170    23264      +94     
==========================================
+ Hits        17681    17782     +101     
+ Misses       5489     5482       -7     
Impacted Files Coverage Δ
datafusion/src/optimizer/filter_push_down.rs 97.73% <100.00%> (+0.07%) ⬆️
datafusion/tests/sql.rs 99.88% <100.00%> (+1.42%) ⬆️
datafusion/src/catalog/information_schema.rs 86.72% <0.00%> (-2.06%) ⬇️
datafusion/src/physical_plan/hash_join.rs 85.94% <0.00%> (-0.49%) ⬇️
datafusion/src/execution/context.rs 92.63% <0.00%> (-0.31%) ⬇️
ballista/rust/executor/src/main.rs 0.00% <0.00%> (ø)
datafusion/src/datasource/datasource.rs 100.00% <0.00%> (ø)
datafusion/src/optimizer/eliminate_limit.rs 88.88% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e86ad26...94c5b3a. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit 2423ff0 into apache:master Apr 30, 2021
Copy link
Member

@jorgecarleitao jorgecarleitao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, ready to sail! :)

@alamb
Copy link
Contributor

alamb commented May 3, 2021

Thanks @Dandandan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Filter / where clause without column names is removed in optimization pass

5 participants