Supporting filters in the left base table for join datasources #10697
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
abhishekagarwal87
force-pushed
the
correlated-subquery
branch
2 times, most recently
from
9c58008
to
8fe0229
Compare
abhishekagarwal87
force-pushed
the
correlated-subquery
branch
from
8fe0229
to
538244a
Compare
abhishekagarwal87
changed the title
clintropolis
reviewed
Mar 4, 2021
| this.preJoinableClauses = preJoinableClauses; | ||
| } | ||
|
|
||
| public static DataSourceAnalysis forDataSource(final DataSource dataSource) | ||
| { | ||
| // Strip outer queries, retaining querySegmentSpecs as we go down (lowest will become the 'baseQuerySegmentSpec'). | ||
| // Strip outer queries, retaining querySegmentSpecs as we go down (lowest will become the 'baseQuerySegmentSpec'o). |
There was a problem hiding this comment.
this was probably an accident?
| return Pair.of(dataSource, toFiltration(filter, virtualColumnRegistry)); | ||
| } | ||
| //TODO: We should avoid promoting the time filter as interval for right outer and full outer joins. This is not | ||
| // donw now as we apply the intervals to left base table today irrespective of the join type. |
clintropolis
approved these changes
Mar 4, 2021
gianm
reviewed
Mar 4, 2021
processing/src/main/java/org/apache/druid/segment/join/HashJoinSegmentStorageAdapter.java
Show resolved
Hide resolved
gianm
reviewed
Mar 5, 2021
| final List<RexNode> newProjectExprs = new ArrayList<>(); | ||
|
|
||
| // Already verified to be present in "matches", so just call "get". | ||
| // Can't be final, because we're going to reassign it up to a couple of times. | ||
| ConditionAnalysis conditionAnalysis = analyzeCondition(join.getCondition(), join.getLeft().getRowType()).get(); | ||
|
|
||
| if (left.getPartialDruidQuery().stage() == PartialDruidQuery.Stage.SELECT_PROJECT | ||
| && left.getPartialDruidQuery().getWhereFilter() == null) { | ||
| if (left.getPartialDruidQuery().stage() == PartialDruidQuery.Stage.SELECT_PROJECT) { |
There was a problem hiding this comment.
This may not work right if left is a DruidJoinQueryRel or DruidOuterQueryRel instead of a DruidQueryRel (since leftFilter is only supported for regular tables).
There was a problem hiding this comment.
oops. good catch. will fix this.
gianm
reviewed
Mar 5, 2021
gianm
reviewed
Mar 5, 2021
processing/src/main/java/org/apache/druid/query/planning/DataSourceAnalysis.java
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As of now, if the left query in the JOIN has a filter present, we plan it as the join between two query data sources even if the left source is a base table. This is of course not efficient. We had even introduced some custom calcite rules so that filters on join query are not pushed into the children tables. This issue is more significant for correlated queries since the generated physical plan is a left join with a filter on the left base table. Hence, the execution of correlation queries is not efficient.
This PR adds the support for pushing filters into the left child. We would add the support for the right child in a different PR and also get rid of the custom calcite rules once that happens.
What does not work
This PR has:
Key changed/added classes in this PR