Add PredicatePushdownOptimizer in tracking-only mode #1262
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Jun 12, 2024
courtneyholcomb
approved these changes
Jun 12, 2024
| if len(node_semantic_models) != 1 or not current_pushdown_state.has_where_filters_to_push_down: | ||
| return self._default_handler(node) | ||
|
|
||
| source_semantic_model, *_ = node_semantic_models |
There was a problem hiding this comment.
Yeah that was mind-blowing. 🤯 🤣
This was referenced Jun 14, 2024
tlento
force-pushed
the
add-join-filter-itests
branch
from
ff2748f
to
8af1ba3
Compare
tlento
force-pushed
the
add-tracking-only-pushdown-optimizer
branch
from
186d5f5
to
c6f8777
Compare
This was referenced Jun 25, 2024
tlento
force-pushed
the
add-join-filter-itests
branch
from
8af1ba3
to
eec446c
Compare
In order to run predicate pushdown in a more robust manner we would be better served applying it as a post-plan-building optimizer rather than a build-time injection of additional where constraints. This PR adds the optimizer class with full tracking of predicate pushdown state, but it simply returns all nodes in the input DataflowPlan without modifying anything, since the predicate pushdown is already applied in the DataflowPlanBuilder. Future updates will move both the where constraint and time range constraint pushdown operations into this optimizer class. For now we simply evaluate the output from console logs to get a basis for future implementation changes.
tlento
force-pushed
the
add-tracking-only-pushdown-optimizer
branch
from
c6f8777
to
16ae775
Compare
This was referenced Jun 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order to run predicate pushdown in a more robust manner we would
be better served applying it as a post-plan-building optimizer rather
than a build-time injection of additional where constraints.
This PR adds the optimizer class with full tracking of predicate
pushdown state, but it simply returns all nodes in the input
DataflowPlan without modifying anything, since the predicate pushdown
is already applied in the DataflowPlanBuilder.
Future updates will move both the where constraint and time range constraint
pushdown operations into this optimizer class. For now we simply evaluate
the output from console logs to get a basis for future implementation changes.