-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Examples of using TreeNode
APIs to walk and manipulate LogicalPlans
#10628
Comments
@alamb This is a great idea. As I learn more perhaps I can help. |
Thank you @cisaacson |
Moving out of slack into Github so it might be more easily found If your usecase is to to get the list of filters and tables that appear in a query, one way to do this is:
So I think it could look something like (untested) let mut referenced_tables = HashSet::new();
let mut filters = vec![];
// recursively visit all nodes in the tree (including subqueries)
logical_plan.apply_with_subqueries(|plan| {
match plan {
// record table names
LogicalPlan::TableScan(table_scan) => {
referenced_tables.insert(table_scan.table_name);
},
// record filters
LogicalPlan::Filter(filter) => {
filters.push(filter.predicate);
}
// ignore other nodes
_ => {}
Ok(TreeNodeRecursion::Continue)
}
})?; |
This looks very good, pretty much what I implemented. The only question I have remaining is: If |
Depending on the value of https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.TableProviderFilterPushDown.html sometimes filters would not be pushed down into the provider Also filters can be duplicated in the table scan and filter (again depending on the value of the FilterPushDown |
Got it, that makes total sense. I will only care about the accepted pushdown filters, so the |
Is your feature request related to a problem or challenge?
The usecase of walking
LogicalPlan
to either analyze the plan or rewrite it has come up multiple times recentlySpecifically I think both @goldmedal and @cisaacson have asked about this recently and so I think the documentation is not clear enough
Describe the solution you'd like
In order to help people do this, I would like to add some examples of how to walk the tree
I suggest first adding simple doc example in the docs of
LogicalPlan
itselfThe example would do somethig like create a plan using
LogicalPlanBuilder
And then show how to:
TreeNode::apply
to recursively find something about the scan (maybe the table name?)TreeNode::rewrite
to recursively rewrite and remove the Filter (or something)I think those would be relatively short examples to write and would illustrate the key apis
Describe alternatives you've considered
Bonus points for a more full featured example (perhaps linked from the
LogicalPlan
docs) of how use / walk / manipulate LogicalPlansSimilarly to https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/expr_api.rs, perahaps
datafusion-examples/examples/plan_walk.rs
or somethingAdditional context
No response
The text was updated successfully, but these errors were encountered: