Simplify Expr::map_children #9876

peter-toth · 2024-03-30T20:00:01Z

Which issue does this PR close?

Closes #9457, part of #8913

Rationale for this change

The current implementation of Expr::map_children() is very complex, this PR tries to simplify its code.

What changes are included in this PR?

This PR adds a map_until_stop_and_collect() macro, which is similar to TransformedIterator::map_until_stop_and_collect() to process sibling tree nodes, but it can process heterogeneous tree node containing expressions.

Are these changes tested?

Yes, with existing UTs.

Are there any user-facing changes?

No.

peter-toth · 2024-04-02T12:20:49Z

This is now ready for review. cc @alamb, @berkaysynnada

alamb

Thank you @peter-toth -- I think this looks quite nice and easier to undertand

I left some suggestions on comments / naming that might help, but I also think we could do them as a follow on PR

alamb · 2024-04-02T19:33:17Z

datafusion/common/src/tree_node.rs

@@ -532,7 +532,7 @@ impl<T> Transformed<T> {
    }
 }

-/// Transformation helper to process tree nodes that are siblings.
+/// Transformation helper to process sequence of iterable tree nodes that are siblings.


I found the reason / logic for continue / jump quite subtle when trying to make TreeNodeMutator. I have some suggested comments below to explain the rationale that might help

Thanks @alamb, merged your suggestion.

datafusion/common/src/tree_node.rs

alamb · 2024-04-02T19:44:31Z

datafusion/common/src/tree_node.rs

+/// Transformation helper to process sequence of tree node containing expressions.
+/// This macro is very similar to [TransformedIterator::map_until_stop_and_collect] to
+/// process nodes that are siblings, but it accepts an initial transformation and a
+/// sequence of pairs of an expression and its transformation.


This is quite clever. I have some suggestions on naming that would have helped me:

Use F to mirror the nomenclature of TransformedIterator. I thought $TRANSFORMED_EXPR was an actually expr (but it is actually a closure that is invoked I think 🤔 )

$TRANSFORMED_EXPR_0 --> F0 $TRANSFORMED_EXPR --> F.

I think it would help to document each parameter and the return value (specifically it looks like it returns Transformed<(data, ..., data)>?

Sure, let me fix these and come back yo you.

Renamed in a36f1aa and added rustdoc to the macro. Let me know if it needs more details.

alamb · 2024-04-02T19:47:23Z

datafusion/common/src/tree_node.rs

+macro_rules! map_until_stop_and_collect {
+    ($TRANSFORMED_EXPR_0:expr, $($EXPR:expr, $TRANSFORMED_EXPR:expr),*) => {{
+        $TRANSFORMED_EXPR_0.and_then(|Transformed { data: data0, mut transformed, mut tnr }| {
+            let data = (


Maybe we could name this something other than data so it was clearer what type it is (maybe all_datas?)

Fixed in a36f1aa

alamb · 2024-04-02T19:49:40Z

datafusion/expr/src/tree_node/expr.rs

-                        null_treatment,
-                    ))
-                }),
+            }) => map_until_stop_and_collect!(


this is very clever

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

peter-toth · 2024-04-03T18:31:33Z

The test failure seems unrelated to this PR and looks like a non-deterministic test...

alamb · 2024-04-03T19:24:38Z

The test failure seems unrelated to this PR and looks like a non-deterministic test...

I agree

https://github.com/apache/arrow-datafusion/actions/runs/8543135777/job/23406267036?pr=9876

External error: query result mismatch:
[SQL] WITH RECURSIVE my_cte AS(
    SELECT 1::int AS a
    UNION ALL
    SELECT a::bigint+2 FROM my_cte WHERE a<3
) SELECT *, arrow_typeof(a) FROM my_cte;
[Diff] (-expected|+actual)
-   1 Int32
-   3 Int32
+   3 Int32
+   1 Int32
at test_files/cte.slt:751

I restarted it

alamb

Thanks again @peter-toth

alamb · 2024-04-03T20:19:58Z

The CI test passed on rerun. I filed #9930 to track the non deterministic test (and we are trying to get to issue 10,000!)

peter-toth · 2024-04-04T06:38:59Z

Thanks @alamb for the review!

add map_until_stop_and_collect macro

cf55bc6

github-actions bot added the logical-expr Logical plan and expressions label Mar 30, 2024

fix clippy

451cab7

peter-toth marked this pull request as draft March 31, 2024 11:52

simplify

86de064

peter-toth mentioned this pull request Apr 2, 2024

DEMO: Introduce TreeNodeMutator for rewriting TreeNodes in place, change optimizer to rewrite LogicalPlan in place - 10% faster planning time #9780

Closed

5 tasks

peter-toth marked this pull request as ready for review April 2, 2024 09:29

alamb mentioned this pull request Apr 2, 2024

DataFusion weekly project plan (Andrew Lamb) - April 1, 2024 #9899

Closed

7 tasks

peter-toth mentioned this pull request Apr 2, 2024

Consistent LogicalPlan subquery handling in TreeNode::apply and TreeNode::visit #9913

Merged

alamb approved these changes Apr 2, 2024

View reviewed changes

peter-toth and others added 3 commits April 3, 2024 12:05

Update datafusion/common/src/tree_node.rs

f113547

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

add documentation

a36f1aa

fix macro

cd18ca3

alamb approved these changes Apr 3, 2024

View reviewed changes

alamb mentioned this pull request Apr 3, 2024

Intermittent failure in test_files/cte.slt:751 #9930

Closed

alamb merged commit 2f55003 into apache:main Apr 3, 2024
24 checks passed

alamb mentioned this pull request Apr 8, 2024

Epic: Unified TreeNode rewrite API #8913

Closed

9 tasks

peter-toth mentioned this pull request Apr 12, 2024

Improve documentation on TreeNode #10035

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify Expr::map_children #9876

Simplify Expr::map_children #9876

peter-toth commented Mar 30, 2024 •

edited

Loading

peter-toth commented Apr 2, 2024

alamb left a comment

alamb Apr 2, 2024

peter-toth Apr 3, 2024

alamb Apr 2, 2024

peter-toth Apr 3, 2024

peter-toth Apr 3, 2024 •

edited

Loading

alamb Apr 2, 2024

peter-toth Apr 3, 2024

alamb Apr 2, 2024

peter-toth commented Apr 3, 2024

alamb commented Apr 3, 2024

alamb left a comment

alamb commented Apr 3, 2024

peter-toth commented Apr 4, 2024

Simplify Expr::map_children #9876

Simplify Expr::map_children #9876

Conversation

peter-toth commented Mar 30, 2024 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

peter-toth commented Apr 2, 2024

alamb left a comment

Choose a reason for hiding this comment

alamb Apr 2, 2024

Choose a reason for hiding this comment

peter-toth Apr 3, 2024

Choose a reason for hiding this comment

alamb Apr 2, 2024

Choose a reason for hiding this comment

peter-toth Apr 3, 2024

Choose a reason for hiding this comment

peter-toth Apr 3, 2024 • edited Loading

Choose a reason for hiding this comment

alamb Apr 2, 2024

Choose a reason for hiding this comment

peter-toth Apr 3, 2024

Choose a reason for hiding this comment

alamb Apr 2, 2024

Choose a reason for hiding this comment

peter-toth commented Apr 3, 2024

alamb commented Apr 3, 2024

alamb left a comment

Choose a reason for hiding this comment

alamb commented Apr 3, 2024

peter-toth commented Apr 4, 2024

peter-toth commented Mar 30, 2024 •

edited

Loading

peter-toth Apr 3, 2024 •

edited

Loading