Skip to content

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Mar 7, 2023

Which issue does this PR close?

Follow on to #5423

Rationale for this change

as @izveigor pointed out on https://github.com/apache/arrow-datafusion/pull/5423/files#r1125686357, the tests in expr simplifier used a more verbose style that he followed, which makes sense.

However, I would like to keep the codebase easier to maintain and thus use less ceremony for creating Exprs.

What changes are included in this PR?

  1. Support ^, &, |, << and >> opertors for building exprs
  2. Simplify expression simply test cases

Are there any user-facing changes?

You can now create exprs like

let expr = lit(0) & lit(45)

@github-actions github-actions bot added logical-expr Logical plan and expressions optimizer Optimizer rules labels Mar 7, 2023
@alamb alamb changed the title Minor: simplify simplify test cases, support ^, &, |, << and >> opertors for building exprs Simplify simplify test cases, support ^, &, |, << and >> opertors for building exprs Mar 7, 2023
}

#[test]
fn test_simplify_simplify_eq_expr() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think check of expr_plus in test_simplify_simplify_eq_expr is redundant because it is already checked in test_simplify_simplify_arithmetic_expr

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks -- I was trying to split up test_simplify_simplify_arithmetic_exprs into two separate tests as it seemed like it was testing two things, However, now that i see this diff It just looks like I added a new one. I will fix it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 1afb570

@comphead
Copy link
Contributor

comphead commented Mar 8, 2023

looks great, just wondering should we also preserve old syntax in tests?

like

 let expr_a = binary_expr(col("c2"), Operator::Multiply, lit(0));
            let expr_b = binary_expr(lit(0), Operator::Multiply, col("c2"));

If this syntax is not deprecated, it should probably be tested as well?

@alamb
Copy link
Contributor Author

alamb commented Mar 9, 2023

If this syntax is not deprecated, it should probably be tested as well?

I agree it should be tested. I will review our existing coverage and see if anything else is neede

fn test_simplify_multiply_by_one() {
let expr_a = binary_expr(col("c2"), Operator::Multiply, lit(1));
let expr_b = binary_expr(lit(1), Operator::Multiply, col("c2"));
let expr_a = col("c2") * lit(1);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no intended logical changes in this file, only simplifying how the tests are expressed (and there is a significant number of lines removed)

col("c1").lt(lit(6)),
),
Operator::BitwiseAnd,
let expr = bitwise_and(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also tried writing this using the new & operator, but I thought the structure (precidence) of the resulting expression was significantly harder to see, so I switched to using the bitwise_and type functions to reduce some duplication

}

#[test]
fn test_simplify_simplify_eq_expr() {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks -- I was trying to split up test_simplify_simplify_arithmetic_exprs into two separate tests as it seemed like it was testing two things, However, now that i see this diff It just looks like I added a new one. I will fix it.

@alamb alamb changed the title Simplify simplify test cases, support ^, &, |, << and >> opertors for building exprs Simplify simplify test cases, support ^, &, |, << and >> operators for building exprs Mar 9, 2023
@alamb
Copy link
Contributor Author

alamb commented Mar 9, 2023

If this syntax is not deprecated, it should probably be tested as well?

I agree it should be tested. I will review our existing coverage and see if anything else is needed

I reviewed the use of binary_expr and I think there is plenty of other coverage (see https://github.com/search?q=repo%3Aapache%2Farrow-datafusion%20binary_expr&type=code)

I didn't see much focused testing for the functions defined in in https://github.com/apache/arrow-datafusion/blob/main/datafusion/expr/src/expr_fn.rs

Thus I conclude this PR doesn't impact the test coverage for expression creation functions.

If we want to cover the functions in expr_fn.rs more deliberately or in a more structured way, I think we could do it as a follow on PR / add tests specifically for that coverage, rather than rely on tests for other functionality (simplification) covering somewhat unrelated functionality (expr creation)

@alamb alamb merged commit aaa4e14 into apache:main Mar 11, 2023
@ursabot
Copy link

ursabot commented Mar 11, 2023

Benchmark runs are scheduled for baseline = 9587339 and contender = aaa4e14. aaa4e14 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@alamb alamb deleted the alamb/simplify_simplifier branch October 18, 2024 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants