[15721] Query Rewrite #1340

zhaoguoquan94 · 2018-05-05T00:04:02Z

This PR adds support for query rewriting optimizations that are oblivious to the cost model in Peloton.

Transitive predicates (optimized in OptimizeLoop)
Constant folding (optimized in PostgresParser).

We implement query rewrite rules using the current optimizer framework:

Transitive predicates: we implemented this rule by adding a new step to the beginning OptimizerLoop using a TopDownRewriter. We added rules to take predicates from nodes, generating all possible predicates by taking a transitive closure and add the new ones;
Trivial predicate simplification: we used the same framework as in the transitive predicates rewrite. This method removes trivial predicates like a = a, some of which are generated by the transitive predicates rule;

Constant folding can transform queries like select * from table where t.column1 = 3*(2-1) to select * from table where t.column1 = 3. These transformation are done in parser level instead of optimizer level because of implementation simplicity. And we don't really lose any information.
It support 3 type of constant folding

Operators like +-*/
Comparison operators like >=, NOT, is null,
Conjunctions : AND OR.

…t expression * clang-format modified files.

* fix expression_util_test

This reverts commit a2467cf.

… into expression_rewrite

…t expression * clang-format modified files.

* fix expression_util_test

This reverts commit a2467cf.

… into expression_rewrite

…he addressSanitizer's output are marked with correct line number or not.

…hether the addressSanitizer's output are marked with correct line number or not." try to make address sanitizer report something. This reverts commit 00020ce.

… into expression_rewrite

nappelson · 2018-05-11T01:16:55Z

src/expression/comparison_expression.cpp

+    return false;
+  }
+
+  auto vcl = children_[0].get();


it isn't clear exactly what vcl stands for. would recommend spelling it out or at least adding a comment. same applies to the other acronyms below

nappelson · 2018-05-11T01:17:45Z

src/expression/comparison_expression.cpp

+        throw Exception("Invalid comparison expression type.");
+    }
+  } catch (std::exception &e) {
+    return false;


you should log out the exception here unless if you follow my proposal earlier

nappelson · 2018-05-11T01:19:33Z

src/expression/comparison_expression.cpp

+        return exp_type_ == oexp_type_ && ocl->ExactlyEquals(*vcl) &&
+               ocr->ExactlyEquals(*vcr);
+      default:
+        throw Exception("Invalid comparison expression type.");


if you are simply catching the exception you throw here, you should probably just return false in default and log information that there was an invalid comparison expression type. If you want the exception to cause a failure, then you can throw but would recommend not catching here. If you are expecting another exception from this try block, then I'd recommend picking a finer grained exception in the catch statement.

nappelson · 2018-05-11T01:25:59Z

src/include/optimizer/rule_query_rewrite.h

+
+  void Transform(std::shared_ptr<OperatorExpression> input,
+                 std::vector<std::shared_ptr<OperatorExpression>> &transformed,
+                 OptimizeContext *context) const override;


i think OptimizeContext is usually passed around as a shared_ptr in other parts of the optimizer. you may want to do that for consistency and to gain the benefits of it

nappelson · 2018-05-11T01:28:10Z

src/include/optimizer/rule_query_rewrite.h

+ public:
+  TransitivePredicatesLogicalGet();
+
+  bool Check(std::shared_ptr<OperatorExpression> plan,


as you are declaring the same methods here and below for all of these classes, i'd recommend creating an abstract class that inherits from Rule so that you don't need to duplicate the declaration of the same Check and Transform methods

nappelson · 2018-05-11T01:30:08Z

src/optimizer/rule_query_rewrite.cpp

+  match_pattern = std::make_shared<Pattern>(OpType::Get);
+}
+
+bool TransitivePredicatesLogicalGet::Check(std::shared_ptr<OperatorExpression> input,


add a TODO for what you actually want to do here. same for the other Check methods that do this

nappelson · 2018-05-11T01:31:26Z

src/optimizer/util.cpp

@@ -23,6 +26,145 @@ namespace peloton {
 namespace optimizer {
 namespace util {

+void FillTransitiveTable(


i think this method should have some inline documentation about what is happening here

nappelson · 2018-05-11T01:32:41Z

src/optimizer/util.cpp

+
+  for (auto predicate : predicates) {
+    std::string exp_type = ExpressionTypeToString(predicate.expr->GetExpressionType());
+    if (exp_type.substr(0, 7) == "COMPARE") {


i'd try and avoid using magic number (7 in this case)

Agreed, you could probably do it like how it is done here

GustavoAngulo

Overall looks great, just some comments and clarifications here and there.

GustavoAngulo · 2018-05-13T14:34:36Z

src/optimizer/optimizer.cpp

+
+  // Apply query rewrite rules
+  task_stack->Push(new TopDownRewrite(root_group_id, root_context,
+                                      RewriteRuleSetName::SIMPLIFY_PREDICATES));


Why is simplifying predicates executed after transitive predicates? Will this not apply the transitive predicate rule on trivial predicates that will just get thrown out by the simplify predicate rule? Seems like we can save some work by flipping the order of these

GustavoAngulo · 2018-05-13T14:38:42Z

src/optimizer/rule_query_rewrite.cpp

+TransitivePredicatesLogicalFilter::TransitivePredicatesLogicalFilter() {
+  type_ = RuleType::TRANSITIVE_PREDICATES_LOGICAL_FILTER;
+
+  std::shared_ptr<Pattern> child(std::make_shared<Pattern>(OpType::InnerJoin));


Why is the child for this a join and not a leaf? Is there a reason we should only apply this to filters with joins as children?

GustavoAngulo · 2018-05-13T14:40:12Z

src/optimizer/rule_query_rewrite.cpp

+SimplifyPredicatesLogicalFilter::SimplifyPredicatesLogicalFilter() {
+  type_ = RuleType::SIMPLIFY_PREDICATES_LOGICAL_FILTER;
+
+  std::shared_ptr<Pattern> child(std::make_shared<Pattern>(OpType::InnerJoin));


Same comment as in earlier LogicalFilter rule, is there a reason for this to be an InnerJoin and not a leaf?

GustavoAngulo · 2018-05-13T14:45:15Z

src/optimizer/util.cpp

+
+  for (auto predicate : predicates) {
+    std::string exp_type = ExpressionTypeToString(predicate.expr->GetExpressionType());
+    if (exp_type.substr(0, 7) == "COMPARE") {


Agreed, you could probably do it like how it is done here

GustavoAngulo · 2018-05-13T14:58:58Z

src/optimizer/util.cpp

+    }
+  }
+
+  return new_predicates;


Do we ever add the original predicate to the new_predicates list?

apavlo · 2018-06-21T13:42:51Z

Closing. We will have @Zeninma build this in the fall with a proper rewrite layer.

zhaoguoquan94 and others added 16 commits April 29, 2018 20:50

Make pattern a template class.

a2467cf

Initial version of transitive predicates

401e2c9

Add simplifying rule

8ae39d7

Remove symmetric predicates from transitive predicates generation

6e064cd

constant folding at parser-level.

860122d

* constant folding at parser level : comparison expression -> constan…

6b75ec0

…t expression * clang-format modified files.

* add constant folding : conjuction expreesion -> constant expression

c093d9d

* fix expression_util_test

add some tests.

7719185

Revert "Make pattern a template class."

d52f858

This reverts commit a2467cf.

Comments for constant folding & clang-format.

0b62167

Code cleanup

5cff2bb

Merge branch 'expression_rewrite' of github.com:zhaoguoquan94/peloton…

5a18e9c

… into expression_rewrite

Add test

68ac990

Fix a memory leak

01eb1a5

Refactor

f4ba551

Refactor more

4b493b9

zhaoguoquan94 added do not merge class-project labels May 5, 2018

Add setting for transitive predicates

194b7ed

zhaoguoquan94 changed the title ~~[15721] Expression Rewrite~~ [15721] Query Rewrite May 5, 2018

zhaoguoquan94 and others added 10 commits May 4, 2018 20:10

Make pattern a template class.

c205367

Initial version of transitive predicates

7168c6b

Add simplifying rule

6c055ee

Remove symmetric predicates from transitive predicates generation

7847470

constant folding at parser-level.

95d8172

* constant folding at parser level : comparison expression -> constan…

bb3c9ab

…t expression * clang-format modified files.

* add constant folding : conjuction expreesion -> constant expression

cef94da

* fix expression_util_test

add some tests.

979c019

Revert "Make pattern a template class."

1d01b74

This reverts commit a2467cf.

Comments for constant folding & clang-format.

efe1915

gangsterveggies and others added 16 commits May 4, 2018 20:10

Code cleanup

5cda514

Add test

9188f56

Fix a memory leak

bccd2c9

Refactor

cce8d7d

Refactor more

b70e981

Add setting for transitive predicates

2a34cd1

Merge branch 'expression_rewrite' of github.com:zhaoguoquan94/peloton…

3f57cd2

… into expression_rewrite

Merge branch 'master' into expression_rewrite

4086d57

Merge branch 'expression_rewrite' of github.com:zhaoguoquan94/peloton…

a9cdd86

… into expression_rewrite

use llvm-symbolizer to get line number.

b0b453f

fix c++ delete grammar error. Stupid...

821953b

This commit should failed. Commit it because we need to see whether t…

00020ce

…he addressSanitizer's output are marked with correct line number or not.

Revert "This commit should failed. Commit it because we need to see w…

ca04a4a

…hether the addressSanitizer's output are marked with correct line number or not." try to make address sanitizer report something. This reverts commit 00020ce.

expression_util_test should be good. Still trying to fix plan_util_test

a9b49fc

Merge branch 'expression_rewrite' of github.com:zhaoguoquan94/peloton…

8d04cee

… into expression_rewrite

Fix plan_util_test

7dcb685

nappelson suggested changes May 11, 2018

View reviewed changes

GustavoAngulo reviewed May 13, 2018

View reviewed changes

apavlo closed this Jun 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[15721] Query Rewrite #1340

[15721] Query Rewrite #1340

zhaoguoquan94 commented May 5, 2018 •

edited

Loading

nappelson May 11, 2018

nappelson May 11, 2018

nappelson May 11, 2018

nappelson May 11, 2018

nappelson May 11, 2018

nappelson May 11, 2018

nappelson May 11, 2018

nappelson May 11, 2018

GustavoAngulo May 13, 2018

GustavoAngulo left a comment

GustavoAngulo May 13, 2018

GustavoAngulo May 13, 2018

GustavoAngulo May 13, 2018

GustavoAngulo May 13, 2018

GustavoAngulo May 13, 2018

apavlo commented Jun 21, 2018

[15721] Query Rewrite #1340

[15721] Query Rewrite #1340

Conversation

zhaoguoquan94 commented May 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GustavoAngulo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apavlo commented Jun 21, 2018

zhaoguoquan94 commented May 5, 2018 •

edited

Loading