feat(planner): Introduce bitmap to record applied rules #10024

dusx1981 · 2023-02-13T13:00:05Z

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

A draft pr for code review.

closes #9536

1: Use RuleFactor class classification to manage rule sets 2: Add rule classification to rule

In order to compare the test results later, the old code is retained.

Add MetadataRef for CascadesOptimizer and Memo

vercel · 2023-02-13T13:00:10Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated
databend	⬜️ Ignored (Inspect)	Visit Preview		Feb 27, 2023 at 11:20PM (UTC)

mergify · 2023-02-13T13:04:20Z

This pull request's description is not fulfill the requirements. @dusx1981 please update it 🙏.

The description should contain the following:

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

## Summary

Summary about this PR

Close #issue

mergify · 2023-02-13T13:04:21Z

This pull request's title is not fulfill the requirements. @dusx1981 please update it 🙏.

Valid format:

fix(query): fix group by string bug
  ^         ^---------------------^
  |         |
  |         +-> Summary in present tense.
  |
  +-------> Type: rfc, feat, fix, refactor, ci, docs, chore

Valid types:

rfc: this PR proposes a new RFC
feat: this PR introduces a new feature to the codebase
fix: this PR patches a bug in codebase
refactor: this PR changes the code base without new features or bugfix
ci: this PR changes build/testing/ci steps
docs: this PR changes the documents or websites
chore: this PR only has small changes that no need to record

src/query/sql/src/planner/optimizer/cascades/tasks/explore_expr.rs

…pplied

dusx1981 · 2023-02-25T08:30:35Z

The problem with rule application has been solved and the code can already be reviewed.

dusx1981 · 2023-02-27T02:37:03Z

Solve the problem:

(x..y) is a half-open and half-closed interval, my initial implementation misses the last rule
The optimization result of HeuristicOptimizer is related to the order of the application rules, and the adjustment of the application rules is consistent with original order.

src/query/sql/src/planner/optimizer/heuristic/rule_list.rs

leiysky

Rest LGTM

dusx1981 · 2023-02-27T07:33:58Z

This issue caused by you didn't use our PR description template, and now it has been fixed.

looking forward to your review :).

xudong963 · 2023-02-27T07:37:19Z

Just notice the ticket, let me take a look before merging...

BohuTANG · 2023-02-27T07:38:45Z

Hi @dusx1981 ,

Nice, thanks for your contribution!
Can you give more summary for this PR? It would be great to add some logic tests for it if possible(Like

databend/tests/sqllogictests/suites/mode/standalone/explain/limit.test

Line 2 in d5ae263

explain select * from (select t.number from numbers(10) as t limit 8) limit 9

).

xudong963 · 2023-02-27T07:43:18Z

It would be great to add some logic tests for it if possible

I think the current tests are covered. The ticket only changes the way to record optimization rules.

src/query/sql/src/planner/optimizer/cascades/cascade.rs

dusx1981 · 2023-02-27T10:20:34Z

Do I need to change the status to resolved after answering the question?

dusx1981 · 2023-02-28T00:58:00Z

Metadata information in CascadesOptimizer and Memo has been removed:)

sundy-li · 2023-03-01T07:57:28Z

LGTM, thanks for the contribution!

dusx1981 added 19 commits February 5, 2023 15:24

Refactoring ruleset

004ab3f

1: Use RuleFactor class classification to manage rule sets 2: Add rule classification to rule

Add candidate_rules interface to Operator

98ac064

Merge branch 'main' of github.com:datafuselabs/databend into main

2830d2f

:Merge branch 'main' of github.com:datafuselabs/databend into main

10f9a18

Add static instance for rule factory

c958b07

Use bitmaps to manage the application of HeuristicOptimizer rules

98cda92

In order to compare the test results later, the old code is retained.

Merge branch 'main' of github.com:datafuselabs/databend into main

2ef6076

Merge branch 'main' of github.com:datafuselabs/databend into main

6c9aff7

Modify the code to be compatible with RuleFactory::create_rule

afa2576

Merge branch 'main' of github.com:datafuselabs/databend into main

3eb9032

Merge branch 'main' of github.com:datafuselabs/databend into main

02c3c13

Merge branch 'main' of github.com:datafuselabs/databend into main

40aaddb

Use RuleFactory to manage application of CascadesOptimizer rules

e201665

Add MetadataRef for CascadesOptimizer and Memo

Merge branch 'main' of github.com:datafuselabs/databend into main

c752d1b

Merge branch 'main' of github.com:datafuselabs/databend into main

0467b2f

Merge branch 'main' of github.com:datafuselabs/databend into main

dd3227c

CascadesOptimizer applies rule tuning

42da281

Merge branch 'main' of github.com:datafuselabs/databend into main

b62c7da

Solve automated inspection problems

bc9da6c

dusx1981 mentioned this pull request Feb 13, 2023

Feature: Introduce bitmap to record applied rules #9536

Closed

dusx1981 marked this pull request as ready for review February 13, 2023 13:03

sundy-li requested a review from leiysky February 13, 2023 13:42

leiysky reviewed Feb 13, 2023

View reviewed changes

src/query/sql/src/planner/optimizer/cascades/tasks/explore_expr.rs Outdated Show resolved Hide resolved

dusx1981 marked this pull request as draft February 14, 2023 00:56

dusx1981 added 2 commits February 14, 2023 20:49

Merge branch 'main' of github.com:datafuselabs/databend into main

2960470

Put unsafe code into safe functions

ace2d37

dusx1981 requested a review from leiysky February 14, 2023 14:31

dusx1981 added 5 commits February 25, 2023 11:11

Force a auto git test

ea7d63f

Adjust the order in which HeuristicOptimizer optimization rules are a…

2467410

…pplied

Use RuleFactory to manage rules for CascadesOptimizer

b132a72

Delete RuleList related logic

3c482b6

Replace bitand with &

7fa3682

dusx1981 requested review from Xuanwo and removed request for leiysky February 25, 2023 08:03

dusx1981 marked this pull request as ready for review February 25, 2023 08:04

BohuTANG requested a review from leiysky February 25, 2023 09:42

leiysky reviewed Feb 27, 2023

View reviewed changes

src/query/sql/src/planner/optimizer/heuristic/rule_list.rs Show resolved Hide resolved

leiysky approved these changes Feb 27, 2023

View reviewed changes

xudong963 reviewed Feb 27, 2023

View reviewed changes

src/query/sql/src/planner/optimizer/cascades/cascade.rs Show resolved Hide resolved

dusx1981 requested review from xudong963 and removed request for Xuanwo February 27, 2023 09:48

dusx1981 added 3 commits February 27, 2023 22:44

Merge branch 'main' of github.com:datafuselabs/databend into main

36abdcd

Remove unused metadata information in CascadesOptimizer and Memo

eaed8b4

fix lint error

21bc589

sundy-li approved these changes Mar 1, 2023

View reviewed changes

sundy-li merged commit a270653 into datafuselabs:main Mar 1, 2023

xudong963 mentioned this pull request Mar 1, 2023

chore: delete useless code #10282

Merged

dusx1981 mentioned this pull request Mar 21, 2023

fix(planner): Correct the mapping relationship between RuleID and DEFAULT_REWRITE_RULES #10508

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(planner): Introduce bitmap to record applied rules #10024

feat(planner): Introduce bitmap to record applied rules #10024

dusx1981 commented Feb 13, 2023 •

edited by sundy-li

Loading

vercel bot commented Feb 13, 2023 •

edited

Loading

mergify bot commented Feb 13, 2023

mergify bot commented Feb 13, 2023

dusx1981 commented Feb 25, 2023

dusx1981 commented Feb 27, 2023 •

edited

Loading

leiysky left a comment

dusx1981 commented Feb 27, 2023

xudong963 commented Feb 27, 2023

BohuTANG commented Feb 27, 2023

xudong963 commented Feb 27, 2023

dusx1981 commented Feb 27, 2023

dusx1981 commented Feb 28, 2023

sundy-li commented Mar 1, 2023

feat(planner): Introduce bitmap to record applied rules #10024

feat(planner): Introduce bitmap to record applied rules #10024

Conversation

dusx1981 commented Feb 13, 2023 • edited by sundy-li Loading

Summary

vercel bot commented Feb 13, 2023 • edited Loading

mergify bot commented Feb 13, 2023

mergify bot commented Feb 13, 2023

dusx1981 commented Feb 25, 2023

The problem with rule application has been solved and the code can already be reviewed.

dusx1981 commented Feb 27, 2023 • edited Loading

leiysky left a comment

Choose a reason for hiding this comment

dusx1981 commented Feb 27, 2023

xudong963 commented Feb 27, 2023

BohuTANG commented Feb 27, 2023

xudong963 commented Feb 27, 2023

dusx1981 commented Feb 27, 2023

dusx1981 commented Feb 28, 2023

sundy-li commented Mar 1, 2023

dusx1981 commented Feb 13, 2023 •

edited by sundy-li

Loading

vercel bot commented Feb 13, 2023 •

edited

Loading

dusx1981 commented Feb 27, 2023 •

edited

Loading