[Tracking] Refactor SQL layer

[leiysky]

To build a robust SQL processing system, it's necessary to divide the components clearly. It is helpful to understand the code.

RFC: https://databend.rs/doc/contributing/rfcs/new-sql-planner-framework

Tracking issue for RFC: #1853

Tasks:

• Parser: the component to parse SQL string into AST(Abstract Syntax Tree) [Tracking] Refactor SQL Parser #1218
• Binder: the component takes AST as input and produce a logical plan [Tracking] Refactor PlanBuilder #1219
• Optimizer: the component can do some optimization to a logical plan, and eventually produce a nice physical plan
• Catalog: the component maintains all items in the database(e.g. table, column, index, function), which can help Binder to build a correct logical plan Catalog API #867
• Executor: the component to execute physical plan and produce query result

[zhang2014]

Maybe we will follow it:

[leiysky]

Maybe we will follow it:

Yes, it's exactly what I mean.

Most of the RDBMS basicly follows the architecture. The benefits are:

• Friendly to database developers, they can get up with the project quickly.
• We can utilize the generic approaches in DB area to improve our system.

And the most important reason is, I'm familiar with this 🤣 .

[zhang2014]

Does the picture come from the lecture of CMU 15-445?

It from https://15721.courses.cs.cmu.edu/spring2020/

[leiysky]

Does the picture come from the lecture of CMU 15-445?

It from https://15721.courses.cs.cmu.edu/spring2020/

I've followed Andy's coureses for years. 15721 is a nice course which presents the latest academic results and industrial development every year.

[xudong963]

RFC: https://github.com/datafuselabs/databend/blob/master/website/databend/docs/rfcs/query/0005-new-sql-planner-framework.md

404

[PsiACE]

RFC: https://github.com/datafuselabs/databend/blob/master/website/databend/docs/rfcs/query/0005-new-sql-planner-framework.md

404

https://github.com/datafuselabs/databend/blob/main/website/databend/docs/dev/rfcs/query/0005-new-sql-planner-framework.md

[FANNG1]

https://github.com/datafuselabs/databend/blob/main/website/databend/docs/dev/rfcs/query/0005-new-sql-planner-framework.md is 404 , @PsiACE could you update the latest rfc doc?

[BohuTANG]

https://github.com/datafuselabs/databend/blob/main/website/databend/docs/dev/rfcs/query/0005-new-sql-planner-framework.md is 404 , @PsiACE could you update the latest rfc doc?

Sure, please forward to https://databend.rs/doc/contributing/rfcs/new-sql-planner-framework

[dusx1981]

Regarding determining the rules used by Operator, GPORCA's solution is to use bitmaps to manage Rules, and define a RuleFactory to manage all Rules in a unified manner. And classify Rule into Exploration and Implementation. Operator holds a bitmap of the rules that can be applied. When applying a rule, take the bitmap in Expression and all the bitmaps in RuleFactory and to determine the final applied rule, then traverse and apply the rule. After the Exploration stage is completed, all equivalent expressions are generated, and a new tree is generated with Pattern. Is this more efficient? I Translated GPORCA from C++ to Go，And IT works fine。

type RuleFactory struct {
	// range of all rules
	IdToRule map[RuleId]Rule

	// bitset of exploration rules
	ExplorationSet RuleSet

	// bitset of implementation rules
	ImplementationSet RuleSet
}

// Logical Operator
type Logical interface {
	Operator

	// rules can be applied to this operator
	CandidateRules() RuleSet
}