Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move all expression related structs to Tarantool's backend #3235

Closed
kyukhin opened this issue Mar 13, 2018 · 5 comments
Closed

Move all expression related structs to Tarantool's backend #3235

kyukhin opened this issue Mar 13, 2018 · 5 comments
Assignees
Labels
feature A new functionality refactoring Code refactoring sql
Milestone

Comments

@kyukhin
Copy link
Contributor

kyukhin commented Mar 13, 2018

One of biggest parts which still live in legacy DD of SQL are structures which
deal with expressions. They're:

  • struct Expr
  • struct ExprList
  • struct ExprSpan

Expr is used by:

  • struct Table to store CHECK constraints. This should be moved to struct space
  • struct Column to store default value for table's column. Probably, this should be moved to struct key_def
  • struct Index to store WHERE clause for partial index. Probably should be moved to struct index.
  • struct Index to store functional index expression. E.g. if we're indexing on (a*a) field. Should go to struct index as well.
@kyukhin
Copy link
Contributor Author

kyukhin commented Mar 14, 2018

F-2-f discussion:

  1. Need to understand if Exprs in DEFAULT can contain sub-queries
  2. High level plan: clean-up/refactor struct Expr, then integrate into Tarantool
  3. FOR EACH ROW constraints should be stored in Tarantool structs and ultimately invoked in Tarantool's triggers, not in VDBE
  4. FK constraints should be checked by VDBE, but stored in Tarantool

kyukhin added a commit that referenced this issue Mar 16, 2018
Make SQL load schema during initialization. It was
initialized on first query before the patch. Replace
check that schema is loaded w/ assert arouns the code base.

As far as DDL is prohibited inside the transaction, schema
cannot be changed inside a transaction and hence, no need to
reload it during rollback. No internal changes
(SQLITE_InernChanges) are possible.

Part of #3235
@kyukhin
Copy link
Contributor Author

kyukhin commented Mar 20, 2018

Let's start from integration of default column attribute into Tarantool core.
The default value is supposed to be executed from any execution environment:
either Lua or SQL and perform AST invocation by VDBE instance.

Proposed changes:

  1. Add two new fields to struct field_def:
    a) default_type. Type of default expression, string type. The only possible value so far:
    sql
    b) default_value. Default expression itself, string type. Should contain string expression
    to be compiled into AST and used when performing default insert. Example: date() * time()
  2. Add single field to struct tuple_field: default, type struct Expr. This field should
    store compiled AST and VDBE executor should be invoked each time when default value
    is needed.

kyukhin added a commit that referenced this issue Mar 20, 2018
Make SQL load schema during initialization. It was
initialized on first query before the patch. Replace
check that schema is loaded w/ assert arouns the code base.

As far as DDL is prohibited inside the transaction, schema
cannot be changed inside a transaction and hence, no need to
reload it during rollback. No internal changes
(SQLITE_InernChanges) are possible.

Part of #3235
@kyukhin
Copy link
Contributor Author

kyukhin commented Mar 22, 2018

The main part of this issue is exposing expressions-related parts of parser to Tarantool's core.
I.e. make it possible to store SQL expression (string value) in, say, field_def structure and during
preparation of tuple_fieldto be able to translate it into AST, struct Expr *.
This AST can be used during byte code generation if, say, a default value is to be inserted.

Need to introduce a flag which will tell to parser, that all is needed is AST, no bytecode should be
emitted. This flag should also signal that no name resolution should be performed: default
expression is parsed during DDL and struct space* is not exists so far. Name resolution
should be performed later on.
The context of expression is really not wide: only constants and built-in functions are allowed
for DEFAULT and only table under construction columns for CHECK constraint. Hence,
approach would be:

  1. Introduce a flag signalling that no name resolution to be performed by parser and no byte-code
    to be emitted
  2. Convert expression string to "SELECT " and invoke the parser
  3. Find parsed struct Expr* in result struct Select* or report error

This approach should work also for CHECK constraints and maybe partial indexes.

@Gerold103
Copy link
Collaborator

There is no struct Select * if you only parse. You should get only ExprList in parser struct. When struct Select * is generated, it resolves names, that is not needed. In fact, you should stop the parser before it starts search FROM clause.

kyukhin added a commit that referenced this issue Mar 27, 2018
Extract expressions parsing into separate routine to
allow Tarantool's backend compile DEFAULT statements w/o
SQL machinery at all. So far, for DEFAULT values no context
is needed: only constant expressions and built-ins are allowed.
In future, table context will be added to allow use column
values for CHECK constraint expressions.

Move DEFAULT string value to space_def. Move compiled expresion
to field_def as well. Reason not to move compiled expression
to tuple_field as follows: we do not want to engage parser
during tuple validation. So, do it in alter.cc.

Closes #3235
kyukhin added a commit that referenced this issue Mar 27, 2018
Extract expressions parsing into separate routine to
allow Tarantool's backend compile DEFAULT statements w/o
SQL machinery at all. So far, for DEFAULT values no context
is needed: only constant expressions and built-ins are allowed.
In future, table context will be added to allow use column
values for CHECK constraint expressions.

Move DEFAULT string value to space_def. Move compiled expresion
to field_def as well. Reason not to move compiled expression
to tuple_field as follows: we do not want to engage parser
during tuple validation. So, do it in alter.cc.

Part of #3235
kyukhin added a commit that referenced this issue Mar 29, 2018
Extract expressions parsing into separate routine to
allow Tarantool's backend compile DEFAULT statements w/o
SQL machinery at all. So far, for DEFAULT values no context
is needed: only constant expressions and built-ins are allowed.
In future, table context will be added to allow use column
values for CHECK constraint expressions.

Move DEFAULT string value to space_def. Move compiled expresion
to field_def as well. Reason not to move compiled expression
to tuple_field as follows: we do not want to engage parser
during tuple validation. So, do it in alter.cc.

In order to allow expression duplication in alter.cc: expose
those routines from expr.c and make their names correspond to
coding style.

Since recovery is performed before SQL fronend initialization:
split it into two pieces: 1. create SQL handler, enable
all subsystems 2. Do recovery.  This will allow to run
parser during recovery, since it needs db handle so far.

Part of #3235
kyukhin added a commit that referenced this issue Mar 29, 2018
Extract expressions parsing into separate routine to
allow Tarantool's backend compile DEFAULT statements w/o
SQL machinery at all. So far, for DEFAULT values no context
is needed: only constant expressions and built-ins are allowed.
In future, table context will be added to allow use column
values for CHECK constraint expressions.

Move DEFAULT string value to space_def. Move compiled expresion
to field_def as well. Reason not to move compiled expression
to tuple_field as follows: we do not want to engage parser
during tuple validation. So, do it in alter.cc.

In order to allow expression duplication in alter.cc: expose
those routines from expr.c and make their names correspond to
coding style.

Since recovery is performed before SQL fronend initialization:
split it into two pieces: 1. create SQL handler, enable
all subsystems 2. Do recovery.  This will allow to run
parser during recovery, since it needs db handle so far.

Part of #3235
kyukhin added a commit that referenced this issue Mar 31, 2018
Extract expressions parsing into separate routine to
allow Tarantool's backend compile DEFAULT statements w/o
SQL machinery at all. So far, for DEFAULT values no context
is needed: only constant expressions and built-ins are allowed.
In future, table context will be added to allow use column
values for CHECK constraint expressions.

Move DEFAULT string value to space_def. Move compiled expresion
to field_def as well. Reason not to move compiled expression
to tuple_field as follows: we do not want to engage parser
during tuple validation. So, do it in alter.cc.

In order to allow expression duplication in alter.cc: expose
those routines from expr.c and make their names correspond to
coding style.

Since recovery is performed before SQL fronend initialization:
split it into two pieces: 1. create SQL handler, enable
all subsystems 2. Do recovery.  This will allow to run
parser during recovery, since it needs db handle so far.

Part of #3235
kyukhin added a commit that referenced this issue Apr 3, 2018
Extract expressions parsing into separate routine to
allow Tarantool's backend compile DEFAULT statements w/o
SQL machinery at all. So far, for DEFAULT values no context
is needed: only constant expressions and built-ins are allowed.
In future, table context will be added to allow use column
values for CHECK constraint expressions.

Move DEFAULT string value to space_def. Move compiled expresion
to field_def as well. Reason not to move compiled expression
to tuple_field as follows: we do not want to engage parser
during tuple validation. So, do it in alter.cc.

In order to allow expression duplication in alter.cc: expose
those routines from expr.c and make their names correspond to
coding style.

Since recovery is performed before SQL fronend initialization:
split it into two pieces: 1. create SQL handler, enable
all subsystems 2. Do recovery.  This will allow to run
parser during recovery, since it needs db handle so far.

Part of #3235
kyukhin added a commit that referenced this issue Apr 19, 2018
As far as view materialization routine uses table name only,
replace struct Table param w/ (table) name of type const char*.
Rewrite using Linux coding style.
Also, remove dead function from box/sql/delete.c

Part of #3235
kyukhin added a commit that referenced this issue Apr 27, 2018
Legacy SQL DD structs contained sort_order, defined per
index column. During integration, those structs are to be
vanished. So, introduce new field to part entity of Tarantool.
This field states for sorting order of given part in give index.
This field is ignored by Tarantool everywhere excpept for
some of nested queries in SQL.

Patch also replaces usages of SQL's stored sort order w/ this new
field.

Part of #3235
kyukhin added a commit that referenced this issue May 8, 2018
KeyInfo is a legacy struct which was heavily used in SQL
front-end. This patch replaces all its usages w/ Tarantool's
natural key description structure called key_def.

This change is a prt of data dictionary intagration effort:
Tarantool indexes don't aware of KeyInfo, that is why it was
evicted.

Legacy KeyInfo memory handling was ref-counting based and now
is replaced w/ memory duplication. This state of affairs should
be improved in future.

Part of #3235
kyukhin added a commit that referenced this issue May 10, 2018
Legacy SQL DD structs contained sort_order, defined per
index column. During integration, those structs are to be
vanished. So, introduce new field to part entity of Tarantool.
This field states for sorting order of given part in give index.
This field is ignored by Tarantool everywhere excpept for
some of nested queries in SQL.

Patch also replaces usages of SQL's stored sort order w/ this new
field.

Part of #3235
kyukhin added a commit that referenced this issue May 10, 2018
KeyInfo is a legacy struct which was heavily used in SQL
front-end. This patch replaces all its usages w/ Tarantool's
natural key description structure called key_def.

This change is a part of data dictionary integration effort:
Tarantool indexes don't aware of KeyInfo, that is why it was
evicted.

Legacy KeyInfo memory handling was ref-counting based and now
is replaced w/ memory duplication. This state of affairs should
be improved in future.

Part of #3235
kyukhin added a commit that referenced this issue May 11, 2018
Legacy SQL DD structs contained sort_order, defined per
index column. During integration, those structs are to be
vanished. So, introduce new field to part entity of Tarantool.
This field states for sorting order of given part in give index.
This field is ignored by Tarantool everywhere excpept for
some of nested queries in SQL.

Patch also replaces usages of SQL's stored sort order w/ this new
field.

Part of #3235
kyukhin added a commit that referenced this issue May 11, 2018
KeyInfo is a legacy struct which was heavily used in SQL
front-end. This patch replaces all its usages w/ Tarantool's
natural key description structure called key_def.

This change is a part of data dictionary integration effort:
Tarantool indexes don't aware of KeyInfo, that is why it was
evicted.

Legacy KeyInfo memory handling was ref-counting based and now
is replaced w/ memory duplication. This state of affairs should
be improved in future.

Part of #3235
kyukhin added a commit that referenced this issue May 14, 2018
Legacy SQL DD structs contained sort_order, defined per
index column. During integration, those structs are to be
vanished. So, introduce new field to part entity of Tarantool.
This field states for sorting order of given part in give index.
This field is ignored by Tarantool everywhere excpept for
some of nested queries in SQL.

Patch also replaces usages of SQL's stored sort order w/ this new
field.

Part of #3235
kyukhin added a commit that referenced this issue May 14, 2018
KeyInfo is a legacy struct which was heavily used in SQL
front-end. This patch replaces all its usages w/ Tarantool's
natural key description structure called key_def.

This change is a part of data dictionary integration effort:
Tarantool indexes don't aware of KeyInfo, that is why it was
evicted.

Legacy KeyInfo memory handling was ref-counting based and now
is replaced w/ memory duplication. This state of affairs should
be improved in future.

Part of #3235
kyukhin added a commit that referenced this issue May 16, 2018
There're many cases in SQL FE, where exact key def passed to
doesn't ephemeral table matter. It is used to store data only.
Only field count makes sense in such a case. Hoewever it is
passed separately (in P2). So, allow P4 field to be NULL for
OP_OpenTEphemeral. Update delte from routine from sql/delete.c
accordingly.

Part of #3235
kyukhin added a commit that referenced this issue May 16, 2018
Refactor DELETE FROM statements translation, update
all live routines according to Tarantool's coding style.
Use key_def instead of Table* where possible.
Remove useless index delete routine as well.

Part of #3235
kyukhin added a commit that referenced this issue May 16, 2018
Refactor DELETE FROM statements translation, update
all live routines according to Tarantool's coding style.
Use key_def instead of Table* where possible.
Remove useless index delete routine as well.

Part of #3235
kyukhin added a commit that referenced this issue May 17, 2018
There're many cases in SQL FE, where exact key def passed to
doesn't ephemeral table matter. It is used to store data only.
Only field count makes sense in such a case. Hoewever it is
passed separately (in P2). So, allow P4 field to be NULL for
OP_OpenTEphemeral. Update delte from routine from sql/delete.c
accordingly.

Part of #3235
kyukhin added a commit that referenced this issue May 31, 2018
Legacy SQL FE was able to create indexes w/ expressions.
Tarantool will employ diofferenct scheme to implement
functional indexes, thus code  handling it in SQL FE is
dead and redundant. Remove it.

Part of #3235
kyukhin added a commit that referenced this issue May 31, 2018
This patch implements support of SQL's DELETE statemets
which a accompanied by point WHERE point constraints.
This patch doesn't support any kinds of nested selects
or JOINs.

Part of #3235
kyukhin added a commit that referenced this issue Jun 1, 2018
This small patch removes usages of primary index throughout
code sql_table_delete_from, limiting use to fetching of
affinities only. We cannot use space_def here, since primary
index might contain calculated columns.

Part of #3235
kyukhin added a commit that referenced this issue Jun 1, 2018
Legacy SQL FE was able to create indexes w/ expressions.
Tarantool will employ diofferenct scheme to implement
functional indexes, thus code  handling it in SQL FE is
dead and redundant. Remove it.

Part of #3235
kyukhin added a commit that referenced this issue Jun 1, 2018
This patch implements support of SQL's DELETE statemets
which a accompanied by point WHERE point constraints.
This patch doesn't support any kinds of nested selects
or JOINs.

Part of #3235
Gerold103 pushed a commit that referenced this issue Jun 1, 2018
This patch implements support of SQL's DELETE statemets
which a accompanied by point WHERE point constraints.
This patch doesn't support any kinds of nested selects
or JOINs.

Part of #3235
kyukhin added a commit that referenced this issue Jun 8, 2018
This small patch removes usages of primary index throughout
code sql_table_delete_from, limiting use to fetching of
affinities only. We cannot use space_def here, since primary
index might contain calculated columns.

Part of #3235
kyukhin added a commit that referenced this issue Jun 8, 2018
Legacy SQL FE was able to create indexes w/ expressions.
Tarantool will employ diofferenct scheme to implement
functional indexes, thus code  handling it in SQL FE is
dead and redundant. Remove it.

Part of #3235
kyukhin added a commit that referenced this issue Jun 8, 2018
This patch implements support of SQL's DELETE statemets
which a accompanied by point WHERE point constraints.
This patch doesn't support any kinds of nested selects
or JOINs.

Part of #3235
kyukhin added a commit that referenced this issue Jun 14, 2018
Legacy SQL FE was able to create indexes w/ expressions.
Tarantool will employ diofferenct scheme to implement
functional indexes, thus code  handling it in SQL FE is
dead and redundant. Remove it.

Part of #3235
kyukhin added a commit that referenced this issue Jun 14, 2018
This patch implements support of SQL's DELETE statemets
which a accompanied by point WHERE point constraints.
This patch doesn't support any kinds of nested selects
or JOINs.

Part of #3235
@Korablev77
Copy link
Contributor

After all, the only artefact remained is struct Table, struct Index and struct Column don't exist anymore. Despite removing table hash, struct Table is still used as a wrapper around struct space and appears for example in struct SrcList.

@kyukhin kyukhin modified the milestones: 2.1.0, 2.1.1 Sep 20, 2018
@kyukhin kyukhin added the feature A new functionality label Nov 26, 2018
@Korablev77 Korablev77 assigned SudoBobo and unassigned kyukhin Jan 15, 2019
SudoBobo added a commit that referenced this issue Jan 25, 2019
Completely removes struct Table. Also the
patch simplifies memory management as in
many cases struct space (which replaces
struct Table) is allocated on region
and shouldn't be explicitly freed.

Closes #3235
SudoBobo added a commit that referenced this issue Jan 25, 2019
Completely removes struct Table. Also the
patch simplifies memory management as in
many cases struct space (which replaces
struct Table) is allocated on region
and shouldn't be explicitly freed.

Closes #3235
SudoBobo added a commit that referenced this issue Jan 25, 2019
Completely removes struct Table. Also the
patch simplifies memory management as in
many cases struct space (which replaces
struct Table) is allocated on region
and shouldn't be explicitly freed.

Closes #3235
SudoBobo added a commit that referenced this issue Jan 27, 2019
Completely removes struct Table. Also the
patch simplifies memory management as in
many cases struct space (which replaces
struct Table) is allocated on region
and shouldn't be explicitly freed.

Closes #3235
SudoBobo added a commit that referenced this issue Feb 1, 2019
completely removes struct Table. Also the
patch simplifies memory management as in
many cases struct space (which replaces
struct Table) is allocated on region and
shouldn't be explicitly freed.

Closes #3235
@kyukhin kyukhin added the refactoring Code refactoring label Feb 11, 2019
Korablev77 pushed a commit that referenced this issue Feb 11, 2019
Lets completely remove struct Table. Also the patch simplifies memory
management as in many cases struct space (which replaces struct Table)
is allocated on region and shouldn't be explicitly freed.  Some wrappers
fetching data from space (such as space_checks_expr_list) have been
removed since now we can get all properties right from space object
right from cache.

Closes #3235
SudoBobo added a commit that referenced this issue Feb 12, 2019
Lets completely remove struct Table. Also the patch simplifies memory
management as in many cases struct space (which replaces struct Table)
is allocated on region and shouldn't be explicitly freed.  Some wrappers
fetching data from space (such as space_checks_expr_list) have been
removed since now we can get all properties right from space object
right from cache.

Closes #3235
SudoBobo added a commit that referenced this issue Feb 12, 2019
Lets completely remove struct Table. Also the patch simplifies memory
management as in many cases struct space (which replaces struct Table)
is allocated on region and shouldn't be explicitly freed.  Some wrappers
fetching data from space (such as space_checks_expr_list) have been
removed since now we can get all properties right from space object
right from cache.

Closes #3235
SudoBobo added a commit that referenced this issue Feb 15, 2019
Lets completely remove struct Table. Also the patch simplifies memory
management as in many cases struct space (which replaces struct Table)
is allocated on region and shouldn't be explicitly freed.  Some wrappers
fetching data from space (such as space_checks_expr_list) have been
removed since now we can get all properties right from space object
right from cache.

Closes #3235
SudoBobo added a commit that referenced this issue Feb 15, 2019
Lets completely remove struct Table. Also the patch simplifies memory
management as in many cases struct space (which replaces struct Table)
is allocated on region and shouldn't be explicitly freed.  Some wrappers
fetching data from space (such as space_checks_expr_list) have been
removed since now we can get all properties right from space object
right from cache.

Closes #3235
@kyukhin kyukhin added the tmp label Mar 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature A new functionality refactoring Code refactoring sql
Projects
None yet
Development

No branches or pull requests

4 participants