Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

planner, executor: refine ColumnPrune for LogicalUnionAll #7930

Merged
merged 5 commits into from
Oct 18, 2018
Merged

planner, executor: refine ColumnPrune for LogicalUnionAll #7930

merged 5 commits into from
Oct 18, 2018

Conversation

XuHuaiyu
Copy link
Contributor

@XuHuaiyu XuHuaiyu commented Oct 17, 2018 •
edited
Loading

What problem does this PR solve? 

create table t1(a bigint, b bigint);
create table t2(a bigint, b bigint);
insert into t1 values(1, 1);
insert into t1 select * from t1; -- repeat this sql 10 times
insert into t2 values(1, 1);

set @@tidb_max_chunk_size=2;
select 1, 1 from (select t1.a, t1.b from t1 left join t2 on t1.a=t2.a union all select t1.a, t1.a from t1 left join t2 on t1.a=t2.a) tmp;  -- panic

What is changed and how it works?

Schema of the parent node upon Union is empty, so after ColumnPruning,
Union.Schema will be empty.
When calling UnionExec.Next(), we'll use chk.SwapColumns() to swap the chunk of UnionExec.newFirstChunk()(whose column length is 0) with UnionExec.Child.newFirstChunk()(whose column length is 2). And this will cause the panic.

Check List

Tests

  • Integration test

Code changes

  • Has exported function/method change

Side effects

Related changes

  • Need to cherry-pick to the release branch

@XuHuaiyu XuHuaiyu added priority/P1 The issue has P1 priority. type/bugfix This PR fixes a bug. sig/execution SIG execution labels Oct 17, 2018
@XuHuaiyu
Copy link
Contributor Author

/run-all-tests

@XuHuaiyu XuHuaiyu changed the title executor: refine UnionExec.newFirstChunk planner, executor: refine ColumnPrune for LogicalUnionAll Oct 17, 2018
winoros
winoros reviewed Oct 17, 2018
View reviewed changes
Copy link
Member

@winoros winoros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
you can add a TODO here or open an issue to record the remained problem.

zz-jason
zz-jason reviewed Oct 17, 2018
View reviewed changes
planner/core/rule_column_pruning.go Outdated
@@ -129,6 +129,13 @@ func (ls *LogicalSort) PruneColumns(parentUsedCols []*expression.Column) {

// PruneColumns implements LogicalPlan interface.
func (p *LogicalUnionAll) PruneColumns(parentUsedCols []*expression.Column) {
used := getUsedList(parentUsedCols, p.schema)
for i := range used {
if used[i] {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why append the unused columns into parentUsedCols?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The used bool slice indicates whether the p.schema is used in parentUsedCols,
so we only need to append the unused columns. @zz-jason

eurekaka
eurekaka approved these changes Oct 18, 2018
View reviewed changes
Copy link
Contributor

@eurekaka eurekaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eurekaka eurekaka added the status/LGT2 Indicates that a PR has LGTM 2. label Oct 18, 2018
@eurekaka
Copy link
Contributor

/run-all-tests

@XuHuaiyu XuHuaiyu merged commit de931ca into pingcap:master Oct 18, 2018
@XuHuaiyu XuHuaiyu deleted the union_schema branch October 18, 2018 05:30
XuHuaiyu added a commit to XuHuaiyu/tidb that referenced this pull request Oct 18, 2018
@XuHuaiyu
planner, executor: refine ColumnPrune for LogicalUnionAll (pingcap#7930)
5946e69
@XuHuaiyu XuHuaiyu mentioned this pull request Oct 18, 2018
Merged
XuHuaiyu added a commit to XuHuaiyu/tidb that referenced this pull request Oct 18, 2018
@XuHuaiyu
planner, executor: refine ColumnPrune for LogicalUnionAll (pingcap#7930)
b314ac9
@XuHuaiyu XuHuaiyu mentioned this pull request Oct 18, 2018
Merged
zz-jason pushed a commit that referenced this pull request Oct 18, 2018
@XuHuaiyu @zz-jason
planner, executor: refine ColumnPrune for LogicalUnionAll (#7930) (#7942
6c50eba 
)
ngaut pushed a commit that referenced this pull request Oct 18, 2018
@XuHuaiyu @ngaut
planner, executor: refine ColumnPrune for LogicalUnionAll (#7930) (#7941
0f587c3 
)
iamzhoug37 pushed a commit to iamzhoug37/tidb that referenced this pull request Oct 25, 2018
@iamzhoug37
fix issue 7518
f15bb5c 
bugfix fixed pingcap#7518

expression: MySQL compatible current_user function (pingcap#7801)

plan: propagate constant over outer join (pingcap#7794)

- extract `outerCol = const` from join conditions and filter conditions,
  substitute `outerCol` in join conditions with `const`;
- extract `outerCol = innerCol` from join conditions, derive new join
  conditions based on this column equal condition and `outerCol` related
  expressions in join conditions and filter conditions;

util/timeutil: fix data race caused by forgetting set stats lease to 0 (pingcap#7901)

stats: handle ddl event for partition table (pingcap#7903)

plan: implement Operand and Pattern of cascades planner. (pingcap#7910)

planner: not convert to TableDual if empty range is derived from deferred constants (pingcap#7808)

plan: move projEliminate behind aggEliminate (pingcap#7909)

admin: fix admin check table bug of byte compare (pingcap#7887)

* admin: remove reflect deepEqual

stats: fix panic caused by empty histogram (pingcap#7912)

plan: fix panic caused by empty schema of LogicalTableDual (pingcap#7906)

* fix drop view if exist error (pingcap#7833)

executor: refine `explain analyze` (pingcap#7888)

executor: add an variable to compatible with MySQL insert for OGG (pingcap#7863)

expression: maintain `DeferredExpr` in aggressive constant folding. (pingcap#7915)

stats: fix histogram boundaries overflow error (pingcap#7883)

ddl:support the definition of `null` change to `not null` using `alter table` (pingcap#7771)

* ddl:support the definition of null change to not null using alter table

ddl: add check when create table with foreign key. (pingcap#7885)

* ddl: add check when create table with foreign key

planner: eliminate if null on non null column (pingcap#7924)

executor: fix a bug in point get (pingcap#7934)

planner, executor: refine ColumnPrune for LogicalUnionAll (pingcap#7930)

executor: fix panic when limit is too large (pingcap#7936)

ddl: add TiDB version to metrics (pingcap#7902)

stats: limit the length of sample values (pingcap#7931)

vendor: update tipb (pingcap#7893)

planner: support the Group and GroupExpr for the cascades planner (pingcap#7917)

store/tikv: log more information when other err occurs (pingcap#7948)

types: fix date time parse (pingcap#7933)

ddl: just print error message when ddl job is normal to calcel, to eliminate noisy log (pingcap#7875)

stats: update delta info for partition table (pingcap#7947)

explaintest: add explain test for partition pruning (pingcap#7505)

util: move disjoint set to util package (pingcap#7950)

util: add PreAlloc4Row and Insert for Chunk and List (pingcap#7916)

executor: add the slow log for commit (pingcap#7951)

expression: add builtin json_keys (pingcap#7776)

privilege: add USAGE in `show grants` for mysql compatibility (pingcap#7955)

ddl: fix invailid ddl job panic (pingcap#7940)

*: move ast.NewValueExpr to standalone parser_driver package (pingcap#7952)

Make the ast package get rid of the dependency of types.Datum

server: allow cors http request (pingcap#7939)

*: move `Statement` and `RecordSet` from ast to sqlexec package (pingcap#7970)

pr suggestion update

executor/aggfuncs: split unit tests to corresponding file (pingcap#7993)

store/tikv: fix typo (pingcap#7990)

executor, planner: clone proj schema for different children in buildProj4Union (pingcap#7999)

executor: let information_schema be the first database in ShowDatabases (pingcap#7938)

stats: use local feedback for partition table (pingcap#7963)

executor: add unit test for aggfuncs (pingcap#7966)

server: add log for binary execute statement (pingcap#7987)

admin: refine admin check decoder (pingcap#7862)

executor: improve wide table insert & update performance (pingcap#7935)

ddl: fix reassigned partition id in `truncate table` does not take effect (pingcap#7919)

fix reassigned partition id in truncate table does not take effect

add changelog for 2.1.0 rc4 (pingcap#8020)

*: make parser package dependency as small as possible (pingcap#7989)

parser: support `:=` in the `set` syntax (pingcap#8018)

According to MySQL document, `set` use the = assignment operator,
but the := assignment operator is also permitted

stats: garbage collect stats for partition table (pingcap#7962)

docs: add the proposal for the column pool (pingcap#7988)

expression: refine built-in func truncate to support uint arg (pingcap#8000)

stats: support show stats for partition table (pingcap#8023)

stats: update error rate for partition table (pingcap#8022)

stats: fix estimation for out of range point queries (pingcap#8015)

*: move parser to a separate repository (pingcap#8036)

executor: fix wrong result when index join on union scan. (pingcap#8031)

Do not modify Plan of dataReaderBuilder directly, because it would
impact next batch of outer rows, as well as other concurrent inner
workers. Instead, build a local child builder to store the child plan.

planner: fix a panic of a cached prepared statement with IndexScan (pingcap#8017)

*: fix the issue of executing DDL after executing SQL failure in txn (pingcap#8044)

* ddl, executor: fix the issue of executing DDL after executing SQL failure in txn

add unit test

remove debug info

add like evaluator case sensitive test

ddl, domain: make schema correct after canceling jobs (pingcap#7997)

unit test fix

code format

proposal: maintaining histograms in plan. (pingcap#7605)

support _tidb_rowid for table scan range (pingcap#8047)

var rename fix
XuHuaiyu added a commit to XuHuaiyu/tidb that referenced this pull request Oct 29, 2018
@XuHuaiyu
planner, executor: refine ColumnPrune for LogicalUnionAll (pingcap#7930)
8213bff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zz-jason zz-jason zz-jason left review comments

@winoros winoros winoros left review comments

@eurekaka eurekaka eurekaka approved these changes

Assignees
No one assigned
Labels
priority/P1 The issue has P1 priority. sig/execution SIG execution status/LGT2 Indicates that a PR has LGTM 2. type/bugfix This PR fixes a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@XuHuaiyu @eurekaka @zz-jason @winoros