Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

planner: support index_lookup_merge_join in physical plan. #11338

Merged
merged 24 commits into from Sep 4, 2019
Merged

planner: support index_lookup_merge_join in physical plan. #11338

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
d647ff5
support index_lookup_merge_join in physical plan
lzmhhh123 Jul 19, 2019
3b69d4b
Merge branch 'master' into dev/support_index_merge_join_in_physical_plan
lzmhhh123 Jul 31, 2019
535f944
fix
lzmhhh123 Jul 31, 2019
38a0de2
fix explain test
lzmhhh123 Aug 1, 2019
d68dcb0
address comments
lzmhhh123 Aug 1, 2019
05b6b5b
add comments for NeedOuterSort, CompareFuncs and OuterCompareFuncs
lzmhhh123 Aug 1, 2019
8ed2c1d
Merge remote-tracking branch 'origin/master' into dev/support_index_m…
lzmhhh123 Aug 8, 2019
4465c5b
fix a little
lzmhhh123 Aug 8, 2019
976913c
if some columns need keep order. No need to construct index join
lzmhhh123 Aug 12, 2019
5e41112
fix error and bug in cost model
lzmhhh123 Aug 12, 2019
a22974c
address comment
lzmhhh123 Aug 12, 2019
09494b0
Merge branch 'master' into dev/support_index_merge_join_in_physical_plan
lzmhhh123 Aug 12, 2019
cdfec19
Merge remote-tracking branch 'origin/master' into dev/support_index_m…
lzmhhh123 Aug 21, 2019
57c411d
fix test
lzmhhh123 Aug 21, 2019
b2387dd
address comments
lzmhhh123 Aug 22, 2019
a94acf7
Merge branch 'master' into dev/support_index_merge_join_in_physical_plan
lzmhhh123 Aug 22, 2019
55bd051
commit suggestion
lzmhhh123 Aug 23, 2019
f8c5f9a
commit suggestion
lzmhhh123 Aug 23, 2019
029cc86
commit suggestion
lzmhhh123 Aug 23, 2019
0ad1387
fix
lzmhhh123 Aug 30, 2019
5b5200b
Merge remote-tracking branch 'origin/master' into dev/support_index_m…
lzmhhh123 Aug 30, 2019
45dacec
Merge remote-tracking branch 'origin/master' into dev/support_index_m…
lzmhhh123 Sep 3, 2019
3d7dc49
rm log.txt
lzmhhh123 Sep 4, 2019
3927127
Merge branch 'master' into dev/support_index_merge_join_in_physical_plan
lzmhhh123 Sep 4, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
48 changes: 24 additions & 24 deletions cmd/explaintest/r/explain_complex.result
View file
Open in desktop
Expand Up @@ -118,24 +118,24 @@ id count task operator info
Projection_13 1.00 root test.gad.id, test.dd.id, test.gad.aid, test.gad.cm, test.dd.dic, test.dd.ip, test.dd.t, test.gad.p1, test.gad.p2, test.gad.p3, test.gad.p4, test.gad.p5, test.gad.p6_md5, test.gad.p7_md5, test.gad.ext, test.gad.t
└─Limit_16 1.00 root offset:0, count:2500
└─HashAgg_19 1.00 root group by:test.dd.dic, test.gad.aid, funcs:firstrow(test.gad.id), firstrow(test.gad.aid), firstrow(test.gad.cm), firstrow(test.gad.p1), firstrow(test.gad.p2), firstrow(test.gad.p3), firstrow(test.gad.p4), firstrow(test.gad.p5), firstrow(test.gad.p6_md5), firstrow(test.gad.p7_md5), firstrow(test.gad.ext), firstrow(test.gad.t), firstrow(test.dd.id), firstrow(test.dd.dic), firstrow(test.dd.ip), firstrow(test.dd.t)
└─IndexJoin_24 0.00 root inner join, inner:IndexLookUp_23, outer key:test.gad.aid, inner key:test.dd.aid, other cond:eq(test.dd.ip, test.gad.ip), gt(test.dd.t, test.gad.t)
├─IndexLookUp_23 0.00 root
│ ├─IndexScan_20 10.00 cop table:dd, index:aid, dic, range: decided by [eq(test.dd.aid, test.gad.aid)], keep order:false, stats:pseudo
│ └─Selection_22 0.00 cop eq(test.dd.bm, 0), eq(test.dd.pt, "android"), gt(test.dd.t, 1478143908), not(isnull(test.dd.ip)), not(isnull(test.dd.t))
│ └─TableScan_21 10.00 cop table:dd, keep order:false, stats:pseudo
└─IndexLookUp_33 3.33 root
├─IndexScan_30 3333.33 cop table:gad, index:t, range:(1478143908,+inf], keep order:false, stats:pseudo
└─Selection_32 3.33 cop eq(test.gad.pt, "android"), not(isnull(test.gad.ip))
└─TableScan_31 3333.33 cop table:st, keep order:false, stats:pseudo
└─IndexMergeJoin_30 0.00 root inner join, inner:IndexLookUp_28, outer key:test.gad.aid, inner key:test.dd.aid, other cond:eq(test.dd.ip, test.gad.ip), gt(test.dd.t, test.gad.t)
├─IndexLookUp_28 0.00 root
│ ├─IndexScan_25 10.00 cop table:dd, index:aid, dic, range: decided by [eq(test.dd.aid, test.gad.aid)], keep order:true, stats:pseudo
│ └─Selection_27 0.00 cop eq(test.dd.bm, 0), eq(test.dd.pt, "android"), gt(test.dd.t, 1478143908), not(isnull(test.dd.ip)), not(isnull(test.dd.t))
│ └─TableScan_26 10.00 cop table:dd, keep order:true, stats:pseudo
└─IndexLookUp_39 3.33 root
├─IndexScan_36 3333.33 cop table:gad, index:t, range:(1478143908,+inf], keep order:false, stats:pseudo
└─Selection_38 3.33 cop eq(test.gad.pt, "android"), not(isnull(test.gad.ip))
└─TableScan_37 3333.33 cop table:st, keep order:false, stats:pseudo
explain select gad.id as gid,sdk.id as sid,gad.aid as aid,gad.cm as cm,sdk.dic as dic,sdk.ip as ip, sdk.t as t, gad.p1 as p1, gad.p2 as p2, gad.p3 as p3, gad.p4 as p4, gad.p5 as p5, gad.p6_md5 as p6, gad.p7_md5 as p7, gad.ext as ext from st gad join dd sdk on gad.aid = sdk.aid and gad.dic = sdk.mac and gad.t < sdk.t where gad.t > 1477971479 and gad.bm = 0 and gad.pt = 'ios' and gad.dit = 'mac' and sdk.t > 1477971479 and sdk.bm = 0 and sdk.pt = 'ios' limit 3000;
id count task operator info
Projection_10 0.00 root test.gad.id, test.sdk.id, test.gad.aid, test.gad.cm, test.sdk.dic, test.sdk.ip, test.sdk.t, test.gad.p1, test.gad.p2, test.gad.p3, test.gad.p4, test.gad.p5, test.gad.p6_md5, test.gad.p7_md5, test.gad.ext
└─Limit_13 0.00 root offset:0, count:3000
└─IndexJoin_18 0.00 root inner join, inner:IndexLookUp_17, outer key:test.gad.aid, inner key:test.sdk.aid, other cond:eq(test.gad.dic, test.sdk.mac), lt(test.gad.t, test.sdk.t)
├─IndexLookUp_27 0.00 root
│ ├─IndexScan_24 3333.33 cop table:gad, index:t, range:(1477971479,+inf], keep order:false, stats:pseudo
│ └─Selection_26 0.00 cop eq(test.gad.bm, 0), eq(test.gad.dit, "mac"), eq(test.gad.pt, "ios"), not(isnull(test.gad.dic))
│ └─TableScan_25 3333.33 cop table:st, keep order:false, stats:pseudo
├─IndexLookUp_33 0.00 root
│ ├─IndexScan_30 3333.33 cop table:gad, index:t, range:(1477971479,+inf], keep order:false, stats:pseudo
│ └─Selection_32 0.00 cop eq(test.gad.bm, 0), eq(test.gad.dit, "mac"), eq(test.gad.pt, "ios"), not(isnull(test.gad.dic))
│ └─TableScan_31 3333.33 cop table:st, keep order:false, stats:pseudo
└─IndexLookUp_17 0.00 root
├─IndexScan_14 10.00 cop table:sdk, index:aid, dic, range: decided by [eq(test.sdk.aid, test.gad.aid)], keep order:false, stats:pseudo
└─Selection_16 0.00 cop eq(test.sdk.bm, 0), eq(test.sdk.pt, "ios"), gt(test.sdk.t, 1477971479), not(isnull(test.sdk.mac)), not(isnull(test.sdk.t))
Expand All @@ -153,9 +153,9 @@ id count task operator info
Projection_10 0.00 root test.dt.id, test.dt.aid, test.dt.pt, test.dt.dic, test.dt.cm, test.rr.gid, test.rr.acd, test.rr.t, test.dt.p1, test.dt.p2, test.dt.p3, test.dt.p4, test.dt.p5, test.dt.p6_md5, test.dt.p7_md5
└─Limit_13 0.00 root offset:0, count:2000
└─IndexJoin_19 0.00 root inner join, inner:IndexLookUp_18, outer key:test.dt.aid, test.dt.dic, inner key:test.rr.aid, test.rr.dic
├─TableReader_41 0.00 root data:Selection_40
│ └─Selection_40 0.00 cop eq(test.dt.bm, 0), eq(test.dt.pt, "ios"), gt(test.dt.t, 1478185592), not(isnull(test.dt.dic))
│ └─TableScan_39 10000.00 cop table:dt, range:[0,+inf], keep order:false, stats:pseudo
├─TableReader_54 0.00 root data:Selection_53
│ └─Selection_53 0.00 cop eq(test.dt.bm, 0), eq(test.dt.pt, "ios"), gt(test.dt.t, 1478185592), not(isnull(test.dt.dic))
│ └─TableScan_52 10000.00 cop table:dt, range:[0,+inf], keep order:false, stats:pseudo
└─IndexLookUp_18 0.00 root
├─IndexScan_15 10.00 cop table:rr, index:aid, dic, range: decided by [eq(test.rr.aid, test.dt.aid) eq(test.rr.dic, test.dt.dic)], keep order:false, stats:pseudo
└─Selection_17 0.00 cop eq(test.rr.pt, "ios"), gt(test.rr.t, 1478185592)
Expand Down Expand Up @@ -245,17 +245,17 @@ id count task operator info
Sort_10 1.00 root test.d.left_value:asc
└─HashAgg_15 1.00 root group by:test.d.id, funcs:firstrow(test.d.id), firstrow(test.d.ctx), firstrow(test.d.name), firstrow(test.d.left_value), firstrow(test.d.right_value), firstrow(test.d.depth), firstrow(test.d.leader_id), firstrow(test.d.status), firstrow(test.d.created_on), firstrow(test.d.updated_on)
└─Selection_22 0.01 root or(eq(test.ep.user_id, 62), or(eq(test.d.id, 20), eq(test.d.id, 20)))
└─HashLeftJoin_23 0.02 root left outer join, inner:TableReader_58, equal:[eq(test.p.id, test.ep.position_id)]
└─HashLeftJoin_23 0.02 root left outer join, inner:TableReader_65, equal:[eq(test.p.id, test.ep.position_id)]
├─IndexJoin_32 0.01 root left outer join, inner:IndexLookUp_31, outer key:test.d.id, inner key:test.p.department_id
│ ├─IndexLookUp_48 0.01 root
│ │ ├─IndexScan_45 10.00 cop table:d, index:ctx, range:[1,1], keep order:false, stats:pseudo
│ │ └─Selection_47 0.01 cop eq(test.d.status, 1000)
│ │ └─TableScan_46 10.00 cop table:org_department, keep order:false, stats:pseudo
│ ├─IndexLookUp_55 0.01 root
│ │ ├─IndexScan_52 10.00 cop table:d, index:ctx, range:[1,1], keep order:false, stats:pseudo
│ │ └─Selection_54 0.01 cop eq(test.d.status, 1000)
│ │ └─TableScan_53 10.00 cop table:org_department, keep order:false, stats:pseudo
│ └─IndexLookUp_31 0.01 root
│ ├─Selection_29 9.99 cop not(isnull(test.p.department_id))
│ │ └─IndexScan_27 10.00 cop table:p, index:department_id, range: decided by [eq(test.p.department_id, test.d.id)], keep order:false, stats:pseudo
│ └─Selection_30 0.01 cop eq(test.p.status, 1000)
│ └─TableScan_28 9.99 cop table:org_position, keep order:false, stats:pseudo
└─TableReader_58 9.99 root data:Selection_57
└─Selection_57 9.99 cop eq(test.ep.status, 1000), not(isnull(test.ep.position_id))
└─TableScan_56 10000.00 cop table:ep, range:[-inf,+inf], keep order:false, stats:pseudo
└─TableReader_65 9.99 root data:Selection_64
└─Selection_64 9.99 cop eq(test.ep.status, 1000), not(isnull(test.ep.position_id))
└─TableScan_63 10000.00 cop table:ep, range:[-inf,+inf], keep order:false, stats:pseudo
48 changes: 24 additions & 24 deletions cmd/explaintest/r/explain_complex_stats.result
View file
Open in desktop
Expand Up @@ -128,26 +128,26 @@ id count task operator info
Projection_13 424.00 root test.gad.id, test.dd.id, test.gad.aid, test.gad.cm, test.dd.dic, test.dd.ip, test.dd.t, test.gad.p1, test.gad.p2, test.gad.p3, test.gad.p4, test.gad.p5, test.gad.p6_md5, test.gad.p7_md5, test.gad.ext, test.gad.t
└─Limit_16 424.00 root offset:0, count:2500
└─HashAgg_19 424.00 root group by:test.dd.dic, test.gad.aid, funcs:firstrow(test.gad.id), firstrow(test.gad.aid), firstrow(test.gad.cm), firstrow(test.gad.p1), firstrow(test.gad.p2), firstrow(test.gad.p3), firstrow(test.gad.p4), firstrow(test.gad.p5), firstrow(test.gad.p6_md5), firstrow(test.gad.p7_md5), firstrow(test.gad.ext), firstrow(test.gad.t), firstrow(test.dd.id), firstrow(test.dd.dic), firstrow(test.dd.ip), firstrow(test.dd.t)
└─IndexJoin_24 424.00 root inner join, inner:IndexLookUp_23, outer key:test.gad.aid, inner key:test.dd.aid, other cond:eq(test.gad.ip, test.dd.ip), gt(test.dd.t, test.gad.t)
├─TableReader_29 424.00 root data:Selection_28
│ └─Selection_28 424.00 cop eq(test.gad.bm, 0), eq(test.gad.pt, "android"), gt(test.gad.t, 1478143908), not(isnull(test.gad.ip))
│ └─TableScan_27 1999.00 cop table:gad, range:[0,+inf], keep order:false
└─IndexLookUp_23 0.23 root
├─IndexScan_20 1.00 cop table:dd, index:aid, dic, range: decided by [eq(test.dd.aid, test.gad.aid)], keep order:false
└─Selection_22 0.23 cop eq(test.dd.bm, 0), eq(test.dd.pt, "android"), gt(test.dd.t, 1478143908), not(isnull(test.dd.ip)), not(isnull(test.dd.t))
└─TableScan_21 1.00 cop table:dd, keep order:false
└─IndexMergeJoin_30 424.00 root inner join, inner:IndexLookUp_28, outer key:test.gad.aid, inner key:test.dd.aid, other cond:eq(test.gad.ip, test.dd.ip), gt(test.dd.t, test.gad.t)
├─TableReader_35 424.00 root data:Selection_34
│ └─Selection_34 424.00 cop eq(test.gad.bm, 0), eq(test.gad.pt, "android"), gt(test.gad.t, 1478143908), not(isnull(test.gad.ip))
│ └─TableScan_33 1999.00 cop table:gad, range:[0,+inf], keep order:false
└─IndexLookUp_28 0.23 root
├─IndexScan_25 1.00 cop table:dd, index:aid, dic, range: decided by [eq(test.dd.aid, test.gad.aid)], keep order:true
└─Selection_27 0.23 cop eq(test.dd.bm, 0), eq(test.dd.pt, "android"), gt(test.dd.t, 1478143908), not(isnull(test.dd.ip)), not(isnull(test.dd.t))
└─TableScan_26 1.00 cop table:dd, keep order:true
explain select gad.id as gid,sdk.id as sid,gad.aid as aid,gad.cm as cm,sdk.dic as dic,sdk.ip as ip, sdk.t as t, gad.p1 as p1, gad.p2 as p2, gad.p3 as p3, gad.p4 as p4, gad.p5 as p5, gad.p6_md5 as p6, gad.p7_md5 as p7, gad.ext as ext from st gad join dd sdk on gad.aid = sdk.aid and gad.dic = sdk.mac and gad.t < sdk.t where gad.t > 1477971479 and gad.bm = 0 and gad.pt = 'ios' and gad.dit = 'mac' and sdk.t > 1477971479 and sdk.bm = 0 and sdk.pt = 'ios' limit 3000;
id count task operator info
Projection_10 170.34 root test.gad.id, test.sdk.id, test.gad.aid, test.gad.cm, test.sdk.dic, test.sdk.ip, test.sdk.t, test.gad.p1, test.gad.p2, test.gad.p3, test.gad.p4, test.gad.p5, test.gad.p6_md5, test.gad.p7_md5, test.gad.ext
└─Limit_13 170.34 root offset:0, count:3000
└─IndexJoin_18 170.34 root inner join, inner:IndexLookUp_17, outer key:test.gad.aid, inner key:test.sdk.aid, other cond:eq(test.gad.dic, test.sdk.mac), lt(test.gad.t, test.sdk.t)
├─TableReader_23 170.34 root data:Selection_22
│ └─Selection_22 170.34 cop eq(test.gad.bm, 0), eq(test.gad.dit, "mac"), eq(test.gad.pt, "ios"), gt(test.gad.t, 1477971479), not(isnull(test.gad.dic))
│ └─TableScan_21 1999.00 cop table:gad, range:[0,+inf], keep order:false
└─IndexLookUp_17 0.25 root
├─IndexScan_14 1.00 cop table:sdk, index:aid, dic, range: decided by [eq(test.sdk.aid, test.gad.aid)], keep order:false
└─Selection_16 0.25 cop eq(test.sdk.bm, 0), eq(test.sdk.pt, "ios"), gt(test.sdk.t, 1477971479), not(isnull(test.sdk.mac)), not(isnull(test.sdk.t))
└─TableScan_15 1.00 cop table:dd, keep order:false
└─IndexMergeJoin_24 170.34 root inner join, inner:IndexLookUp_22, outer key:test.gad.aid, inner key:test.sdk.aid, other cond:eq(test.gad.dic, test.sdk.mac), lt(test.gad.t, test.sdk.t)
├─TableReader_29 170.34 root data:Selection_28
│ └─Selection_28 170.34 cop eq(test.gad.bm, 0), eq(test.gad.dit, "mac"), eq(test.gad.pt, "ios"), gt(test.gad.t, 1477971479), not(isnull(test.gad.dic))
│ └─TableScan_27 1999.00 cop table:gad, range:[0,+inf], keep order:false
└─IndexLookUp_22 0.25 root
├─IndexScan_19 1.00 cop table:sdk, index:aid, dic, range: decided by [eq(test.sdk.aid, test.gad.aid)], keep order:true
└─Selection_21 0.25 cop eq(test.sdk.bm, 0), eq(test.sdk.pt, "ios"), gt(test.sdk.t, 1477971479), not(isnull(test.sdk.mac)), not(isnull(test.sdk.t))
└─TableScan_20 1.00 cop table:dd, keep order:true
explain SELECT cm, p1, p2, p3, p4, p5, p6_md5, p7_md5, count(1) as click_pv, count(DISTINCT ip) as click_ip FROM st WHERE (t between 1478188800 and 1478275200) and aid='cn.sbkcq' and pt='android' GROUP BY cm, p1, p2, p3, p4, p5, p6_md5, p7_md5;
id count task operator info
Projection_5 39.28 root test.st.cm, test.st.p1, test.st.p2, test.st.p3, test.st.p4, test.st.p5, test.st.p6_md5, test.st.p7_md5, 3_col_0, 3_col_1
Expand All @@ -160,14 +160,14 @@ explain select dt.id as id, dt.aid as aid, dt.pt as pt, dt.dic as dic, dt.cm as
id count task operator info
Projection_10 428.32 root test.dt.id, test.dt.aid, test.dt.pt, test.dt.dic, test.dt.cm, test.rr.gid, test.rr.acd, test.rr.t, test.dt.p1, test.dt.p2, test.dt.p3, test.dt.p4, test.dt.p5, test.dt.p6_md5, test.dt.p7_md5
└─Limit_13 428.32 root offset:0, count:2000
└─IndexJoin_19 428.32 root inner join, inner:IndexLookUp_18, outer key:test.dt.aid, test.dt.dic, inner key:test.rr.aid, test.rr.dic
├─TableReader_41 428.32 root data:Selection_40
│ └─Selection_40 428.32 cop eq(test.dt.bm, 0), eq(test.dt.pt, "ios"), gt(test.dt.t, 1478185592), not(isnull(test.dt.dic))
│ └─TableScan_39 2000.00 cop table:dt, range:[0,+inf], keep order:false
└─IndexLookUp_18 0.48 root
├─IndexScan_15 1.00 cop table:rr, index:aid, dic, range: decided by [eq(test.rr.aid, test.dt.aid) eq(test.rr.dic, test.dt.dic)], keep order:false
└─Selection_17 0.48 cop eq(test.rr.pt, "ios"), gt(test.rr.t, 1478185592)
└─TableScan_16 1.00 cop table:rr, keep order:false
└─IndexMergeJoin_25 428.32 root inner join, inner:IndexLookUp_23, outer key:test.dt.aid, test.dt.dic, inner key:test.rr.aid, test.rr.dic
├─TableReader_54 428.32 root data:Selection_53
│ └─Selection_53 428.32 cop eq(test.dt.bm, 0), eq(test.dt.pt, "ios"), gt(test.dt.t, 1478185592), not(isnull(test.dt.dic))
│ └─TableScan_52 2000.00 cop table:dt, range:[0,+inf], keep order:false
└─IndexLookUp_23 0.48 root
├─IndexScan_20 1.00 cop table:rr, index:aid, dic, range: decided by [eq(test.rr.aid, test.dt.aid) eq(test.rr.dic, test.dt.dic)], keep order:true
└─Selection_22 0.48 cop eq(test.rr.pt, "ios"), gt(test.rr.t, 1478185592)
└─TableScan_21 1.00 cop table:rr, keep order:true
explain select pc,cr,count(DISTINCT uid) as pay_users,count(oid) as pay_times,sum(am) as am from pp where ps=2 and ppt>=1478188800 and ppt<1478275200 and pi in ('510017','520017') and uid in ('18089709','18090780') group by pc,cr;
id count task operator info
Projection_5 207.86 root test.pp.pc, test.pp.cr, 3_col_0, 3_col_1, 3_col_2
Expand Down