Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug](tablet-shuffle) tablet shuffle sink data should use input block rather than convert_block #41293

Merged
merged 1 commit into from
Sep 26, 2024
Merged

[Bug](tablet-shuffle) tablet shuffle sink data should use input block rather than convert_block #41293

merged 1 commit into from
Sep 26, 2024

Conversation

zhangstar333
Copy link
Contributor

Proposed changes

the convert_block is empty firstly, after execute exprs will be filled.
so convert_block maybe different with block.
so when send data we still need use block.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41146 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 583bafc0bed3aa03b00580788a22aafdbe74dd3c, data reload: false

------ Round 1 ----------------------------------
q1	18066	7458	7340	7340
q2	3081	162	152	152
q3	11131	1167	1161	1161
q4	10939	825	728	728
q5	7822	2906	2889	2889
q6	242	152	156	152
q7	976	632	625	625
q8	9678	1900	1964	1900
q9	6543	6374	6351	6351
q10	7006	2258	2357	2258
q11	432	244	251	244
q12	404	219	216	216
q13	17793	3018	2994	2994
q14	244	220	214	214
q15	567	517	523	517
q16	684	627	624	624
q17	975	544	603	544
q18	7291	6672	6728	6672
q19	1395	1082	1099	1082
q20	573	290	291	290
q21	4049	3204	3234	3204
q22	1103	989	1027	989
Total cold run time: 110994 ms
Total hot run time: 41146 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7287	7273	7303	7273
q2	333	236	230	230
q3	2918	2799	2777	2777
q4	2023	1785	1731	1731
q5	5431	5435	5475	5435
q6	226	145	143	143
q7	2087	1695	1740	1695
q8	3240	3378	3404	3378
q9	8508	8506	8542	8506
q10	3495	3463	3425	3425
q11	577	479	471	471
q12	802	601	619	601
q13	5701	3024	3035	3024
q14	289	260	258	258
q15	566	504	512	504
q16	701	681	648	648
q17	1793	1589	1579	1579
q18	7668	7309	7353	7309
q19	1673	1574	1437	1437
q20	2036	1814	1823	1814
q21	5465	5302	5247	5247
q22	1116	1006	1020	1006
Total cold run time: 63935 ms
Total hot run time: 58491 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191543 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 583bafc0bed3aa03b00580788a22aafdbe74dd3c, data reload: false

query1	967	367	373	367
query2	6537	2148	2121	2121
query3	6706	219	226	219
query4	34772	23665	23472	23472
query5	4542	477	463	463
query6	272	179	171	171
query7	4632	312	326	312
query8	283	226	222	222
query9	9696	2688	2676	2676
query10	444	277	269	269
query11	18155	15334	15251	15251
query12	168	94	99	94
query13	1612	419	391	391
query14	10438	7634	7580	7580
query15	309	170	174	170
query16	7963	429	478	429
query17	1756	571	567	567
query18	2092	310	305	305
query19	359	151	152	151
query20	115	107	108	107
query21	211	102	104	102
query22	4452	4370	4092	4092
query23	34662	33972	33954	33954
query24	11172	2834	2808	2808
query25	659	404	409	404
query26	1545	165	160	160
query27	2786	304	296	296
query28	8267	2471	2466	2466
query29	947	410	418	410
query30	321	155	154	154
query31	1030	790	803	790
query32	103	58	56	56
query33	770	291	286	286
query34	993	480	499	480
query35	886	729	717	717
query36	1106	945	943	943
query37	148	90	85	85
query38	4036	3910	3918	3910
query39	1607	1430	1425	1425
query40	275	98	95	95
query41	49	47	48	47
query42	113	99	96	96
query43	534	504	502	502
query44	1216	813	785	785
query45	193	167	166	166
query46	1139	696	713	696
query47	1920	1841	1814	1814
query48	471	370	370	370
query49	1202	397	391	391
query50	833	401	406	401
query51	7172	6961	6886	6886
query52	100	86	85	85
query53	259	188	186	186
query54	1237	453	477	453
query55	81	71	77	71
query56	261	253	255	253
query57	1226	1093	1105	1093
query58	243	231	234	231
query59	3199	3017	3055	3017
query60	287	272	280	272
query61	134	100	115	100
query62	868	682	658	658
query63	223	186	190	186
query64	5102	628	609	609
query65	3280	3211	3232	3211
query66	1412	311	305	305
query67	15855	15378	15514	15378
query68	3551	567	595	567
query69	430	300	293	293
query70	1170	1148	1148	1148
query71	328	274	271	271
query72	6069	4284	4113	4113
query73	768	341	354	341
query74	9830	9017	8966	8966
query75	3374	2624	2688	2624
query76	1931	895	894	894
query77	432	307	309	307
query78	10025	9260	9230	9230
query79	1820	600	601	600
query80	1347	474	470	470
query81	564	246	248	246
query82	936	144	144	144
query83	244	142	151	142
query84	254	80	79	79
query85	1350	368	365	365
query86	434	274	298	274
query87	4523	4253	4338	4253
query88	3631	2398	2372	2372
query89	397	288	291	288
query90	2041	192	194	192
query91	186	140	143	140
query92	75	49	50	49
query93	1538	545	541	541
query94	1062	300	295	295
query95	361	253	252	252
query96	608	277	278	277
query97	3251	3173	3095	3095
query98	218	196	202	196
query99	1722	1287	1319	1287
Total cold run time: 300873 ms
Total hot run time: 191543 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 583bafc0bed3aa03b00580788a22aafdbe74dd3c, data reload: false

query1	0.05	0.05	0.05
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.52	0.49	0.51
query6	1.13	0.75	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.55	0.58	0.54
query11	0.14	0.11	0.11
query12	0.14	0.10	0.11
query13	0.61	0.60	0.59
query14	3.02	3.07	2.95
query15	0.89	0.83	0.84
query16	0.39	0.39	0.40
query17	1.07	1.03	1.06
query18	0.24	0.23	0.22
query19	1.98	1.87	1.94
query20	0.01	0.01	0.01
query21	15.37	0.59	0.55
query22	2.36	2.47	1.80
query23	17.20	0.90	0.89
query24	2.49	0.52	1.72
query25	0.16	0.06	0.11
query26	0.53	0.13	0.14
query27	0.04	0.04	0.03
query28	10.87	1.09	1.08
query29	12.54	3.26	3.25
query30	0.26	0.07	0.06
query31	2.87	0.39	0.39
query32	3.28	0.46	0.47
query33	2.94	3.01	3.04
query34	17.18	4.48	4.45
query35	4.50	4.48	4.49
query36	0.69	0.51	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.15	0.12	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 107.02 s
Total hot run time: 32.57 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.29% (9625/25808)
Line Coverage: 28.71% (79674/277495)
Region Coverage: 28.12% (41180/146434)
Branch Coverage: 24.77% (20985/84736)
Coverage Report: http://coverage.selectdb-in.cc/coverage/583bafc0bed3aa03b00580788a22aafdbe74dd3c_583bafc0bed3aa03b00580788a22aafdbe74dd3c/report/index.html

HappenLee
HappenLee approved these changes Sep 26, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@zhangstar333 zhangstar333 merged commit eb0e4b8 into apache:master Sep 26, 2024
24 of 29 checks passed
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Sep 26, 2024
@zhangstar333
[Bug](tablet-shuffle) tablet shuffle sink data should use input block…
379180b 
… rather than convert_block (apache#41293)

## Proposed changes
 the convert_block is empty firstly, after execute exprs will be filled.
 so convert_block maybe different with block.
 so when send data we still need use block.

<!--Describe your changes.-->
@zhangstar333 zhangstar333 mentioned this pull request Sep 26, 2024
Merged
@wm1581066 wm1581066 added the usercase Important user case type label label Sep 26, 2024
dataroaring pushed a commit that referenced this pull request Sep 27, 2024
@zhangstar333 @dataroaring
[Bug](tablet-shuffle) tablet shuffle sink data should use input block…
1955124 
… rather than convert_block (#41293)

## Proposed changes
 the convert_block is empty firstly, after execute exprs will be filled.
 so convert_block maybe different with block.
 so when send data we still need use block.

<!--Describe your changes.-->
yiguolei pushed a commit to zhangstar333/incubator-doris that referenced this pull request Oct 10, 2024
@zhangstar333 @yiguolei
[Bug](tablet-shuffle) tablet shuffle sink data should use input block…
9431202 
… rather than convert_block (apache#41293)

## Proposed changes
 the convert_block is empty firstly, after execute exprs will be filled.
 so convert_block maybe different with block.
 so when send data we still need use block.

<!--Describe your changes.-->
cjj2010 pushed a commit to cjj2010/doris that referenced this pull request Oct 12, 2024
@zhangstar333 @cjj2010
[Bug](tablet-shuffle) tablet shuffle sink data should use input block…
21be006 
… rather than convert_block (apache#41293)

## Proposed changes
 the convert_block is empty firstly, after execute exprs will be filled.
 so convert_block maybe different with block.
 so when send data we still need use block.

<!--Describe your changes.-->
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Oct 12, 2024
@zhangstar333
[Bug](tablet-shuffle) tablet shuffle sink data should use input block…
bc0d1e5 
… rather than convert_block (apache#41293)

## Proposed changes
 the convert_block is empty firstly, after execute exprs will be filled.
 so convert_block maybe different with block.
 so when send data we still need use block.

<!--Describe your changes.-->
yiguolei pushed a commit that referenced this pull request Oct 14, 2024
@zhangstar333
[cherry-pick](branch-21) fix tablet sink shuffle without project not …
d97642e 
…match the output tuple (#40299)(#41293) (#41327)

## Proposed changes

cherry-pick from master  (#40299)(#41293)

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Mryange Mryange Mryange approved these changes

@HappenLee HappenLee HappenLee approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.2-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@zhangstar333 @doris-robot @HappenLee @Mryange @yiguolei @wm1581066 @dataroaring