[pipelineX](bug) Add some logs by Gabriel39 · Pull Request #27596 · apache/doris

Gabriel39 · 2023-11-27T02:31:12Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Gabriel39 · 2023-11-27T02:31:20Z

run buildall

github-actions · 2023-11-27T02:37:40Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-11-27T04:26:13Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 67f5c4bb84ebdb278b730208e3a83b4b0c83aeb4, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4891	4670	4661	4661
q2	378	172	156	156
q3	2032	1926	1860	1860
q4	1405	1259	1245	1245
q5	3947	3925	3986	3925
q6	252	125	132	125
q7	1432	887	865	865
q8	2778	2785	2773	2773
q9	9748	9584	9549	9549
q10	3444	3538	3532	3532
q11	380	239	253	239
q12	428	295	288	288
q13	4597	3824	3819	3819
q14	316	291	290	290
q15	583	538	529	529
q16	668	575	591	575
q17	1118	955	902	902
q18	7915	7520	7626	7520
q19	1653	1667	1663	1663
q20	580	330	288	288
q21	4474	4029	4042	4029
q22	472	372	372	372
Total cold run time: 53491 ms
Total hot run time: 49205 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4568	4593	4609	4593
q2	340	227	268	227
q3	4050	4012	4063	4012
q4	2730	2696	2710	2696
q5	9759	9738	9712	9712
q6	248	120	124	120
q7	3064	2455	2473	2455
q8	4452	4485	4491	4485
q9	12894	12882	12823	12823
q10	4073	4176	4152	4152
q11	802	685	646	646
q12	966	801	814	801
q13	4315	3579	3533	3533
q14	381	352	339	339
q15	577	518	529	518
q16	740	651	671	651
q17	3867	3843	3822	3822
q18	9492	9088	9298	9088
q19	1824	1791	1770	1770
q20	2384	2075	2043	2043
q21	8867	8773	8699	8699
q22	895	791	784	784
Total cold run time: 81288 ms
Total hot run time: 77969 ms

doris-robot · 2023-11-27T04:37:41Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.09 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 28 seconds loaded 2358488459 Bytes, about 80 MB/s
stream load orc: 69 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17098941305 Bytes

Gabriel39 · 2023-11-27T05:19:51Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2023-11-27T05:24:42Z

be/src/pipeline/exec/exchange_sink_operator.cpp

    return final_st;
 }

+std::string ExchangeSinkLocalState::debug_string(int indentation_level) const {


warning: method 'debug_string' can be made static [readability-convert-member-functions-to-static]

Suggested change

std::string ExchangeSinkLocalState::debug_string(int indentation_level) const {

std::string ExchangeSinkLocalState::debug_string(int indentation_level) {

be/src/pipeline/exec/exchange_sink_operator.h:188:

- std::string debug_string(int indentation_level) const override; + static std::string debug_string(int indentation_level) override;

doris-robot · 2023-11-27T06:09:31Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 0dc7242fcd82331b7e3a23923bcf389ab8fb2b56, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4930	4630	4640	4630
q2	357	164	160	160
q3	2063	1945	1920	1920
q4	1404	1275	1213	1213
q5	3966	3930	4042	3930
q6	258	132	134	132
q7	1405	890	877	877
q8	2771	2803	2786	2786
q9	9876	9519	9367	9367
q10	3456	3518	3493	3493
q11	385	256	244	244
q12	431	288	295	288
q13	4548	3765	3795	3765
q14	321	283	297	283
q15	578	539	523	523
q16	663	582	589	582
q17	1141	952	945	945
q18	7933	7623	7450	7450
q19	1691	1663	1649	1649
q20	538	292	305	292
q21	4397	4002	4037	4002
q22	483	366	388	366
Total cold run time: 53595 ms
Total hot run time: 48897 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4605	4612	4556	4556
q2	350	222	246	222
q3	4036	4008	3997	3997
q4	2728	2705	2695	2695
q5	9654	9616	9674	9616
q6	241	121	122	121
q7	3022	2508	2480	2480
q8	4435	4463	4408	4408
q9	12944	12832	12815	12815
q10	4029	4167	4139	4139
q11	735	680	636	636
q12	966	779	795	779
q13	4284	3614	3609	3609
q14	389	362	344	344
q15	569	515	520	515
q16	745	712	683	683
q17	3848	3860	3907	3860
q18	9534	9128	9109	9109
q19	1775	1800	1785	1785
q20	2389	2066	2057	2057
q21	8846	8587	8582	8582
q22	905	762	756	756
Total cold run time: 81029 ms
Total hot run time: 77764 ms

doris-robot · 2023-11-27T06:20:41Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.66 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 17 seconds loaded 2358488459 Bytes, about 132 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17099269247 Bytes

Gabriel39 · 2023-11-27T10:05:59Z

run buildall

doris-robot · 2023-11-27T11:15:51Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 26 seconds loaded 2358488459 Bytes, about 86 MB/s
stream load orc: 71 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17097538251 Bytes

doris-robot · 2023-11-27T11:18:58Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 5a5a0bfd63fe4a9b8538bbc4a5cf6cfdae31ca95, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4918	4656	4656	4656
q2	366	147	159	147
q3	2051	1950	1927	1927
q4	1426	1307	1291	1291
q5	3989	3952	4013	3952
q6	266	133	128	128
q7	1421	891	883	883
q8	2836	2848	2802	2802
q9	9997	9895	10108	9895
q10	3471	3536	3566	3536
q11	386	259	249	249
q12	445	288	287	287
q13	4588	3824	3781	3781
q14	330	286	295	286
q15	584	528	523	523
q16	660	577	586	577
q17	1146	1017	964	964
q18	8006	7543	7502	7502
q19	1690	1692	1670	1670
q20	582	296	289	289
q21	4536	4066	4115	4066
q22	472	390	382	382
Total cold run time: 54166 ms
Total hot run time: 49793 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4582	4568	4578	4568
q2	354	225	278	225
q3	4084	4017	4022	4017
q4	2758	2735	2728	2728
q5	9614	9591	9652	9591
q6	248	123	125	123
q7	3049	2490	2518	2490
q8	4443	4446	4471	4446
q9	13227	13139	13174	13139
q10	4090	4172	4159	4159
q11	789	637	645	637
q12	978	809	816	809
q13	4314	3550	3560	3550
q14	385	344	353	344
q15	571	524	514	514
q16	738	661	684	661
q17	3853	3893	3873	3873
q18	9676	8897	9170	8897
q19	1819	1822	1803	1803
q20	2416	2081	2059	2059
q21	8871	8393	8781	8393
q22	942	815	761	761
Total cold run time: 81801 ms
Total hot run time: 77787 ms

Gabriel39 · 2023-11-27T12:19:08Z

run buildall

doris-robot · 2023-11-27T13:04:04Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit f9d73cfc0ad7f6a45b55e663e661b7cb3afc354d, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4917	4612	4650	4612
q2	357	158	161	158
q3	2041	1909	1959	1909
q4	1387	1245	1251	1245
q5	3996	3926	4025	3926
q6	259	128	130	128
q7	1434	896	882	882
q8	2797	2814	2767	2767
q9	9807	10027	9558	9558
q10	3457	3550	3510	3510
q11	389	254	245	245
q12	444	293	305	293
q13	4589	3816	3841	3816
q14	322	298	286	286
q15	581	538	525	525
q16	662	584	587	584
q17	1145	970	957	957
q18	7887	7488	7513	7488
q19	1681	1695	1678	1678
q20	571	324	297	297
q21	4444	4016	4055	4016
q22	472	365	389	365
Total cold run time: 53639 ms
Total hot run time: 49245 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4596	4538	4558	4538
q2	337	217	248	217
q3	4050	4061	4011	4011
q4	2747	2730	2720	2720
q5	9608	9544	9628	9544
q6	247	121	123	121
q7	3030	2498	2487	2487
q8	4527	4494	4481	4481
q9	13262	13081	13150	13081
q10	4068	4193	4168	4168
q11	744	699	658	658
q12	981	805	808	805
q13	4290	3586	3627	3586
q14	385	378	358	358
q15	571	512	510	510
q16	732	703	696	696
q17	3983	3892	3895	3892
q18	9677	9191	9240	9191
q19	1808	1779	1792	1779
q20	2409	2074	2051	2051
q21	8932	8699	8583	8583
q22	920	857	786	786
Total cold run time: 81904 ms
Total hot run time: 78263 ms

doris-robot · 2023-11-27T13:15:37Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.89 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 26 seconds loaded 2358488459 Bytes, about 86 MB/s
stream load orc: 70 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17099156772 Bytes

Gabriel39 · 2023-11-27T13:55:42Z

run pipelinex_p0 10

yiguolei

LGTM

github-actions · 2023-11-28T01:59:51Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-11-28T01:59:54Z

PR approved by anyone and no changes requested.

HappenLee

LGTM

update

0dc7242

Gabriel39 force-pushed the fix_1127 branch from 67f5c4b to 0dc7242 Compare November 27, 2023 05:18

github-actions bot reviewed Nov 27, 2023

View reviewed changes

Gabriel39 marked this pull request as draft November 27, 2023 05:26

update

5a5a0bf

update

f9d73cf

Gabriel39 marked this pull request as ready for review November 28, 2023 01:57

Gabriel39 changed the title ~~[pipelineX](bug) Fix timeout~~ [pipelineX](bug) Add some logs Nov 28, 2023

yiguolei approved these changes Nov 28, 2023

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 28, 2023

HappenLee approved these changes Nov 28, 2023

View reviewed changes

Gabriel39 merged commit ea7eca9 into apache:master Nov 28, 2023

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[pipelineX](bug) Add some logs (apache#27596)

d64c8aa

	std::string ExchangeSinkLocalState::debug_string(int indentation_level) const {
	std::string ExchangeSinkLocalState::debug_string(int indentation_level) {

Conversation

Gabriel39 commented Nov 27, 2023

Proposed changes

Further comments

Uh oh!

Gabriel39 commented Nov 27, 2023

Uh oh!

github-actions bot commented Nov 27, 2023

Uh oh!

doris-robot commented Nov 27, 2023

Uh oh!

doris-robot commented Nov 27, 2023

Uh oh!

Gabriel39 commented Nov 27, 2023

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Nov 27, 2023

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Nov 27, 2023

Uh oh!

doris-robot commented Nov 27, 2023

Uh oh!

Gabriel39 commented Nov 27, 2023

Uh oh!

doris-robot commented Nov 27, 2023

Uh oh!

doris-robot commented Nov 27, 2023

Uh oh!

Gabriel39 commented Nov 27, 2023

Uh oh!

doris-robot commented Nov 27, 2023

Uh oh!

doris-robot commented Nov 27, 2023

Uh oh!

Gabriel39 commented Nov 27, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 28, 2023

Uh oh!

github-actions bot commented Nov 28, 2023

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants