Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhancement](util) print if using nereids planner when be coredump #31981 #41605

Merged
merged 2 commits into from
Oct 10, 2024
Merged

[enhancement](util) print if using nereids planner when be coredump #31981 #41605

merged 2 commits into from
Oct 10, 2024

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Oct 9, 2024

Proposed changes

pick #31752
pick #31981

*** Query id: 77433c763b14535-be989f185c71e240 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1728464190 (unix time) try "date -d @1728464190" if you are using GNU date ***
*** Current BE git commitID: eaaf726019 ***
*** SIGABRT unknown detail explain (@0x47000060661) received by PID 394849 (TID 395426 OR 0x7ff4a5f36700) from PID 394849; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/yanxuecheng/branch-2.0/doris/be/src/common/signal_handler.h:421
 1# 0x00007FF6CED47B50 in /lib64/libc.so.6
 2# gsignal in /lib64/libc.so.6



*** Query id: 1b3633126e640e0-b1f06a5bb6c0b1cc ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1728464469 (unix time) try "date -d @1728464469" if you are using GNU date ***
*** Current BE git commitID: eaaf726019 ***
*** SIGABRT unknown detail explain (@0x4700006f5b8) received by PID 456120 (TID 456600 OR 0x7f72da8df700) from PID 456120; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/yanxuecheng/branch-2.0/doris/be/src/common/signal_handler.h:421
 1# 0x00007F745021BB50 in /lib64/libc.so.6

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Mryange
Copy link
Contributor Author

Mryange commented Oct 9, 2024

run buildall

@Mryange
Copy link
Contributor Author

Mryange commented Oct 9, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Oct 9, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/runtime/query_context.h
@@ -222,12 +224,16 @@ class QueryContext {
public:
TUniqueId query_id;
DescriptorTbl* desc_tbl;

bool is_nereids() const { return _is_nereids; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'is_nereids' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
bool is_nereids() const { return _is_nereids; }
[[nodiscard]] bool is_nereids() const { return _is_nereids; }

be/src/runtime/runtime_state.h
@@ -450,6 +450,9 @@ class RuntimeState {
: 0;
}


bool is_nereids() const;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'is_nereids' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
bool is_nereids() const;
[[nodiscard]] bool is_nereids() const;

@doris-robot
Copy link

TPC-H: Total hot run time: 49219 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c11dd3354205d5f4e67d6ef3ac71ffe19171864a, data reload: false

------ Round 1 ----------------------------------
q1	18097	4417	4404	4404
q2	2077	153	144	144
q3	10465	1915	1921	1915
q4	10322	1252	1333	1252
q5	8573	3932	3934	3932
q6	234	124	127	124
q7	2064	1625	1600	1600
q8	9319	2762	2724	2724
q9	10204	9983	9739	9739
q10	8648	3522	3537	3522
q11	419	246	247	246
q12	483	297	307	297
q13	18368	3948	4066	3948
q14	343	332	324	324
q15	510	470	456	456
q16	579	459	452	452
q17	1135	958	938	938
q18	7213	6813	6870	6813
q19	1705	1578	1580	1578
q20	549	335	315	315
q21	4442	4156	4117	4117
q22	485	399	379	379
Total cold run time: 116234 ms
Total hot run time: 49219 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4331	4347	4339	4339
q2	323	223	236	223
q3	4201	4194	4152	4152
q4	2764	2748	2749	2748
q5	7205	7101	7089	7089
q6	240	121	122	121
q7	3280	2860	2824	2824
q8	4315	4459	4477	4459
q9	13699	13696	13641	13641
q10	4239	4272	4231	4231
q11	729	727	653	653
q12	1021	842	861	842
q13	6866	3743	3710	3710
q14	447	419	429	419
q15	491	464	469	464
q16	621	584	589	584
q17	3744	3834	3774	3774
q18	8803	8762	8856	8762
q19	1745	1719	1664	1664
q20	2424	2132	2125	2125
q21	8411	8402	8439	8402
q22	1039	919	936	919
Total cold run time: 80938 ms
Total hot run time: 76145 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212151 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c11dd3354205d5f4e67d6ef3ac71ffe19171864a, data reload: false

query1	936	385	437	385
query2	6550	2322	2074	2074
query3	6917	205	204	204
query4	23435	21798	21529	21529
query5	19723	6508	6500	6500
query6	278	224	227	224
query7	4328	301	307	301
query8	255	260	268	260
query9	3124	2667	2605	2605
query10	464	318	305	305
query11	15869	15104	15385	15104
query12	133	76	73	73
query13	1031	446	437	437
query14	17517	13063	13741	13063
query15	374	219	227	219
query16	6484	284	268	268
query17	1918	928	873	873
query18	895	324	303	303
query19	216	149	145	145
query20	102	99	102	99
query21	186	99	102	99
query22	5325	5168	5017	5017
query23	34174	33814	33251	33251
query24	7871	6297	6314	6297
query25	504	437	428	428
query26	1275	160	162	160
query27	2398	297	291	291
query28	6087	2276	2261	2261
query29	2768	2803	2582	2582
query30	247	169	167	167
query31	987	740	751	740
query32	70	65	59	59
query33	458	265	272	265
query34	855	483	466	466
query35	1124	906	921	906
query36	1406	1314	1190	1190
query37	173	58	59	58
query38	3089	2893	2889	2889
query39	1382	1321	1316	1316
query40	314	98	91	91
query41	40	37	36	36
query42	90	91	86	86
query43	660	633	600	600
query44	1205	737	732	732
query45	248	231	229	229
query46	1244	970	958	958
query47	1921	1764	1699	1699
query48	518	417	407	407
query49	647	381	366	366
query50	875	614	603	603
query51	4821	4707	4602	4602
query52	89	84	87	84
query53	241	189	196	189
query54	2670	2459	2437	2437
query55	86	84	88	84
query56	216	214	210	210
query57	1346	1146	1279	1146
query58	217	211	205	205
query59	3526	3510	3279	3279
query60	218	209	210	209
query61	99	95	96	95
query62	817	509	458	458
query63	198	175	178	175
query64	3458	1559	1380	1380
query65	3617	3552	3549	3549
query66	786	397	439	397
query67	17307	15548	15852	15548
query68	10526	674	651	651
query69	492	263	275	263
query70	1801	1455	1571	1455
query71	406	296	301	296
query72	6741	4800	4782	4782
query73	753	326	327	326
query74	6332	5783	5874	5783
query75	5334	3752	3608	3608
query76	6194	1134	1200	1134
query77	1035	260	259	259
query78	12415	11694	11779	11694
query79	6115	681	643	643
query80	1235	372	385	372
query81	486	238	243	238
query82	1644	95	94	94
query83	169	135	141	135
query84	253	73	70	70
query85	878	329	318	318
query86	344	300	301	300
query87	3193	3031	2961	2961
query88	4394	2317	2308	2308
query89	405	287	300	287
query90	1994	217	212	212
query91	157	126	124	124
query92	65	51	52	51
query93	6040	564	571	564
query94	703	209	208	208
query95	2043	1966	1955	1955
query96	636	321	326	321
query97	6500	6396	6392	6392
query98	208	217	205	205
query99	2847	883	875	875
Total cold run time: 320740 ms
Total hot run time: 212151 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c11dd3354205d5f4e67d6ef3ac71ffe19171864a, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.03
query3	0.25	0.06	0.05
query4	1.79	0.06	0.07
query5	0.54	0.52	0.51
query6	1.25	0.63	0.62
query7	0.01	0.01	0.01
query8	0.04	0.03	0.03
query9	0.51	0.49	0.47
query10	0.52	0.53	0.52
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.62	0.62	0.61
query14	0.79	0.79	0.79
query15	0.78	0.75	0.77
query16	0.37	0.38	0.35
query17	1.03	1.01	1.00
query18	0.22	0.27	0.22
query19	1.94	1.87	1.86
query20	0.01	0.01	0.01
query21	15.45	0.57	0.58
query22	1.86	2.04	1.61
query23	17.23	1.07	1.00
query24	4.22	1.12	2.99
query25	0.38	0.07	0.06
query26	0.68	0.16	0.16
query27	0.04	0.04	0.03
query28	6.97	0.76	0.73
query29	12.76	2.16	2.36
query30	0.59	0.52	0.49
query31	2.81	0.39	0.37
query32	3.37	0.50	0.49
query33	3.06	3.05	3.05
query34	15.24	4.78	4.77
query35	4.88	4.86	4.85
query36	1.05	1.01	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.13
query41	0.08	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 101.98 s
Total hot run time: 30.79 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c11dd3354205d5f4e67d6ef3ac71ffe19171864a with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@Mryange
Copy link
Contributor Author

Mryange commented Oct 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49307 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 09afd490765c6f5e17e1b0f2bbb16869e2018ac1, data reload: false

------ Round 1 ----------------------------------
q1	17624	4419	4368	4368
q2	2090	151	146	146
q3	10302	1902	1947	1902
q4	10727	1259	1333	1259
q5	8388	3836	3900	3836
q6	238	121	125	121
q7	2035	1625	1604	1604
q8	9696	2735	2708	2708
q9	10239	9891	9675	9675
q10	8877	3557	3527	3527
q11	423	248	250	248
q12	463	300	293	293
q13	18711	4112	4117	4112
q14	363	338	354	338
q15	521	471	459	459
q16	568	477	513	477
q17	1135	1011	1002	1002
q18	7645	7224	6934	6934
q19	3855	1500	1509	1500
q20	785	313	318	313
q21	4524	4157	4095	4095
q22	498	390	393	390
Total cold run time: 119707 ms
Total hot run time: 49307 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4321	4352	4297	4297
q2	323	229	220	220
q3	4152	4127	4079	4079
q4	2731	2743	2746	2743
q5	7172	7080	7129	7080
q6	237	120	118	118
q7	3161	2842	2810	2810
q8	4369	4473	4453	4453
q9	13824	13579	13676	13579
q10	4242	4297	4258	4258
q11	767	680	693	680
q12	1024	844	843	843
q13	7019	3735	3750	3735
q14	454	427	425	425
q15	492	473	463	463
q16	626	596	598	596
q17	3766	3836	3896	3836
q18	8796	8692	8645	8645
q19	1702	1657	1612	1612
q20	2372	2128	2070	2070
q21	8353	8423	8421	8421
q22	1013	948	910	910
Total cold run time: 80916 ms
Total hot run time: 75873 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (8150/21549)
Line Coverage: 29.56% (67095/226976)
Region Coverage: 29.03% (34588/119129)
Branch Coverage: 24.94% (17820/71442)
Coverage Report: http://coverage.selectdb-in.cc/coverage/09afd490765c6f5e17e1b0f2bbb16869e2018ac1_09afd490765c6f5e17e1b0f2bbb16869e2018ac1/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 210862 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 09afd490765c6f5e17e1b0f2bbb16869e2018ac1, data reload: false

query1	936	401	404	401
query2	6539	2265	2297	2265
query3	6928	207	202	202
query4	23204	21596	21258	21258
query5	19721	6507	6377	6377
query6	280	228	243	228
query7	4329	304	301	301
query8	255	267	243	243
query9	3043	2631	2590	2590
query10	464	301	299	299
query11	15876	15048	14773	14773
query12	123	71	72	71
query13	1040	440	437	437
query14	17598	13244	13233	13233
query15	403	217	232	217
query16	6474	275	261	261
query17	1744	897	925	897
query18	898	326	316	316
query19	213	158	154	154
query20	100	97	99	97
query21	188	93	96	93
query22	5270	4915	4959	4915
query23	34137	33310	33502	33310
query24	7799	6183	6294	6183
query25	513	432	425	425
query26	1264	162	157	157
query27	2432	291	297	291
query28	6201	2290	2256	2256
query29	2866	2656	2853	2656
query30	245	168	161	161
query31	978	777	719	719
query32	68	62	58	58
query33	459	247	257	247
query34	949	474	469	469
query35	1146	898	947	898
query36	1286	1193	1064	1064
query37	172	59	59	59
query38	3029	2942	2869	2869
query39	1363	1313	1313	1313
query40	313	93	96	93
query41	42	37	37	37
query42	90	89	98	89
query43	639	636	620	620
query44	1144	734	731	731
query45	246	227	232	227
query46	1232	967	951	951
query47	1862	1853	1795	1795
query48	496	421	412	412
query49	652	373	365	365
query50	849	572	574	572
query51	4779	4683	4655	4655
query52	86	87	78	78
query53	227	181	198	181
query54	2615	2461	2484	2461
query55	85	82	84	82
query56	232	218	201	201
query57	1202	1178	1116	1116
query58	218	214	209	209
query59	3626	3267	3412	3267
query60	214	214	216	214
query61	97	93	94	93
query62	832	451	502	451
query63	201	189	176	176
query64	5852	1620	1337	1337
query65	3708	3553	3519	3519
query66	618	411	401	401
query67	17444	15490	16993	15490
query68	9270	654	646	646
query69	500	254	273	254
query70	1680	1299	1458	1299
query71	388	300	301	300
query72	6833	4752	4316	4316
query73	814	336	340	336
query74	6295	5884	5841	5841
query75	4843	3641	3786	3641
query76	4907	1151	1188	1151
query77	688	254	244	244
query78	12528	11623	12767	11623
query79	10767	651	632	632
query80	1705	377	389	377
query81	505	237	233	233
query82	1383	96	92	92
query83	171	137	137	137
query84	255	67	73	67
query85	1076	322	315	315
query86	340	288	295	288
query87	3207	3047	3030	3030
query88	4612	2307	2292	2292
query89	471	288	285	285
query90	1902	206	213	206
query91	159	124	124	124
query92	56	50	53	50
query93	6082	607	590	590
query94	770	205	211	205
query95	1951	1961	1921	1921
query96	651	331	328	328
query97	6506	6405	6394	6394
query98	219	213	202	202
query99	2779	882	857	857
Total cold run time: 324447 ms
Total hot run time: 210862 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.78 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 09afd490765c6f5e17e1b0f2bbb16869e2018ac1, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.06
query4	1.78	0.06	0.06
query5	0.54	0.52	0.51
query6	1.25	0.66	0.62
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.51	0.48	0.47
query10	0.53	0.52	0.52
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.62	0.62	0.62
query14	0.78	0.78	0.78
query15	0.78	0.76	0.75
query16	0.35	0.34	0.36
query17	1.02	0.98	1.01
query18	0.21	0.25	0.25
query19	1.94	1.87	1.74
query20	0.01	0.00	0.01
query21	15.48	0.57	0.56
query22	1.97	3.12	2.18
query23	17.19	0.98	0.97
query24	5.71	1.78	1.65
query25	0.36	0.10	0.05
query26	0.80	0.15	0.16
query27	0.05	0.03	0.04
query28	6.05	0.71	0.76
query29	12.58	2.31	2.28
query30	0.57	0.52	0.52
query31	2.80	0.39	0.36
query32	3.40	0.49	0.49
query33	3.07	3.03	3.03
query34	15.23	4.80	4.79
query35	4.87	4.83	4.86
query36	1.04	1.00	1.01
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 102.49 s
Total hot run time: 31.78 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 09afd490765c6f5e17e1b0f2bbb16869e2018ac1 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@Mryange
Copy link
Contributor Author

Mryange commented Oct 10, 2024

run p0

HappenLee
HappenLee approved these changes Oct 10, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 10, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@HappenLee HappenLee merged commit 9115b64 into apache:branch-2.0 Oct 10, 2024
20 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@HappenLee HappenLee HappenLee approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Mryange @doris-robot @HappenLee