Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](move-memtable) avoid raw pointer to RuntimeState in DeltaWriterV2 #33682

Closed
wants to merge 5 commits into from

Conversation

kaijchen
Copy link
Contributor

@kaijchen kaijchen commented Apr 15, 2024

Proposed changes

Avoid using raw pointer to RuntimeState in DeltaWriterV2.
Which was introduced in #29280 and #30610.
To fix the following coredump:

*** Aborted at 1713017220 (unix time) try "date -d @1713017220" if you are using GNU date ***
*** Current BE git commitID: babec88aa90 ***
*** SIGSEGV address not mapped to object (@0x6bbb13) received by PID 2977685 (TID 2978173 OR 0x7fa3fea63700) from PID 7060243; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/common/signal_handler.h:421
 1# 0x00007FA4B30000A7 in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
 3# 0x00007FA4B2FF902C in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
 4# 0x00007FA4B85B5090 in /lib/x86_64-linux-gnu/libc.so.6
 5# memcpy at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
 6# doris::RuntimeState::cancel_reason[abi:cxx11]() const at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/runtime_state.cpp:375
 7# doris::DeltaWriterV2::write(doris::vectorized::Block const*, std::vector<unsigned int, std::allocator<unsigned int> > const&, bool) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/olap/delta_writer_v2.cpp:164
 8# doris::vectorized::VTabletWriterV2::_write_memtable(std::shared_ptr<doris::vectorized::Block>, long, doris::vectorized::Rows const&, std::vector<std::shared_ptr<doris::LoadStreamStub>, std::allocator<std::shared_ptr<doris::LoadStreamStub> > > const&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:461
 9# doris::vectorized::VTabletWriterV2::write(doris::vectorized::Block&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:413
10# _ZN5doris10vectorized15AsyncWriterSinkINS0_15VTabletWriterV2EXadsoKcL_ZNS0_19VOLAP_TABLE_SINK_V2EEEEE4sendEPNS_12RuntimeStateEPNS0_5BlockEb at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/sink/async_writer_sink.h:89
11# doris::PlanFragmentExecutor::open_vectorized_internal() at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/plan_fragment_executor.cpp:341
12# doris::PlanFragmentExecutor::open() at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/plan_fragment_executor.cpp:273
13# doris::PlanFragmentExecutor::execute() at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/plan_fragment_executor.cpp:404
14# doris::FragmentMgr::_exec_actual(std::shared_ptr<doris::PlanFragmentExecutor>, std::function<void (doris::RuntimeState*, doris::Status*)> const&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/fragment_mgr.cpp:455
15# std::_Function_handler<void (), doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function<void (doris::RuntimeState*, doris::Status*)> const&)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
16# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
17# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:499
18# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478
19# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaijchen
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38631 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 87aed94bd6c77d49c777e22a7073214e636e163d, data reload: false

------ Round 1 ----------------------------------
q1	17602	4401	4217	4217
q2	2003	199	188	188
q3	10437	1179	1237	1179
q4	10196	768	818	768
q5	7576	2720	2642	2642
q6	217	133	133	133
q7	1024	613	582	582
q8	9213	2094	2060	2060
q9	7200	6631	6591	6591
q10	8604	3568	3560	3560
q11	446	231	224	224
q12	462	212	210	210
q13	17771	2936	2961	2936
q14	271	235	228	228
q15	519	483	482	482
q16	542	394	390	390
q17	977	689	682	682
q18	7474	6792	6756	6756
q19	6020	1507	1526	1507
q20	655	306	324	306
q21	3536	2689	2845	2689
q22	366	301	301	301
Total cold run time: 113111 ms
Total hot run time: 38631 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4462	4209	4192	4192
q2	381	270	275	270
q3	3065	2775	2803	2775
q4	1858	1614	1582	1582
q5	5288	5293	5333	5293
q6	204	122	125	122
q7	2235	1812	1849	1812
q8	3208	3384	3349	3349
q9	8598	8564	8643	8564
q10	4093	3885	4020	3885
q11	600	517	496	496
q12	803	657	636	636
q13	17334	3278	3131	3131
q14	319	290	303	290
q15	531	492	491	491
q16	527	468	469	468
q17	1822	1490	1545	1490
q18	8191	7856	7847	7847
q19	1666	1587	1567	1567
q20	2076	1882	1884	1882
q21	7996	4980	4979	4979
q22	550	462	455	455
Total cold run time: 75807 ms
Total hot run time: 55576 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184010 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 87aed94bd6c77d49c777e22a7073214e636e163d, data reload: false

query1	882	370	359	359
query2	6393	2625	2343	2343
query3	6649	202	199	199
query4	24827	21301	21296	21296
query5	4134	398	400	398
query6	286	188	183	183
query7	4578	283	296	283
query8	219	169	163	163
query9	8425	2374	2355	2355
query10	411	234	252	234
query11	14998	14306	14362	14306
query12	134	84	83	83
query13	1628	352	362	352
query14	9193	7496	7953	7496
query15	241	182	184	182
query16	8149	288	255	255
query17	1928	558	543	543
query18	2104	270	260	260
query19	266	148	144	144
query20	89	86	83	83
query21	193	134	123	123
query22	4977	4831	4882	4831
query23	33705	32972	33481	32972
query24	10890	2918	2969	2918
query25	630	402	381	381
query26	707	157	151	151
query27	2456	367	357	357
query28	5989	2124	2125	2124
query29	860	638	610	610
query30	313	179	177	177
query31	1029	766	746	746
query32	95	53	53	53
query33	663	269	243	243
query34	896	491	480	480
query35	855	687	697	687
query36	1099	908	923	908
query37	107	74	69	69
query38	3425	3322	3344	3322
query39	1641	1599	1571	1571
query40	177	128	132	128
query41	43	44	41	41
query42	110	99	102	99
query43	561	528	508	508
query44	1089	747	747	747
query45	293	260	283	260
query46	1083	765	725	725
query47	2028	1906	1919	1906
query48	376	303	309	303
query49	835	377	366	366
query50	822	399	393	393
query51	6917	6785	6823	6785
query52	99	95	93	93
query53	340	282	276	276
query54	293	224	217	217
query55	75	69	71	69
query56	244	235	222	222
query57	1198	1134	1130	1130
query58	221	198	201	198
query59	3371	3314	3155	3155
query60	260	234	234	234
query61	108	106	132	106
query62	587	448	429	429
query63	300	280	284	280
query64	4667	3874	3964	3874
query65	3034	3056	3019	3019
query66	752	329	322	322
query67	15240	15061	14968	14968
query68	5204	528	551	528
query69	480	292	291	291
query70	1241	1137	1114	1114
query71	416	276	259	259
query72	6382	2728	2446	2446
query73	711	319	325	319
query74	6829	6409	6304	6304
query75	3307	2581	2666	2581
query76	3258	1043	969	969
query77	419	258	260	258
query78	10922	10331	10212	10212
query79	3955	525	508	508
query80	1895	424	434	424
query81	527	242	240	240
query82	793	97	95	95
query83	289	166	166	166
query84	256	91	86	86
query85	1892	323	260	260
query86	477	273	280	273
query87	3474	3295	3271	3271
query88	4839	2403	2399	2399
query89	513	372	375	372
query90	1936	181	175	175
query91	120	94	96	94
query92	57	44	46	44
query93	5193	517	502	502
query94	1099	182	177	177
query95	380	289	294	289
query96	597	268	268	268
query97	3124	2930	2962	2930
query98	227	222	221	221
query99	1225	842	857	842
Total cold run time: 283224 ms
Total hot run time: 184010 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.52 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 87aed94bd6c77d49c777e22a7073214e636e163d, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.04	0.04
query4	1.68	0.07	0.07
query5	0.48	0.49	0.49
query6	1.47	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.55	0.49	0.48
query10	0.56	0.55	0.54
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.62	0.59	0.58
query14	0.76	0.77	0.78
query15	0.83	0.79	0.80
query16	0.37	0.38	0.37
query17	0.99	0.95	1.01
query18	0.22	0.22	0.27
query19	1.83	1.65	1.70
query20	0.02	0.01	0.01
query21	15.39	0.64	0.64
query22	4.43	8.19	1.25
query23	18.24	1.29	1.21
query24	1.68	0.31	0.23
query25	0.14	0.08	0.08
query26	0.25	0.16	0.16
query27	0.08	0.08	0.08
query28	13.32	0.99	0.98
query29	12.56	3.26	3.28
query30	0.26	0.06	0.06
query31	2.87	0.38	0.37
query32	3.28	0.47	0.46
query33	2.87	2.80	2.79
query34	17.05	4.41	4.47
query35	4.55	4.48	4.45
query36	0.65	0.48	0.46
query37	0.18	0.15	0.14
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.09	0.06	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.42 s
Total hot run time: 29.52 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 87aed94bd6c77d49c777e22a7073214e636e163d with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@kaijchen
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38316 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 80a6115151220ae6d5808b1fab158ef95ca0c54d, data reload: false

------ Round 1 ----------------------------------
q1	17629	4266	4204	4204
q2	2010	191	184	184
q3	10448	1196	1194	1194
q4	10188	732	824	732
q5	7517	2672	2659	2659
q6	215	131	129	129
q7	1009	606	585	585
q8	9239	2063	2038	2038
q9	8874	6596	6493	6493
q10	8534	3510	3509	3509
q11	430	240	226	226
q12	475	221	213	213
q13	17781	2956	2983	2956
q14	258	238	221	221
q15	518	494	485	485
q16	559	388	380	380
q17	934	655	743	655
q18	7389	6676	6717	6676
q19	4661	1476	1528	1476
q20	643	300	299	299
q21	3437	2709	2856	2709
q22	358	293	300	293
Total cold run time: 113106 ms
Total hot run time: 38316 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4383	4200	4225	4200
q2	365	265	279	265
q3	2984	2705	2761	2705
q4	1885	1651	1581	1581
q5	5298	5292	5325	5292
q6	209	125	122	122
q7	2236	1866	1838	1838
q8	3198	3338	3308	3308
q9	8623	8502	8609	8502
q10	4104	3887	4048	3887
q11	604	499	502	499
q12	797	652	655	652
q13	17329	3206	3166	3166
q14	324	319	294	294
q15	521	486	479	479
q16	519	455	487	455
q17	1826	1494	1496	1494
q18	8004	7917	7736	7736
q19	1660	1558	1550	1550
q20	2050	1874	1834	1834
q21	5183	5031	5115	5031
q22	540	473	472	472
Total cold run time: 72642 ms
Total hot run time: 55362 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184308 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 80a6115151220ae6d5808b1fab158ef95ca0c54d, data reload: false

query1	903	379	363	363
query2	6223	2688	2376	2376
query3	6655	200	195	195
query4	23332	21412	21266	21266
query5	4177	404	392	392
query6	278	182	185	182
query7	4591	290	286	286
query8	239	174	174	174
query9	8539	2386	2361	2361
query10	428	248	250	248
query11	14702	14214	14261	14214
query12	133	92	85	85
query13	1651	358	364	358
query14	9268	7986	7471	7471
query15	252	178	175	175
query16	8150	266	264	264
query17	1912	568	553	553
query18	2107	275	273	273
query19	262	151	149	149
query20	89	83	84	83
query21	201	132	122	122
query22	5031	4867	4850	4850
query23	33764	33146	33368	33146
query24	6987	3017	3041	3017
query25	559	394	419	394
query26	709	166	149	149
query27	2197	360	381	360
query28	3965	2145	2146	2145
query29	887	624	616	616
query30	285	182	178	178
query31	1024	779	749	749
query32	93	51	55	51
query33	503	254	247	247
query34	894	511	506	506
query35	809	704	719	704
query36	1089	936	944	936
query37	107	68	72	68
query38	3487	3349	3288	3288
query39	1643	1607	1584	1584
query40	183	131	131	131
query41	47	44	46	44
query42	116	104	117	104
query43	600	558	532	532
query44	1089	746	743	743
query45	304	271	264	264
query46	1117	738	749	738
query47	2005	1915	1968	1915
query48	389	324	312	312
query49	804	379	387	379
query50	771	390	401	390
query51	6871	6853	6782	6782
query52	105	90	97	90
query53	352	273	276	273
query54	264	235	236	235
query55	77	73	71	71
query56	249	226	228	226
query57	1179	1161	1107	1107
query58	228	212	217	212
query59	3509	3253	3004	3004
query60	262	240	240	240
query61	108	108	106	106
query62	544	451	438	438
query63	310	278	279	278
query64	4958	3894	3936	3894
query65	3054	3033	3033	3033
query66	761	328	329	328
query67	15403	14977	14793	14793
query68	6024	543	532	532
query69	522	300	300	300
query70	1276	1193	1184	1184
query71	465	270	259	259
query72	6438	2647	2424	2424
query73	727	325	319	319
query74	6847	6576	6452	6452
query75	3568	2634	2642	2634
query76	3625	996	936	936
query77	603	262	262	262
query78	10941	10283	10123	10123
query79	8776	545	513	513
query80	1847	442	429	429
query81	532	242	239	239
query82	1567	99	98	98
query83	304	162	167	162
query84	260	85	88	85
query85	1132	259	265	259
query86	463	322	279	279
query87	3503	3253	3281	3253
query88	5203	2417	2425	2417
query89	561	362	388	362
query90	1939	177	177	177
query91	121	96	97	96
query92	56	47	45	45
query93	6905	509	502	502
query94	1067	178	181	178
query95	392	292	297	292
query96	596	264	263	263
query97	3171	2920	2937	2920
query98	239	218	214	214
query99	1249	871	860	860
Total cold run time: 284703 ms
Total hot run time: 184308 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.75 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 80a6115151220ae6d5808b1fab158ef95ca0c54d, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.65	0.09	0.08
query5	0.49	0.53	0.50
query6	1.48	0.72	0.71
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.55	0.48	0.50
query10	0.54	0.56	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.58	0.59
query14	0.75	0.78	0.78
query15	0.83	0.81	0.81
query16	0.37	0.37	0.36
query17	1.02	1.02	1.01
query18	0.22	0.25	0.23
query19	1.81	1.71	1.71
query20	0.01	0.01	0.01
query21	15.41	0.67	0.64
query22	4.16	7.31	2.29
query23	18.29	1.34	1.23
query24	2.05	0.21	0.20
query25	0.15	0.08	0.08
query26	0.26	0.17	0.16
query27	0.08	0.07	0.08
query28	13.29	1.00	0.97
query29	12.60	3.29	3.26
query30	0.25	0.07	0.06
query31	2.86	0.38	0.38
query32	3.29	0.46	0.47
query33	2.80	2.82	2.87
query34	16.95	4.38	4.41
query35	4.48	4.47	4.48
query36	0.65	0.49	0.47
query37	0.17	0.16	0.15
query38	0.16	0.15	0.14
query39	0.04	0.04	0.03
query40	0.18	0.14	0.14
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.03	0.03	0.04
Total cold run time: 109.34 s
Total hot run time: 30.75 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 80a6115151220ae6d5808b1fab158ef95ca0c54d with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@kaijchen
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39493 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40a4bae2e1e28f9d2bc744eaf32a87175ba58ccf, data reload: false

------ Round 1 ----------------------------------
q1	17740	4424	4269	4269
q2	2025	191	185	185
q3	10542	1197	1155	1155
q4	11189	792	823	792
q5	7762	2829	2694	2694
q6	231	138	137	137
q7	1074	659	621	621
q8	10166	2150	2103	2103
q9	8206	7327	7204	7204
q10	8811	3565	3700	3565
q11	432	235	230	230
q12	485	223	212	212
q13	18040	2966	2997	2966
q14	268	225	231	225
q15	535	480	471	471
q16	547	418	390	390
q17	965	619	645	619
q18	7456	6821	6728	6728
q19	2003	1547	1550	1547
q20	648	333	307	307
q21	3563	2758	3033	2758
q22	379	315	320	315
Total cold run time: 113067 ms
Total hot run time: 39493 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4370	4236	4250	4236
q2	376	269	278	269
q3	3048	2777	2772	2772
q4	1908	1639	1621	1621
q5	5388	5351	5354	5351
q6	210	123	124	123
q7	2257	1891	1885	1885
q8	3225	3321	3419	3321
q9	9350	9324	9335	9324
q10	3913	3690	3731	3690
q11	594	493	469	469
q12	747	571	582	571
q13	16540	3111	2965	2965
q14	294	261	285	261
q15	516	476	462	462
q16	489	446	442	442
q17	1777	1500	1491	1491
q18	7613	7627	7431	7431
q19	1726	1503	1558	1503
q20	1971	1734	1746	1734
q21	4881	4781	4812	4781
q22	551	450	464	450
Total cold run time: 71744 ms
Total hot run time: 55152 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184014 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40a4bae2e1e28f9d2bc744eaf32a87175ba58ccf, data reload: false

query1	908	390	371	371
query2	7313	2550	2412	2412
query3	6672	203	206	203
query4	23144	21406	21396	21396
query5	4220	406	425	406
query6	264	176	169	169
query7	4582	294	280	280
query8	225	178	170	170
query9	8451	2315	2317	2315
query10	587	245	248	245
query11	15105	14329	14157	14157
query12	144	89	87	87
query13	1638	369	364	364
query14	9739	7899	7940	7899
query15	300	187	179	179
query16	8185	265	267	265
query17	1905	593	563	563
query18	2112	285	277	277
query19	328	154	158	154
query20	95	85	89	85
query21	198	126	130	126
query22	5079	4817	4819	4817
query23	33974	33324	33015	33015
query24	12052	2923	2845	2845
query25	665	366	378	366
query26	1714	160	150	150
query27	3000	305	314	305
query28	7726	2026	1994	1994
query29	1017	592	591	591
query30	312	171	176	171
query31	965	716	737	716
query32	94	57	55	55
query33	752	272	236	236
query34	1086	462	478	462
query35	832	685	698	685
query36	1025	948	930	930
query37	274	67	69	67
query38	3424	3166	3192	3166
query39	1584	1527	1509	1509
query40	274	126	125	125
query41	45	42	43	42
query42	105	94	98	94
query43	565	529	520	520
query44	1242	708	717	708
query45	292	256	269	256
query46	1069	728	729	728
query47	1913	1869	1863	1863
query48	378	298	291	291
query49	1184	379	375	375
query50	747	372	374	372
query51	6801	6636	6820	6636
query52	98	88	90	88
query53	346	287	275	275
query54	310	229	223	223
query55	76	70	70	70
query56	242	228	225	225
query57	1233	1140	1100	1100
query58	225	200	215	200
query59	3522	3231	3340	3231
query60	256	254	233	233
query61	88	89	88	88
query62	653	450	445	445
query63	307	282	276	276
query64	6243	3926	4099	3926
query65	3168	3029	3053	3029
query66	1382	343	377	343
query67	15659	15101	15029	15029
query68	5136	539	532	532
query69	525	301	293	293
query70	1244	1197	1152	1152
query71	421	277	267	267
query72	6695	2583	2423	2423
query73	702	318	314	314
query74	6822	6385	6495	6385
query75	3668	2746	2648	2648
query76	3860	938	1002	938
query77	627	267	265	265
query78	10953	10410	10208	10208
query79	4438	530	522	522
query80	1724	481	428	428
query81	528	250	242	242
query82	827	94	97	94
query83	312	172	186	172
query84	267	85	83	83
query85	1970	293	258	258
query86	519	297	297	297
query87	3486	3284	3277	3277
query88	4548	2317	2316	2316
query89	498	365	365	365
query90	1964	177	175	175
query91	126	98	97	97
query92	56	46	46	46
query93	5429	507	496	496
query94	1226	183	178	178
query95	386	302	298	298
query96	593	260	266	260
query97	3158	2928	2925	2925
query98	237	226	219	219
query99	1279	866	847	847
Total cold run time: 294337 ms
Total hot run time: 184014 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40a4bae2e1e28f9d2bc744eaf32a87175ba58ccf, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.70	0.07	0.06
query5	0.50	0.50	0.50
query6	1.47	0.71	0.71
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.55	0.50	0.49
query10	0.55	0.55	0.54
query11	0.17	0.12	0.12
query12	0.14	0.12	0.13
query13	0.60	0.59	0.59
query14	0.77	0.76	0.78
query15	0.84	0.80	0.81
query16	0.37	0.35	0.35
query17	0.96	0.94	0.98
query18	0.21	0.25	0.23
query19	1.76	1.72	1.65
query20	0.01	0.01	0.01
query21	15.41	0.64	0.64
query22	4.39	6.49	2.47
query23	18.28	1.32	1.37
query24	1.51	0.37	0.21
query25	0.15	0.08	0.08
query26	0.25	0.16	0.15
query27	0.08	0.08	0.07
query28	13.31	1.02	0.98
query29	12.54	3.20	3.25
query30	0.27	0.06	0.06
query31	2.88	0.37	0.37
query32	3.27	0.45	0.46
query33	2.80	2.85	2.85
query34	17.18	4.33	4.41
query35	4.51	4.49	4.48
query36	0.66	0.46	0.47
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.17	0.13	0.13
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.23 s
Total hot run time: 30.73 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 40a4bae2e1e28f9d2bc744eaf32a87175ba58ccf with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.3 seconds inserted 10000000 Rows, about 751K ops/s

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memtable thread will access many other members in runtime state. We should find why runtime state is deconstructed.

@kaijchen kaijchen closed this Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants