Skip to content

[Improment]Add CustomStdAllocator#40918

Closed
wangbo wants to merge 1 commit intoapache:masterfrom
wangbo:0913_std_mem
Closed

[Improment]Add CustomStdAllocator#40918
wangbo wants to merge 1 commit intoapache:masterfrom
wangbo:0913_std_mem

Conversation

@wangbo
Copy link
Contributor

@wangbo wangbo commented Sep 18, 2024

Proposed changes

Add a custom allocator to instead of std::allocator, used for std::vector, so that we can add std::vector's memory to thread context's memory tracker.

Currently just add the CustomStdAllocator and using it in some aggregation operator, later all block used by aggregation operator should use the CustomStdAllocator.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wangbo
Copy link
Contributor Author

wangbo commented Sep 18, 2024

run buildall

@wangbo wangbo changed the title [Improment]Add customStdAllocator [Improment]Add CustomStdAllocator Sep 18, 2024
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.32% (9581/25676)
Line Coverage: 28.70% (79187/275886)
Region Coverage: 28.18% (41002/145508)
Branch Coverage: 24.81% (20898/84248)
Coverage Report: http://coverage.selectdb-in.cc/coverage/164932abd9cd00ce89a75208a24acc1741fae63f_164932abd9cd00ce89a75208a24acc1741fae63f/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41312 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 164932abd9cd00ce89a75208a24acc1741fae63f, data reload: false

------ Round 1 ----------------------------------
q1	17557	7340	7234	7234
q2	2042	181	153	153
q3	10729	1063	1180	1063
q4	10531	789	747	747
q5	7719	3056	3051	3051
q6	236	155	151	151
q7	1007	650	617	617
q8	9426	2076	2045	2045
q9	6814	6413	6410	6410
q10	7022	2282	2278	2278
q11	439	247	250	247
q12	398	215	215	215
q13	17774	2953	2971	2953
q14	251	224	223	223
q15	579	527	512	512
q16	709	601	622	601
q17	975	796	797	796
q18	7259	6582	6836	6582
q19	1412	982	1049	982
q20	578	305	276	276
q21	3977	3207	3223	3207
q22	1096	969	1004	969
Total cold run time: 108530 ms
Total hot run time: 41312 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7198	7227	7223	7223
q2	325	223	222	222
q3	2923	2973	2958	2958
q4	2005	1861	1827	1827
q5	5565	5622	5562	5562
q6	234	144	145	144
q7	2218	1826	1823	1823
q8	3288	3470	3410	3410
q9	8736	8891	8739	8739
q10	3522	3380	3423	3380
q11	585	500	489	489
q12	799	605	606	605
q13	9809	3137	3143	3137
q14	307	285	274	274
q15	592	541	517	517
q16	711	679	659	659
q17	1833	1624	1622	1622
q18	8290	7839	7694	7694
q19	1734	1550	1553	1550
q20	2123	1908	1868	1868
q21	5504	5330	5511	5330
q22	1150	1071	1051	1051
Total cold run time: 69451 ms
Total hot run time: 60084 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198706 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 164932abd9cd00ce89a75208a24acc1741fae63f, data reload: false

query1	1293	893	883	883
query2	6383	2026	1993	1993
query3	10809	4096	3865	3865
query4	62612	29848	23481	23481
query5	5146	480	476	476
query6	399	160	157	157
query7	5439	306	300	300
query8	300	217	223	217
query9	8474	2711	2658	2658
query10	456	291	282	282
query11	17610	15166	15737	15166
query12	158	109	105	105
query13	1495	428	427	427
query14	10884	7656	7468	7468
query15	208	173	182	173
query16	6876	496	488	488
query17	1175	615	604	604
query18	1488	319	321	319
query19	243	149	148	148
query20	122	106	111	106
query21	212	102	102	102
query22	4853	4834	4477	4477
query23	34466	33662	33807	33662
query24	6024	2852	2873	2852
query25	513	377	390	377
query26	655	149	154	149
query27	1660	285	279	279
query28	4246	2466	2421	2421
query29	655	406	401	401
query30	226	148	160	148
query31	949	792	785	785
query32	80	53	53	53
query33	435	287	281	281
query34	900	489	479	479
query35	838	728	719	719
query36	1057	955	943	943
query37	144	81	85	81
query38	4021	3902	3890	3890
query39	1462	1423	1381	1381
query40	198	94	92	92
query41	46	46	52	46
query42	118	93	92	92
query43	515	513	476	476
query44	1156	808	795	795
query45	190	164	161	161
query46	1115	754	769	754
query47	1931	1793	1825	1793
query48	475	349	353	349
query49	677	390	389	389
query50	835	399	401	399
query51	7055	6863	6879	6863
query52	97	86	84	84
query53	242	175	181	175
query54	573	448	454	448
query55	76	78	81	78
query56	278	264	262	262
query57	1237	1102	1113	1102
query58	240	249	241	241
query59	3110	2899	3095	2899
query60	281	255	257	255
query61	103	104	109	104
query62	765	662	660	660
query63	216	186	193	186
query64	1353	625	622	622
query65	3306	3197	3209	3197
query66	698	298	307	298
query67	16276	15719	15452	15452
query68	1196	867	846	846
query69	432	331	344	331
query70	1174	1212	1205	1205
query71	336	334	321	321
query72	4414	3354	3285	3285
query73	594	567	570	567
query74	9186	9058	9013	9013
query75	3004	2864	2943	2864
query76	1135	851	859	851
query77	415	355	351	351
query78	9496	9297	9263	9263
query79	922	884	879	879
query80	567	569	560	560
query81	448	247	250	247
query82	231	226	232	226
query83	173	156	154	154
query84	215	98	93	93
query85	698	390	368	368
query86	323	307	320	307
query87	4530	4290	4520	4290
query88	4415	4056	4031	4031
query89	367	353	355	353
query90	1356	310	303	303
query91	162	161	162	161
query92	72	70	73	70
query93	891	879	884	879
query94	497	352	374	352
query95	422	406	409	406
query96	489	481	486	481
query97	3150	3120	3086	3086
query98	230	224	215	215
query99	1305	1289	1280	1280
Total cold run time: 300973 ms
Total hot run time: 198706 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 164932abd9cd00ce89a75208a24acc1741fae63f, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.64	0.10	0.10
query5	0.50	0.49	0.50
query6	1.14	0.73	0.72
query7	0.01	0.01	0.02
query8	0.04	0.03	0.02
query9	0.57	0.51	0.50
query10	0.54	0.57	0.55
query11	0.14	0.10	0.10
query12	0.15	0.11	0.11
query13	0.62	0.59	0.58
query14	3.04	3.01	2.95
query15	0.90	0.83	0.81
query16	0.38	0.39	0.40
query17	1.06	1.02	1.03
query18	0.19	0.19	0.20
query19	1.85	1.85	2.00
query20	0.01	0.01	0.01
query21	15.38	0.58	0.59
query22	2.37	2.75	1.36
query23	17.25	0.84	0.86
query24	2.76	0.74	1.15
query25	0.15	0.14	0.13
query26	0.45	0.14	0.13
query27	0.04	0.05	0.03
query28	10.92	1.09	1.07
query29	12.56	3.22	3.22
query30	0.26	0.06	0.07
query31	2.87	0.38	0.38
query32	3.28	0.46	0.47
query33	2.97	3.03	3.09
query34	16.72	4.44	4.41
query35	4.42	4.37	4.42
query36	0.70	0.50	0.49
query37	0.08	0.06	0.05
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.03
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.67 s
Total hot run time: 32.12 s

@wangbo wangbo closed this Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants