Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](merge-on-write) Skip the alignment process of some rowsets in partial update #38487 #38686

Merged
merged 3 commits into from
Aug 8, 2024

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Aug 1, 2024

Proposed changes

picks #38487

regression cases are commented currently.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@bobhan1
Copy link
Contributor Author

bobhan1 commented Aug 1, 2024

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -165,6 +165,7 @@ class Rowset : public std::enable_shared_from_this<Rowset> {
bool is_segments_overlapping() const { return rowset_meta()->is_segments_overlapping(); }
KeysType keys_type() { return _schema->keys_type(); }
RowsetStatePB rowset_meta_state() const { return rowset_meta()->rowset_state(); }
bool produced_by_compaction() const { return rowset_meta()->produced_by_compaction(); }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'produced_by_compaction' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
bool produced_by_compaction() const { return rowset_meta()->produced_by_compaction(); }
; }[[nodiscard]]

@@ -302,6 +302,12 @@ class RowsetMeta {
return num_segments() > 1 && is_singleton_delta() && segments_overlap() != NONOVERLAPPING;
}

bool produced_by_compaction() const {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'produced_by_compaction' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
bool produced_by_compaction() const {
[[nodiscard]] bool produced_by_compaction() const {

@bobhan1
Copy link
Contributor Author

bobhan1 commented Aug 1, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49933 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 84a410a5899691c371199b64fc1529fdda1b7e0a, data reload: false

------ Round 1 ----------------------------------
q1	17686	4462	4347	4347
q2	2077	168	147	147
q3	10251	1928	1926	1926
q4	10110	1270	1350	1270
q5	8437	3900	3984	3900
q6	238	128	149	128
q7	2056	1623	1616	1616
q8	9536	2754	2735	2735
q9	13768	10562	10237	10237
q10	8668	3522	3507	3507
q11	422	247	253	247
q12	470	309	303	303
q13	18378	3957	4011	3957
q14	351	325	328	325
q15	519	465	459	459
q16	680	579	575	575
q17	1138	945	924	924
q18	7215	6837	6947	6837
q19	1807	1631	1631	1631
q20	525	329	308	308
q21	4487	4145	4110	4110
q22	537	444	463	444
Total cold run time: 119356 ms
Total hot run time: 49933 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4341	4318	4320	4318
q2	326	226	230	226
q3	4196	4118	4136	4118
q4	2782	2734	2744	2734
q5	7158	7147	7077	7077
q6	238	127	126	126
q7	3256	2872	2792	2792
q8	4370	4479	4478	4478
q9	16924	16832	16553	16553
q10	4211	4224	4275	4224
q11	792	691	695	691
q12	1030	872	847	847
q13	7173	3733	3732	3732
q14	445	424	426	424
q15	507	463	452	452
q16	733	688	672	672
q17	3762	3809	3933	3809
q18	8826	8697	8817	8697
q19	1737	1673	1727	1673
q20	2378	2103	2093	2093
q21	8448	8543	8471	8471
q22	1057	983	975	975
Total cold run time: 84690 ms
Total hot run time: 79182 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.74% (8102/21470)
Line Coverage: 29.38% (66358/225865)
Region Coverage: 28.89% (34240/118520)
Branch Coverage: 24.78% (17599/71018)
Coverage Report: http://coverage.selectdb-in.cc/coverage/84a410a5899691c371199b64fc1529fdda1b7e0a_84a410a5899691c371199b64fc1529fdda1b7e0a/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 204996 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 84a410a5899691c371199b64fc1529fdda1b7e0a, data reload: false

query1	928	389	414	389
query2	6528	2824	2678	2678
query3	6916	211	199	199
query4	21157	17998	17913	17913
query5	19730	6547	6491	6491
query6	303	223	229	223
query7	4158	306	325	306
query8	455	429	404	404
query9	3204	2736	2650	2650
query10	409	315	309	309
query11	11219	10763	10741	10741
query12	123	75	78	75
query13	5638	712	710	710
query14	17718	13734	13564	13564
query15	364	240	238	238
query16	6471	286	265	265
query17	1701	1452	868	868
query18	2339	415	421	415
query19	211	156	153	153
query20	78	76	77	76
query21	189	102	100	100
query22	5219	5076	5105	5076
query23	32726	31961	31914	31914
query24	7125	6497	6556	6497
query25	530	442	430	430
query26	624	162	162	162
query27	1994	301	305	301
query28	6212	2408	2342	2342
query29	2843	2692	2698	2692
query30	241	173	167	167
query31	922	768	757	757
query32	68	60	62	60
query33	397	253	255	253
query34	859	493	494	493
query35	1122	879	911	879
query36	1243	1126	1152	1126
query37	90	61	69	61
query38	3117	2973	2919	2919
query39	1393	1328	1339	1328
query40	236	97	100	97
query41	46	45	44	44
query42	90	82	79	79
query43	738	652	726	652
query44	1144	730	731	730
query45	249	243	233	233
query46	1244	966	984	966
query47	1959	1810	1591	1591
query48	1010	725	706	706
query49	622	375	372	372
query50	862	614	637	614
query51	4756	4702	4725	4702
query52	85	89	83	83
query53	458	327	319	319
query54	2649	2453	2469	2453
query55	99	88	86	86
query56	233	219	229	219
query57	1264	1078	1169	1078
query58	222	202	208	202
query59	4307	4046	3886	3886
query60	217	193	227	193
query61	96	92	94	92
query62	862	567	520	520
query63	490	346	356	346
query64	4322	1591	1498	1498
query65	3642	3573	3575	3573
query66	685	377	384	377
query67	18165	16945	16150	16150
query68	8180	678	674	674
query69	558	337	345	337
query70	1751	1339	1425	1339
query71	397	301	311	301
query72	6618	3526	3465	3465
query73	739	326	323	323
query74	6220	5820	5823	5820
query75	4522	3661	3604	3604
query76	4624	1095	1115	1095
query77	571	249	263	249
query78	12829	11918	12267	11918
query79	7894	650	658	650
query80	2084	400	403	400
query81	514	238	234	234
query82	1370	96	103	96
query83	182	138	135	135
query84	255	70	71	70
query85	1345	349	339	339
query86	371	290	333	290
query87	3278	2995	3033	2995
query88	4926	2316	2298	2298
query89	376	287	286	286
query90	1743	218	216	216
query91	192	138	142	138
query92	68	51	53	51
query93	3942	614	577	577
query94	885	212	211	211
query95	1094	1075	1067	1067
query96	658	323	318	318
query97	6502	6368	6374	6368
query98	186	167	170	167
query99	2873	880	927	880
Total cold run time: 315063 ms
Total hot run time: 204996 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 84a410a5899691c371199b64fc1529fdda1b7e0a, data reload: false

query1	0.03	0.02	0.01
query2	0.07	0.03	0.02
query3	0.25	0.04	0.04
query4	1.80	0.06	0.07
query5	0.52	0.52	0.52
query6	1.25	0.62	0.62
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.53	0.49	0.48
query10	0.53	0.54	0.52
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.62	0.60	0.60
query14	0.78	0.78	0.77
query15	0.76	0.75	0.75
query16	0.37	0.36	0.37
query17	1.01	1.02	1.01
query18	0.22	0.25	0.25
query19	1.89	1.84	1.88
query20	0.01	0.02	0.01
query21	15.48	0.56	0.57
query22	2.16	2.52	1.60
query23	17.32	1.10	0.94
query24	5.96	0.83	0.56
query25	0.39	0.10	0.05
query26	0.56	0.15	0.15
query27	0.05	0.05	0.03
query28	7.88	0.76	0.71
query29	12.58	2.33	2.38
query30	0.55	0.54	0.54
query31	2.81	0.38	0.36
query32	3.39	0.49	0.51
query33	3.07	3.07	3.04
query34	15.24	4.78	4.76
query35	4.82	4.83	4.82
query36	1.05	1.01	1.01
query37	0.05	0.05	0.05
query38	0.03	0.03	0.03
query39	0.02	0.01	0.01
query40	0.16	0.14	0.13
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 104.61 s
Total hot run time: 30.27 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 84a410a5899691c371199b64fc1529fdda1b7e0a with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.8 seconds inserted 10000000 Rows, about 458K ops/s

@xiaokang xiaokang changed the title [branch-2.0] Picks "[opt](merge-on-write) Skip the alignment process of some rowsets in partial update #38487" [opt](merge-on-write) Skip the alignment process of some rowsets in partial update #38487 Aug 1, 2024
@bobhan1
Copy link
Contributor Author

bobhan1 commented Aug 2, 2024

run p0

@bobhan1
Copy link
Contributor Author

bobhan1 commented Aug 2, 2024

run p1

2 similar comments
@bobhan1
Copy link
Contributor Author

bobhan1 commented Aug 2, 2024

run p1

@zhannngchen
Copy link
Contributor

run p1

@bobhan1
Copy link
Contributor Author

bobhan1 commented Aug 7, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.74% (8102/21469)
Line Coverage: 29.37% (66355/225917)
Region Coverage: 28.88% (34244/118580)
Branch Coverage: 24.76% (17598/71070)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0f63bcf78ebc770eef8e0f68a64622b1f59b403a_0f63bcf78ebc770eef8e0f68a64622b1f59b403a/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 50139 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0f63bcf78ebc770eef8e0f68a64622b1f59b403a, data reload: false

------ Round 1 ----------------------------------
q1	17843	4415	4388	4388
q2	2064	160	146	146
q3	10464	1877	1907	1877
q4	10317	1257	1327	1257
q5	8681	3922	3979	3922
q6	235	127	150	127
q7	2069	1613	1621	1613
q8	9297	2742	2732	2732
q9	10948	10621	10455	10455
q10	8637	3565	3555	3555
q11	432	244	240	240
q12	462	301	297	297
q13	18378	3946	4126	3946
q14	361	323	322	322
q15	526	471	457	457
q16	684	566	563	563
q17	1129	932	957	932
q18	7256	6987	6887	6887
q19	1799	1625	1647	1625
q20	565	326	298	298
q21	4405	4143	4064	4064
q22	525	436	449	436
Total cold run time: 117077 ms
Total hot run time: 50139 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4329	4350	4319	4319
q2	326	228	220	220
q3	4206	4188	4118	4118
q4	2757	2758	2756	2756
q5	7246	7158	7038	7038
q6	241	120	116	116
q7	3270	2866	2877	2866
q8	4380	4478	4532	4478
q9	17385	17089	17034	17034
q10	4262	4282	4253	4253
q11	762	700	837	700
q12	1051	874	848	848
q13	6902	3736	3782	3736
q14	454	426	420	420
q15	495	460	464	460
q16	751	672	696	672
q17	3842	3794	3856	3794
q18	8807	8772	8778	8772
q19	1734	1749	1663	1663
q20	2411	2117	2083	2083
q21	8550	8570	8474	8474
q22	1043	978	1007	978
Total cold run time: 85204 ms
Total hot run time: 79798 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 204038 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0f63bcf78ebc770eef8e0f68a64622b1f59b403a, data reload: false

query1	938	389	409	389
query2	6543	2672	2728	2672
query3	6912	212	198	198
query4	20162	18120	17959	17959
query5	19780	6523	6537	6523
query6	314	222	230	222
query7	4259	295	321	295
query8	412	463	380	380
query9	3063	2653	2557	2557
query10	449	298	296	296
query11	11378	10801	10881	10801
query12	131	79	72	72
query13	5630	692	704	692
query14	18223	13656	13803	13656
query15	366	247	250	247
query16	6452	277	260	260
query17	1702	1457	889	889
query18	2298	419	417	417
query19	217	168	157	157
query20	78	81	81	81
query21	190	103	102	102
query22	5344	5030	5059	5030
query23	32336	31743	31823	31743
query24	6792	6525	6559	6525
query25	542	425	433	425
query26	533	169	158	158
query27	1861	296	296	296
query28	6127	2375	2325	2325
query29	2992	2644	2660	2644
query30	245	167	163	163
query31	887	722	740	722
query32	70	65	61	61
query33	397	274	258	258
query34	863	472	465	465
query35	1169	933	941	933
query36	1343	1168	1143	1143
query37	93	60	63	60
query38	3113	2945	2940	2940
query39	1383	1312	1327	1312
query40	211	101	100	100
query41	48	45	51	45
query42	84	80	86	80
query43	723	722	693	693
query44	1173	710	720	710
query45	247	235	234	234
query46	1236	988	970	970
query47	1913	1598	1945	1598
query48	1012	723	717	717
query49	626	372	374	372
query50	869	565	614	565
query51	4754	4675	4675	4675
query52	90	81	78	78
query53	452	316	315	315
query54	2662	2454	2424	2424
query55	91	80	84	80
query56	249	219	206	206
query57	1177	1192	1188	1188
query58	216	198	205	198
query59	4335	3965	3846	3846
query60	208	191	219	191
query61	98	97	100	97
query62	704	526	526	526
query63	486	339	341	339
query64	2602	1564	1529	1529
query65	3574	3590	3551	3551
query66	769	379	383	379
query67	16431	15257	17417	15257
query68	8747	653	639	639
query69	583	362	349	349
query70	1586	1424	1483	1424
query71	426	310	315	310
query72	6563	3521	3513	3513
query73	747	317	317	317
query74	6392	6004	6057	6004
query75	5433	3645	3706	3645
query76	5288	1109	1215	1109
query77	921	255	253	253
query78	12975	11639	11989	11639
query79	8115	631	632	631
query80	1066	405	397	397
query81	490	243	238	238
query82	1497	103	100	100
query83	163	131	134	131
query84	258	71	75	71
query85	902	341	333	333
query86	312	297	302	297
query87	3256	3025	3096	3025
query88	5105	2302	2316	2302
query89	387	305	285	285
query90	1899	203	220	203
query91	172	139	140	139
query92	71	53	49	49
query93	4909	581	572	572
query94	684	208	210	208
query95	1109	1050	1066	1050
query96	650	324	327	324
query97	6636	6452	6495	6452
query98	205	180	177	177
query99	3014	853	821	821
Total cold run time: 313188 ms
Total hot run time: 204038 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.65 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0f63bcf78ebc770eef8e0f68a64622b1f59b403a, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.04
query4	1.80	0.07	0.07
query5	0.53	0.52	0.53
query6	1.24	0.62	0.61
query7	0.01	0.01	0.01
query8	0.03	0.03	0.02
query9	0.53	0.49	0.48
query10	0.54	0.55	0.55
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.61
query14	0.80	0.79	0.81
query15	0.80	0.77	0.76
query16	0.40	0.39	0.37
query17	1.04	1.03	1.00
query18	0.24	0.26	0.23
query19	1.93	1.87	1.85
query20	0.02	0.01	0.01
query21	15.47	0.56	0.55
query22	2.14	2.76	1.71
query23	17.29	0.97	0.95
query24	6.72	0.65	0.86
query25	0.33	0.10	0.05
query26	0.71	0.16	0.18
query27	0.03	0.04	0.05
query28	6.86	0.75	0.72
query29	12.78	2.36	2.31
query30	0.56	0.51	0.53
query31	2.84	0.39	0.38
query32	3.38	0.50	0.50
query33	3.06	3.08	3.10
query34	15.26	4.81	4.78
query35	4.83	4.85	4.85
query36	1.05	1.01	1.02
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.15
query41	0.06	0.02	0.02
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.79 s
Total hot run time: 30.65 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0f63bcf78ebc770eef8e0f68a64622b1f59b403a with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@zhannngchen zhannngchen merged commit 4220c3f into apache:branch-2.0 Aug 8, 2024
22 of 25 checks passed
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
…artial update apache#38487 (apache#38686)

## Proposed changes

picks apache#38487

regression cases are commented currently.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants