Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](compress) Fix occasional crushes when serializing blocks #32672

Merged
merged 1 commit into from Mar 22, 2024

Conversation

HappenLee
Copy link
Contributor

@HappenLee HappenLee commented Mar 22, 2024

Proposed changes

before:
26925e7c28dc4bdcb5f233cef33bb1e

  1. add the padding length In compress length
  2. add fuzzy test config
  3. update 2.0 -> 2.1 change compress way to none

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@HappenLee
Copy link
Contributor Author

run buildall

Gabriel39
Gabriel39 previously approved these changes Mar 22, 2024
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 37373 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba27f8a96f208bf376767b240b063ec84f4e2d4c, data reload: false

------ Round 1 ----------------------------------
q1	17624	4158	4070	4070
q2	2102	156	153	153
q3	10593	1153	1180	1153
q4	10222	765	718	718
q5	7473	2976	2955	2955
q6	203	123	121	121
q7	1026	571	552	552
q8	9341	1953	1960	1953
q9	6983	6456	6425	6425
q10	8420	3290	3444	3290
q11	442	245	219	219
q12	427	193	197	193
q13	17801	2867	2850	2850
q14	243	198	214	198
q15	520	461	457	457
q16	495	365	364	364
q17	935	501	580	501
q18	7168	6769	6482	6482
q19	3994	1388	1405	1388
q20	541	250	245	245
q21	3501	2784	2976	2784
q22	344	302	305	302
Total cold run time: 110398 ms
Total hot run time: 37373 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4142	4047	4044	4044
q2	331	232	232	232
q3	2940	2800	2791	2791
q4	1813	1571	1496	1496
q5	5186	5237	5218	5218
q6	200	119	117	117
q7	2227	1855	1815	1815
q8	3164	3285	3248	3248
q9	8580	8542	8542	8542
q10	3671	3670	3659	3659
q11	548	446	440	440
q12	725	567	542	542
q13	16923	2833	2838	2833
q14	283	243	252	243
q15	503	454	453	453
q16	460	429	426	426
q17	1710	1488	1451	1451
q18	7441	7127	7279	7127
q19	1627	1538	1521	1521
q20	1888	1731	1704	1704
q21	4771	4548	4656	4548
q22	530	455	478	455
Total cold run time: 69663 ms
Total hot run time: 52905 ms

@@ -1757,6 +1757,17 @@ public void initFuzzyModeVariables() {
this.enableFunctionPushdown = true;
this.enableDeleteSubPredicateV2 = true;
}

switch (randomInt) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using PR_ID % 3 to fuzzy this variable.

@doris-robot
Copy link

TPC-DS: Total hot run time: 180893 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba27f8a96f208bf376767b240b063ec84f4e2d4c, data reload: false

query1	924	369	344	344
query2	7432	1937	1973	1937
query3	6707	213	220	213
query4	30869	20525	20486	20486
query5	4376	407	416	407
query6	276	180	172	172
query7	4616	288	310	288
query8	237	176	175	175
query9	9245	2309	2281	2281
query10	561	237	245	237
query11	15093	13945	14042	13945
query12	137	89	86	86
query13	1618	409	410	409
query14	12209	10864	10794	10794
query15	321	197	201	197
query16	8236	269	259	259
query17	2059	573	549	549
query18	2100	293	280	280
query19	353	153	154	153
query20	93	91	88	88
query21	198	130	131	130
query22	4618	4460	4475	4460
query23	32008	31275	31114	31114
query24	10740	2836	2771	2771
query25	593	387	385	385
query26	1191	157	160	157
query27	2401	366	354	354
query28	7297	1906	1887	1887
query29	898	667	642	642
query30	302	149	151	149
query31	974	734	759	734
query32	97	58	58	58
query33	769	278	257	257
query34	1011	479	487	479
query35	831	615	619	615
query36	1006	882	864	864
query37	126	78	76	76
query38	3595	3442	3455	3442
query39	1415	1416	1385	1385
query40	227	116	112	112
query41	54	48	46	46
query42	103	99	97	97
query43	461	444	450	444
query44	1115	727	746	727
query45	273	262	277	262
query46	1083	696	675	675
query47	1653	1584	1610	1584
query48	452	358	347	347
query49	1113	349	334	334
query50	756	373	372	372
query51	6758	6758	6685	6685
query52	108	99	97	97
query53	349	286	281	281
query54	317	253	246	246
query55	98	79	77	77
query56	252	223	228	223
query57	1080	1019	1023	1019
query58	233	200	205	200
query59	2681	2666	2443	2443
query60	263	250	245	245
query61	95	97	96	96
query62	602	396	441	396
query63	305	279	283	279
query64	5615	3828	3788	3788
query65	3038	2982	3013	2982
query66	915	345	342	342
query67	14872	14174	14195	14174
query68	5472	512	524	512
query69	554	379	389	379
query70	1225	1102	1150	1102
query71	414	286	284	284
query72	6288	2635	2494	2494
query73	703	314	315	314
query74	7152	6538	6502	6502
query75	3549	2835	2862	2835
query76	3569	901	884	884
query77	395	277	270	270
query78	10279	9832	9595	9595
query79	8453	542	520	520
query80	2081	425	413	413
query81	564	221	214	214
query82	1650	193	204	193
query83	335	150	146	146
query84	291	82	83	82
query85	1630	387	365	365
query86	485	292	300	292
query87	3787	3559	3565	3559
query88	5209	2314	2318	2314
query89	545	377	372	372
query90	1992	179	184	179
query91	198	137	139	137
query92	66	51	49	49
query93	6938	510	492	492
query94	1233	180	181	180
query95	435	329	336	329
query96	605	268	271	268
query97	3024	2847	2860	2847
query98	226	209	205	205
query99	1080	746	734	734
Total cold run time: 302830 ms
Total hot run time: 180893 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ba27f8a96f208bf376767b240b063ec84f4e2d4c with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

yiguolei
yiguolei previously approved these changes Mar 22, 2024
@HappenLee
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38006 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01b474c6e705e458a706b91164316bd65c9ff7b1, data reload: false

------ Round 1 ----------------------------------
q1	17642	4762	4137	4137
q2	2107	161	154	154
q3	10583	1187	1209	1187
q4	10236	819	759	759
q5	7462	2995	2991	2991
q6	207	126	124	124
q7	1029	590	560	560
q8	9340	2067	2014	2014
q9	7145	6498	6466	6466
q10	8442	3428	3544	3428
q11	429	232	224	224
q12	437	198	200	198
q13	17808	2910	2897	2897
q14	236	201	203	201
q15	516	463	470	463
q16	484	382	363	363
q17	949	560	598	560
q18	7237	6582	6423	6423
q19	2698	1427	1497	1427
q20	549	256	250	250
q21	3556	2970	2879	2879
q22	350	309	301	301
Total cold run time: 109442 ms
Total hot run time: 38006 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4166	4125	4057	4057
q2	329	235	230	230
q3	2938	2867	2891	2867
q4	1863	1573	1561	1561
q5	5230	5321	5301	5301
q6	193	116	117	116
q7	2242	1912	1837	1837
q8	3133	3281	3304	3281
q9	8589	8625	8602	8602
q10	3760	3747	3722	3722
q11	536	450	463	450
q12	747	572	556	556
q13	16933	2891	2914	2891
q14	284	245	264	245
q15	501	452	458	452
q16	477	440	433	433
q17	1746	1486	1503	1486
q18	7567	7205	7055	7055
q19	1630	1483	1537	1483
q20	1914	1751	1737	1737
q21	4989	4826	4833	4826
q22	543	468	445	445
Total cold run time: 70310 ms
Total hot run time: 53633 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182084 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01b474c6e705e458a706b91164316bd65c9ff7b1, data reload: false

query1	941	361	350	350
query2	7463	2040	2020	2020
query3	6710	210	222	210
query4	31187	20600	20653	20600
query5	4294	406	509	406
query6	279	182	171	171
query7	4647	299	298	298
query8	234	170	179	170
query9	9584	2368	2377	2368
query10	590	254	265	254
query11	17309	14131	14049	14049
query12	138	89	94	89
query13	1629	412	428	412
query14	12456	11056	10985	10985
query15	262	204	196	196
query16	8232	266	257	257
query17	1997	561	538	538
query18	2099	299	297	297
query19	350	160	161	160
query20	92	87	90	87
query21	203	131	128	128
query22	4660	4509	4432	4432
query23	31945	31068	31196	31068
query24	10626	2821	2848	2821
query25	622	383	390	383
query26	1179	162	165	162
query27	2498	357	374	357
query28	7042	1955	1899	1899
query29	872	655	636	636
query30	308	153	150	150
query31	953	744	761	744
query32	104	62	59	59
query33	789	354	258	258
query34	943	489	495	489
query35	851	613	609	609
query36	998	894	878	878
query37	117	80	80	80
query38	3609	3490	3471	3471
query39	1437	1362	1392	1362
query40	216	115	120	115
query41	54	49	51	49
query42	106	97	97	97
query43	483	453	439	439
query44	1195	744	742	742
query45	289	268	257	257
query46	1100	729	718	718
query47	1687	1601	1601	1601
query48	445	353	362	353
query49	1104	348	344	344
query50	761	382	387	382
query51	6773	6791	6708	6708
query52	101	98	88	88
query53	352	282	287	282
query54	323	253	256	253
query55	85	83	82	82
query56	258	241	242	241
query57	1082	1021	1010	1010
query58	241	206	209	206
query59	2740	2579	2520	2520
query60	271	264	256	256
query61	123	120	119	119
query62	596	396	384	384
query63	300	282	288	282
query64	5422	3809	4008	3809
query65	3087	2995	2974	2974
query66	896	365	352	352
query67	14955	14419	14316	14316
query68	8692	572	560	560
query69	629	389	415	389
query70	1201	1098	1148	1098
query71	515	287	280	280
query72	6322	2686	2521	2521
query73	750	328	321	321
query74	8322	6620	6670	6620
query75	4373	2913	2852	2852
query76	5278	940	899	899
query77	657	281	266	266
query78	10284	9710	9487	9487
query79	11084	515	525	515
query80	1831	408	381	381
query81	526	212	215	212
query82	350	195	200	195
query83	215	145	149	145
query84	285	81	83	81
query85	1123	320	317	317
query86	362	298	320	298
query87	3711	3563	3554	3554
query88	5422	2371	2390	2371
query89	470	357	374	357
query90	2100	177	176	176
query91	177	138	138	138
query92	66	50	48	48
query93	6173	521	512	512
query94	1338	189	188	188
query95	451	348	338	338
query96	597	273	272	272
query97	3043	2880	2862	2862
query98	232	219	209	209
query99	1037	740	753	740
Total cold run time: 312910 ms
Total hot run time: 182084 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 01b474c6e705e458a706b91164316bd65c9ff7b1 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       19.9 seconds inserted 10000000 Rows, about 502K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.27% (8733/24757)
Line Coverage: 27.07% (71520/264172)
Region Coverage: 26.31% (37101/141007)
Branch Coverage: 23.22% (18980/81752)
Coverage Report: http://coverage.selectdb-in.cc/coverage/01b474c6e705e458a706b91164316bd65c9ff7b1_01b474c6e705e458a706b91164316bd65c9ff7b1/report/index.html

@HappenLee HappenLee merged commit fc1cf2b into apache:master Mar 22, 2024
27 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. meta-change reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants