Skip to content

[fix][mow] segment_creator should not flush data when encouter abnormal exit #33802

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Apr 18, 2024
Merged

[fix][mow] segment_creator should not flush data when encouter abnormal exit #33802

merged 1 commit into from
Apr 18, 2024

Conversation

hust-hhb
Copy link
Contributor

when encouter abnormal exit, the data maybe inconsistent in primary_key_index, and no need to flush data, so should not call close in here.

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hust-hhb
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

gavinchou
gavinchou reviewed Apr 17, 2024
View reviewed changes
be/src/olap/rowset/beta_rowset_writer.cpp
BaseBetaRowsetWriter::~BaseBetaRowsetWriter() {
// TODO(lingbin): Should wrapper exception logic, no need to know file ops directly.
if (!_already_built) { // abnormal exit, remove all files generated
WARN_IF_ERROR(_segment_creator.close(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we may need to check if the state of this writer.

zhannngchen
zhannngchen approved these changes Apr 17, 2024
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 17, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38711 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a79a909163c22c6086b8104cc8949428ca11969f, data reload: false

------ Round 1 ----------------------------------
q1	17610	4352	4242	4242
q2	2013	189	184	184
q3	10505	1238	1244	1238
q4	11398	738	768	738
q5	7818	2805	2674	2674
q6	225	137	134	134
q7	1051	631	616	616
q8	9610	2146	2089	2089
q9	7733	6732	6576	6576
q10	8609	3550	3518	3518
q11	439	236	220	220
q12	516	224	218	218
q13	17755	2951	2944	2944
q14	271	228	234	228
q15	522	481	472	472
q16	527	375	381	375
q17	969	717	623	623
q18	7392	6744	6780	6744
q19	1664	1546	1531	1531
q20	637	320	319	319
q21	3665	2724	2845	2724
q22	355	304	309	304
Total cold run time: 111284 ms
Total hot run time: 38711 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4301	4224	4203	4203
q2	383	264	271	264
q3	3005	2728	2804	2728
q4	1831	1578	1560	1560
q5	5394	5380	5307	5307
q6	205	122	121	121
q7	2272	1906	1871	1871
q8	3184	3371	3350	3350
q9	8586	8531	8568	8531
q10	3910	3728	3699	3699
q11	577	491	491	491
q12	767	595	595	595
q13	16615	3010	2934	2934
q14	310	282	275	275
q15	526	470	463	463
q16	475	433	437	433
q17	1769	1496	1462	1462
q18	7554	7559	7439	7439
q19	1985	1561	1546	1546
q20	1960	1746	1738	1738
q21	4975	4688	4704	4688
q22	565	460	458	458
Total cold run time: 71149 ms
Total hot run time: 54156 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.45% (8908/25127)
Line Coverage: 27.15% (73172/269553)
Region Coverage: 26.29% (37839/143933)
Branch Coverage: 23.08% (19280/83520)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a79a909163c22c6086b8104cc8949428ca11969f_a79a909163c22c6086b8104cc8949428ca11969f/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 184973 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a79a909163c22c6086b8104cc8949428ca11969f, data reload: false

query1	912	370	363	363
query2	6498	2567	2319	2319
query3	6664	206	211	206
query4	23251	21275	21212	21212
query5	4134	406	406	406
query6	267	175	167	167
query7	4584	285	282	282
query8	225	186	185	185
query9	8629	2405	2394	2394
query10	586	247	248	247
query11	14848	14305	14260	14260
query12	137	94	85	85
query13	1635	369	370	369
query14	10156	8105	7980	7980
query15	266	189	182	182
query16	8247	260	262	260
query17	2055	582	549	549
query18	2109	279	276	276
query19	345	155	157	155
query20	91	87	84	84
query21	200	127	135	127
query22	5014	4815	4756	4756
query23	33939	33202	33491	33202
query24	11762	2948	2986	2948
query25	678	424	377	377
query26	1761	148	148	148
query27	2999	311	309	309
query28	7690	2034	2013	2013
query29	1027	598	593	593
query30	308	177	165	165
query31	956	726	737	726
query32	93	52	52	52
query33	750	248	260	248
query34	1088	470	474	470
query35	818	693	685	685
query36	1075	933	914	914
query37	262	70	67	67
query38	3390	3156	3164	3156
query39	1600	1550	1524	1524
query40	274	125	122	122
query41	45	44	43	43
query42	105	95	99	95
query43	577	559	560	559
query44	1192	714	728	714
query45	277	264	276	264
query46	1066	684	720	684
query47	1902	1830	1837	1830
query48	369	290	294	290
query49	1122	382	370	370
query50	770	400	382	382
query51	6789	6505	6607	6505
query52	99	90	86	86
query53	364	284	281	281
query54	312	235	227	227
query55	79	73	72	72
query56	245	219	216	216
query57	1225	1124	1114	1114
query58	216	191	192	191
query59	3383	3058	3077	3058
query60	249	229	234	229
query61	87	87	93	87
query62	650	431	441	431
query63	315	284	282	282
query64	6165	4004	3822	3822
query65	3138	3013	3035	3013
query66	1397	352	328	328
query67	15417	15059	15070	15059
query68	5214	580	545	545
query69	487	310	302	302
query70	1257	1125	1171	1125
query71	1396	1264	1267	1264
query72	6297	2643	2431	2431
query73	700	320	319	319
query74	6851	6343	6418	6343
query75	3642	2673	2609	2609
query76	3435	975	981	975
query77	406	289	265	265
query78	10994	10306	10300	10300
query79	4617	525	519	519
query80	1759	414	466	414
query81	538	244	242	242
query82	841	97	97	97
query83	272	170	171	170
query84	272	89	84	84
query85	2146	265	255	255
query86	541	282	306	282
query87	3499	3295	3338	3295
query88	4800	2407	2395	2395
query89	506	380	370	370
query90	2049	184	180	180
query91	122	98	101	98
query92	65	47	45	45
query93	5173	520	499	499
query94	1221	177	173	173
query95	376	300	286	286
query96	604	266	257	257
query97	3176	2977	2946	2946
query98	240	221	220	220
query99	1221	853	874	853
Total cold run time: 293595 ms
Total hot run time: 184973 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a79a909163c22c6086b8104cc8949428ca11969f, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.04
query4	1.69	0.07	0.07
query5	0.48	0.48	0.50
query6	1.43	0.71	0.73
query7	0.03	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.52	0.50
query10	0.55	0.56	0.54
query11	0.16	0.12	0.12
query12	0.15	0.11	0.12
query13	0.60	0.59	0.59
query14	0.75	0.78	0.78
query15	0.82	0.81	0.80
query16	0.36	0.36	0.38
query17	0.96	0.96	1.02
query18	0.21	0.23	0.25
query19	1.88	1.69	1.71
query20	0.02	0.01	0.01
query21	15.41	0.64	0.64
query22	4.53	6.39	2.24
query23	18.31	1.28	1.16
query24	1.42	0.43	0.23
query25	0.14	0.09	0.09
query26	0.26	0.17	0.17
query27	0.08	0.08	0.07
query28	13.27	1.01	0.98
query29	12.53	3.27	3.25
query30	0.32	0.06	0.06
query31	3.40	0.38	0.38
query32	3.40	0.46	0.46
query33	3.01	2.77	2.82
query34	17.01	4.41	4.41
query35	4.45	4.50	4.48
query36	0.65	0.46	0.47
query37	0.19	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.18	0.14	0.15
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.96 s
Total hot run time: 30.55 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a79a909163c22c6086b8104cc8949428ca11969f with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@dataroaring dataroaring merged commit 6dbbf2e into apache:master Apr 18, 2024
dataroaring pushed a commit that referenced this pull request Apr 18, 2024
@hust-hhb @dataroaring
[fix][mow] segment_creator should not flush data when encouter abnorm…
6130aa3 
…al exit (#33802)
yiguolei pushed a commit that referenced this pull request Apr 18, 2024
@hust-hhb @Doris-Extras
[fix][mow] segment_creator should not flush data when encouter abnorm…
ad80a65 
…al exit (#33802)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou left review comments

@platoneko platoneko platoneko approved these changes

@zhannngchen zhannngchen zhannngchen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged p0_c reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@hust-hhb @doris-robot @gavinchou @platoneko @zhannngchen @xiaokang @dataroaring