Skip to content

[fix](load) delete bytes_appended debug point which may cause bad page - #31602

Merged
yiguolei merged 1 commit into
apache:masterfrom
liaoxin01:fix_inject
Feb 29, 2024
Merged

[fix](load) delete bytes_appended debug point which may cause bad page#31602
yiguolei merged 1 commit into
apache:masterfrom
liaoxin01:fix_inject

Conversation

@liaoxin01

@liaoxin01 liaoxin01 commented Feb 29, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

F20240229 05:10:37.714732 2801470 tablet.cpp:2911] Check failed: expected_st || st.is() unexpected error status while lookup_row_key:[CORRUPTION]Bad page: too small size (0), file=/mnt/hdd03/ci/branch21-data/doris.HDD/data/1/10124/779557774/020000000003f960ce4d7ae53de30448374c635caabfc090_0.dat

When running this injection case, the statistical information is executed in the background, which may cause the table of statistical information to generate a bad page.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot

doris-robot commented Feb 29, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@liaoxin01

liaoxin01 commented Feb 29, 2024

Copy link
Copy Markdown
Contributor Author

run buildall

@liaoxin01

liaoxin01 commented Feb 29, 2024

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions

github-actions Bot commented Feb 29, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actions Bot commented Feb 29, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Feb 29, 2024
@github-actions

github-actions Bot commented Feb 29, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actions Bot commented Feb 29, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Feb 29, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 38265 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0abd85591501e83b78eb298391ecee10d357bff5, data reload: false

------ Round 1 ----------------------------------
q1	17616	4072	4072	4072
q2	2051	143	143	143
q3	10579	946	964	946
q4	4804	965	982	965
q5	7907	2923	3033	2923
q6	179	125	125	125
q7	1226	755	757	755
q8	9256	2035	2045	2035
q9	7242	6338	6308	6308
q10	8189	2539	2526	2526
q11	416	209	245	209
q12	773	316	307	307
q13	17968	3220	3211	3211
q14	283	256	244	244
q15	535	496	499	496
q16	479	398	429	398
q17	936	900	898	898
q18	6569	6074	6035	6035
q19	1583	1545	1530	1530
q20	552	288	280	280
q21	6604	3576	3576	3576
q22	810	306	283	283
Total cold run time: 106557 ms
Total hot run time: 38265 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4072	4040	4069	4040
q2	303	217	222	217
q3	2991	2912	2948	2912
q4	1856	1839	1819	1819
q5	5150	5105	5159	5105
q6	201	113	116	113
q7	2168	1707	1682	1682
q8	3161	3238	3233	3233
q9	8285	8311	8249	8249
q10	6253	3690	3666	3666
q11	508	423	445	423
q12	679	522	533	522
q13	3794	3091	3070	3070
q14	286	253	263	253
q15	537	485	497	485
q16	509	462	458	458
q17	1718	1687	1691	1687
q18	7735	7521	7551	7521
q19	1699	1655	1650	1650
q20	2142	1925	1899	1899
q21	4768	4666	4609	4609
q22	525	438	430	430
Total cold run time: 59340 ms
Total hot run time: 54043 ms

@doris-robot

doris-robot commented Feb 29, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 35.68% (8550/23964)
Line Coverage: 27.49% (69390/252457)
Region Coverage: 26.64% (35983/135077)
Branch Coverage: 23.45% (18389/78434)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0abd85591501e83b78eb298391ecee10d357bff5_0abd85591501e83b78eb298391ecee10d357bff5/report/index.html

@wm1581066 wm1581066 added the usercase Important user case type label label Feb 29, 2024
@doris-robot

doris-robot commented Feb 29, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 168776 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0abd85591501e83b78eb298391ecee10d357bff5, data reload: false

query1	915	342	331	331
query2	6544	1745	1746	1745
query3	6712	218	215	215
query4	23122	20454	20541	20454
query5	4283	363	355	355
query6	252	176	169	169
query7	4605	294	286	286
query8	259	192	211	192
query9	8487	2237	2203	2203
query10	419	215	202	202
query11	14565	14176	14206	14176
query12	134	82	83	82
query13	1642	424	426	424
query14	8369	6820	6568	6568
query15	212	172	176	172
query16	7091	262	256	256
query17	1032	520	494	494
query18	1896	262	259	259
query19	188	145	150	145
query20	90	83	87	83
query21	196	128	129	128
query22	4609	4567	4542	4542
query23	30612	29917	29887	29887
query24	11213	3140	3043	3043
query25	627	356	349	349
query26	1645	152	153	152
query27	2994	307	312	307
query28	6936	1795	1809	1795
query29	1104	547	540	540
query30	283	134	141	134
query31	864	689	690	689
query32	92	61	61	61
query33	713	221	230	221
query34	1051	475	482	475
query35	844	739	735	735
query36	925	887	847	847
query37	129	60	63	60
query38	3072	3001	2986	2986
query39	1335	1239	1244	1239
query40	284	99	100	99
query41	39	37	37	37
query42	107	101	97	97
query43	484	420	421	420
query44	1064	692	703	692
query45	198	184	173	173
query46	1031	788	771	771
query47	1469	1387	1493	1387
query48	418	346	342	342
query49	1131	290	301	290
query50	775	362	376	362
query51	4371	4261	4339	4261
query52	99	95	91	91
query53	337	269	266	266
query54	311	226	229	226
query55	89	77	86	77
query56	222	197	193	193
query57	960	899	827	827
query58	215	198	191	191
query59	2103	2113	2143	2113
query60	236	219	207	207
query61	91	84	84	84
query62	617	327	365	327
query63	285	251	258	251
query64	6618	3026	3137	3026
query65	3235	3222	3208	3208
query66	1431	328	313	313
query67	14560	14164	13817	13817
query68	5018	559	553	553
query69	544	377	372	372
query70	1236	1218	1158	1158
query71	356	250	243	243
query72	6192	2777	2640	2640
query73	722	318	316	316
query74	6288	6068	5992	5992
query75	3043	2479	2449	2449
query76	2832	995	1142	995
query77	349	234	244	234
query78	8898	8790	8416	8416
query79	1090	527	509	509
query80	665	363	361	361
query81	454	199	194	194
query82	1280	94	84	84
query83	238	123	117	117
query84	282	80	78	78
query85	1137	348	334	334
query86	309	303	301	301
query87	3291	3084	3086	3084
query88	2725	2320	2296	2296
query89	391	327	330	327
query90	2009	164	165	164
query91	150	122	122	122
query92	56	48	51	48
query93	1040	521	498	498
query94	1042	185	177	177
query95	439	350	346	346
query96	581	268	264	264
query97	4269	4102	4144	4102
query98	224	207	192	192
query99	1104	661	678	661
Total cold run time: 261095 ms
Total hot run time: 168776 ms

@doris-robot

doris-robot commented Feb 29, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0abd85591501e83b78eb298391ecee10d357bff5, data reload: false

query1	0.04	0.04	0.04
query2	0.05	0.02	0.03
query3	0.24	0.07	0.06
query4	1.67	0.10	0.09
query5	0.52	0.51	0.51
query6	1.31	0.68	0.68
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.50
query10	0.56	0.56	0.57
query11	0.13	0.10	0.10
query12	0.13	0.11	0.11
query13	0.62	0.61	0.63
query14	0.78	0.80	0.80
query15	0.84	0.82	0.81
query16	0.37	0.36	0.38
query17	1.00	0.99	0.95
query18	0.27	0.25	0.25
query19	1.79	1.70	1.68
query20	0.01	0.01	0.02
query21	15.42	0.68	0.61
query22	2.79	4.38	2.14
query23	17.65	1.11	1.01
query24	2.13	0.37	0.38
query25	0.30	0.05	0.03
query26	0.20	0.15	0.14
query27	0.04	0.03	0.03
query28	12.36	0.84	0.83
query29	12.57	3.20	3.29
query30	0.56	0.54	0.57
query31	2.78	0.34	0.37
query32	3.36	0.48	0.48
query33	3.17	3.21	3.22
query34	15.37	4.76	4.71
query35	4.75	4.76	4.76
query36	1.13	1.06	1.07
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.01	0.02
query43	0.02	0.02	0.03
Total cold run time: 105.96 s
Total hot run time: 31.91 s

@doris-robot

doris-robot commented Feb 29, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0abd85591501e83b78eb298391ecee10d357bff5 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       17.8 seconds inserted 10000000 Rows, about 561K ops/s

yujun777
yujun777 approved these changes Feb 29, 2024
View reviewed changes

@yujun777 yujun777 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei
yiguolei merged commit 95fc7a2 into apache:master Feb 29, 2024
yiguolei pushed a commit that referenced this pull request Feb 29, 2024
@liaoxin01 @Doris-Extras
[fix](load) delete bytes_appended debug point which may cause bad page (
4a05e5a 
#31602)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@liaoxin01 @HappenLee
[fix](load) delete bytes_appended debug point which may cause bad page (
f7865af 
apache#31602)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes
+1 more reviewer
@yujun777 yujun777 yujun777 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@liaoxin01 @doris-robot @yujun777 @yiguolei @xiaokang @wm1581066