[Fix](core) Fix null ptr introduced by #42949 by Yukang-Lian · Pull Request #46074 · apache/doris

Yukang-Lian · 2024-12-27T03:07:40Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #42949

Problem Summary:

In PR #42949, during the rowset ID initialization process, we used a random ID to replace the rowset ID that failed during serialization. However, the generation of random IDs depends on the storage engine, which hasn't been initialized during the rowset ID initialization process, leading to a core dump. This PR fixes this issue by uniformly using MAX_ROWSET_ID-1 to replace the failed rowset ID. This approach is safe because the rowset ID generator won't generate such a large ID, and we can consider all rowsets with rowset ID equal to MAX_ROWSET_ID-1 as failed initialization rowsets that should rely on multiple replicas for automatic recovery.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2024-12-27T03:07:45Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Yukang-Lian · 2024-12-27T03:07:51Z

run buildall

Yukang-Lian · 2024-12-27T03:10:38Z

run buildall

doris-robot · 2024-12-27T04:37:05Z

TPC-H: Total hot run time: 32534 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f123b37324b8cfd79afa4a81a62f5f359f75d27d, data reload: false

------ Round 1 ----------------------------------
q1	17573	6265	6080	6080
q2	2046	305	166	166
q3	10499	1302	704	704
q4	10219	873	435	435
q5	8065	2216	1965	1965
q6	205	182	146	146
q7	910	739	628	628
q8	9233	1369	1169	1169
q9	5282	4950	5032	4950
q10	6737	2310	1870	1870
q11	511	279	264	264
q12	346	367	220	220
q13	17797	3665	2998	2998
q14	242	239	211	211
q15	556	511	495	495
q16	623	605	582	582
q17	555	857	326	326
q18	7164	6581	6357	6357
q19	2852	981	548	548
q20	298	315	185	185
q21	2838	2142	1931	1931
q22	358	327	304	304
Total cold run time: 104909 ms
Total hot run time: 32534 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6398	6277	6250	6250
q2	232	321	229	229
q3	2224	2642	2342	2342
q4	1468	1869	1420	1420
q5	4345	4785	5038	4785
q6	189	176	137	137
q7	2148	1950	1772	1772
q8	2599	2790	2647	2647
q9	7380	7249	7328	7249
q10	3048	3354	2812	2812
q11	587	502	493	493
q12	654	755	597	597
q13	3350	3812	3116	3116
q14	300	300	283	283
q15	577	511	491	491
q16	640	673	616	616
q17	1210	1733	1261	1261
q18	7704	7327	7250	7250
q19	783	1079	1037	1037
q20	1932	1995	1813	1813
q21	5412	5175	4840	4840
q22	592	584	602	584
Total cold run time: 53772 ms
Total hot run time: 52024 ms

doris-robot · 2024-12-27T04:49:07Z

TPC-DS: Total hot run time: 188964 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f123b37324b8cfd79afa4a81a62f5f359f75d27d, data reload: false

query1	961	381	392	381
query2	6523	2408	2292	2292
query3	6709	216	216	216
query4	34064	23868	23112	23112
query5	4399	612	467	467
query6	293	202	181	181
query7	4656	481	317	317
query8	281	240	214	214
query9	9381	2727	2708	2708
query10	471	300	239	239
query11	18173	15322	15116	15116
query12	142	101	101	101
query13	1667	525	414	414
query14	10194	6713	7404	6713
query15	226	196	189	189
query16	8919	611	404	404
query17	1609	756	559	559
query18	2137	407	293	293
query19	224	189	156	156
query20	122	112	113	112
query21	215	122	105	105
query22	4289	4318	4158	4158
query23	34214	33342	33603	33342
query24	6600	2244	2219	2219
query25	496	446	385	385
query26	1063	268	146	146
query27	2094	451	341	341
query28	5426	2477	2426	2426
query29	743	524	412	412
query30	227	181	147	147
query31	971	871	810	810
query32	90	58	55	55
query33	502	336	288	288
query34	744	831	484	484
query35	787	802	755	755
query36	1025	1032	940	940
query37	114	97	74	74
query38	4133	4184	4140	4140
query39	1476	1439	1450	1439
query40	203	114	102	102
query41	47	46	46	46
query42	112	99	103	99
query43	512	529	478	478
query44	1292	787	812	787
query45	175	190	172	172
query46	841	1023	638	638
query47	1899	1897	1861	1861
query48	368	396	320	320
query49	749	463	378	378
query50	604	653	376	376
query51	7162	6971	7108	6971
query52	96	99	89	89
query53	220	248	190	190
query54	468	479	395	395
query55	82	81	77	77
query56	256	252	232	232
query57	1202	1167	1133	1133
query58	225	218	228	218
query59	2963	3115	2988	2988
query60	289	256	252	252
query61	106	107	103	103
query62	868	806	733	733
query63	227	190	189	189
query64	4289	1003	634	634
query65	3261	3195	3209	3195
query66	1051	412	296	296
query67	15964	15737	15452	15452
query68	9489	754	510	510
query69	454	286	252	252
query70	1229	1135	1168	1135
query71	424	285	251	251
query72	5773	3756	3812	3756
query73	677	763	354	354
query74	9698	8967	8836	8836
query75	4532	3138	2627	2627
query76	5184	1188	750	750
query77	862	354	264	264
query78	10116	10234	9927	9927
query79	3189	886	592	592
query80	708	508	432	432
query81	479	270	234	234
query82	577	151	122	122
query83	198	161	146	146
query84	285	88	72	72
query85	775	358	295	295
query86	354	311	285	285
query87	4384	4596	4354	4354
query88	4194	2233	2220	2220
query89	396	340	295	295
query90	1939	187	190	187
query91	138	143	105	105
query92	67	57	54	54
query93	1237	894	525	525
query94	668	392	288	288
query95	335	263	259	259
query96	491	605	281	281
query97	2810	2795	2701	2701
query98	246	209	191	191
query99	1707	1528	1410	1410
Total cold run time: 295506 ms
Total hot run time: 188964 ms

doris-robot · 2024-12-27T04:54:28Z

ClickBench: Total hot run time: 31.92 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f123b37324b8cfd79afa4a81a62f5f359f75d27d, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.04	0.03
query3	0.24	0.06	0.07
query4	1.61	0.10	0.10
query5	0.43	0.40	0.39
query6	1.16	0.65	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.51	0.52
query10	0.54	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.59
query14	2.84	2.84	2.75
query15	0.90	0.83	0.82
query16	0.37	0.38	0.39
query17	1.01	1.02	1.07
query18	0.22	0.22	0.21
query19	1.90	1.73	1.95
query20	0.02	0.01	0.01
query21	15.38	0.93	0.60
query22	0.74	0.74	0.71
query23	15.32	1.42	0.51
query24	2.93	1.73	1.87
query25	0.20	0.25	0.11
query26	0.17	0.13	0.14
query27	0.05	0.06	0.05
query28	14.32	1.42	1.04
query29	12.61	3.90	3.23
query30	0.25	0.10	0.07
query31	2.81	0.57	0.38
query32	3.23	0.56	0.45
query33	3.10	3.10	3.03
query34	16.91	5.15	4.53
query35	4.50	4.49	4.51
query36	0.65	0.49	0.47
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.53 s
Total hot run time: 31.92 s

Yukang-Lian · 2024-12-27T08:14:41Z

run beut

Yukang-Lian · 2024-12-27T08:14:50Z

run external

doris-robot · 2024-12-27T08:52:49Z

TeamCity be ut coverage result:
Function Coverage: 38.89% (10119/26021)
Line Coverage: 29.88% (85493/286099)
Region Coverage: 29.02% (43695/150559)
Branch Coverage: 25.56% (22293/87212)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f123b37324b8cfd79afa4a81a62f5f359f75d27d_f123b37324b8cfd79afa4a81a62f5f359f75d27d/report/index.html

github-actions · 2024-12-27T09:45:45Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-27T09:45:47Z

PR approved by anyone and no changes requested.

Hastyshell

LGTM

dataroaring

LGTM

In PR apache#42949, during the rowset ID initialization process, we used a random ID to replace the rowset ID that failed during serialization. However, the generation of random IDs depends on the storage engine, which hasn't been initialized during the rowset ID initialization process, leading to a core dump. This PR fixes this issue by uniformly using MAX_ROWSET_ID-1 to replace the failed rowset ID. This approach is safe because the rowset ID generator won't generate such a large ID, and we can consider all rowsets with rowset ID equal to MAX_ROWSET_ID-1 as failed initialization rowsets that should rely on multiple replicas for automatic recovery.

…#42949 (#46074)" (#48346) Pick #46074 In PR #42949, during the rowset ID initialization process, we used a random ID to replace the rowset ID that failed during serialization. However, the generation of random IDs depends on the storage engine, which hasn't been initialized during the rowset ID initialization process, leading to a core dump. This PR fixes this issue by uniformly using MAX_ROWSET_ID-1 to replace the failed rowset ID. This approach is safe because the rowset ID generator won't generate such a large ID, and we can consider all rowsets with rowset ID equal to MAX_ROWSET_ID-1 as failed initialization rowsets that should rely on multiple replicas for automatic recovery.

In PR apache#42949, during the rowset ID initialization process, we used a random ID to replace the rowset ID that failed during serialization. However, the generation of random IDs depends on the storage engine, which hasn't been initialized during the rowset ID initialization process, leading to a core dump. This PR fixes this issue by uniformly using MAX_ROWSET_ID-1 to replace the failed rowset ID. This approach is safe because the rowset ID generator won't generate such a large ID, and we can consider all rowsets with rowset ID equal to MAX_ROWSET_ID-1 as failed initialization rowsets that should rely on multiple replicas for automatic recovery.

…#42949 (#46074)" (#53865)

Yukang-Lian added 2 commits December 27, 2024 10:25

1

1f88e63

2

7b12d16

3

f123b37

Yukang-Lian mentioned this pull request Dec 27, 2024

[cherry-pick](branch-2.0) Pick "[Fix](Rowset Id) Use a randomly generated rowset ID to handle memory write failures (#42949) #46102

Merged

16 tasks

gavinchou approved these changes Dec 27, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 27, 2024

github-actions bot added the reviewed label Dec 27, 2024

Hastyshell approved these changes Dec 27, 2024

View reviewed changes

dataroaring added dev/2.1.x dev/3.0.x labels Dec 27, 2024

dataroaring approved these changes Dec 27, 2024

View reviewed changes

gavinchou merged commit 2c7b883 into apache:master Dec 27, 2024
19 of 21 checks passed

gavinchou added the p0_c label Feb 25, 2025

Yukang-Lian mentioned this pull request Feb 26, 2025

[cherry-pick](branch-3.0) Pick "[Fix](core) Fix null ptr introduced by #42949 (#46074)" #48346

Merged

16 tasks

dataroaring added dev/3.0.5-merged and removed p0_c dev/3.0.x labels Feb 26, 2025

Yukang-Lian mentioned this pull request Jul 24, 2025

[Cherry-Pick](branch-2.1) Pick "[Fix](core) Fix null ptr introduced by #42949 (#46074)" #53865

Merged

16 tasks

yiguolei pushed a commit that referenced this pull request Jul 26, 2025

[Cherry-Pick](branch-2.1) Pick "[Fix](core) Fix null ptr introduced by …

101ea84

…#42949 (#46074)" (#53865)

yiguolei added dev/2.1.11-merged and removed dev/2.1.x labels Jul 26, 2025

Conversation

Yukang-Lian commented Dec 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Dec 27, 2024

Uh oh!

Yukang-Lian commented Dec 27, 2024

Uh oh!

Yukang-Lian commented Dec 27, 2024

Uh oh!

doris-robot commented Dec 27, 2024

Uh oh!

doris-robot commented Dec 27, 2024

Uh oh!

doris-robot commented Dec 27, 2024

Uh oh!

Yukang-Lian commented Dec 27, 2024

Uh oh!

Yukang-Lian commented Dec 27, 2024

Uh oh!

doris-robot commented Dec 27, 2024

Uh oh!

github-actions bot commented Dec 27, 2024

Uh oh!

github-actions bot commented Dec 27, 2024

Uh oh!

Hastyshell left a comment

Choose a reason for hiding this comment

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Yukang-Lian commented Dec 27, 2024 •

edited

Loading