Skip to content

[chore](config) disable restore_reset_index_id by default#46104

Merged
w41ter merged 1 commit intomasterfrom
disable_reset_index_id
Dec 27, 2024
Merged

[chore](config) disable restore_reset_index_id by default#46104
w41ter merged 1 commit intomasterfrom
disable_reset_index_id

Conversation

@w41ter
Copy link
Copy Markdown
Contributor

@w41ter w41ter commented Dec 27, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

otherwise the restored inverted index files is not readable.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Dec 27, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter
Copy link
Copy Markdown
Contributor Author

w41ter commented Dec 27, 2024

run buildall

airborne12
airborne12 approved these changes Dec 27, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 27, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Dec 27, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Dec 27, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Dec 27, 2024

TPC-H: Total hot run time: 33500 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae361996fc04a81f11c0bb9df5cb401c4c829af5, data reload: false

------ Round 1 ----------------------------------
q1	17648	6440	6119	6119
q2	2043	312	164	164
q3	12141	1216	750	750
q4	10217	876	445	445
q5	7510	2209	2012	2012
q6	205	177	152	152
q7	918	756	613	613
q8	9861	1384	1184	1184
q9	5309	4964	5050	4964
q10	6901	2349	1989	1989
q11	489	292	273	273
q12	362	377	244	244
q13	18351	3894	3215	3215
q14	260	245	232	232
q15	560	528	512	512
q16	683	625	617	617
q17	813	871	328	328
q18	7929	6489	6440	6440
q19	3040	994	586	586
q20	308	324	189	189
q21	2825	2290	2160	2160
q22	389	355	312	312
Total cold run time: 108762 ms
Total hot run time: 33500 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6367	6237	6115	6115
q2	244	331	237	237
q3	2313	2718	2388	2388
q4	1438	1794	1392	1392
q5	4342	5053	4885	4885
q6	185	177	143	143
q7	2106	1934	1818	1818
q8	2607	2868	2686	2686
q9	7293	7268	7291	7268
q10	2993	3355	2849	2849
q11	615	524	498	498
q12	692	753	600	600
q13	3441	3797	3146	3146
q14	289	313	287	287
q15	564	524	518	518
q16	667	671	641	641
q17	1224	1735	1275	1275
q18	7741	7422	7314	7314
q19	909	1232	1083	1083
q20	2027	2000	1908	1908
q21	5722	5120	5097	5097
q22	618	629	605	605
Total cold run time: 54397 ms
Total hot run time: 52753 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Dec 27, 2024

TPC-DS: Total hot run time: 197019 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae361996fc04a81f11c0bb9df5cb401c4c829af5, data reload: false

query1	1306	985	916	916
query2	6413	2372	2273	2273
query3	10973	4820	4831	4820
query4	32976	24018	23400	23400
query5	4314	595	457	457
query6	311	207	187	187
query7	4009	500	296	296
query8	293	238	239	238
query9	9606	2718	2712	2712
query10	439	299	254	254
query11	17870	15364	15440	15364
query12	152	103	102	102
query13	1581	536	401	401
query14	10539	7144	7203	7144
query15	243	209	186	186
query16	8093	586	478	478
query17	1522	743	592	592
query18	2146	403	329	329
query19	198	195	162	162
query20	125	114	125	114
query21	243	126	117	117
query22	4717	4660	4707	4660
query23	35165	33175	34170	33175
query24	6379	2298	2414	2298
query25	492	451	398	398
query26	947	278	166	166
query27	2252	467	337	337
query28	5470	2473	2481	2473
query29	707	585	446	446
query30	213	196	154	154
query31	973	986	860	860
query32	93	58	62	58
query33	492	357	319	319
query34	787	866	530	530
query35	841	834	798	798
query36	1025	1065	951	951
query37	120	103	78	78
query38	4332	4116	4187	4116
query39	1556	1452	1474	1452
query40	208	112	99	99
query41	44	45	45	45
query42	127	104	101	101
query43	525	533	506	506
query44	1322	843	849	843
query45	182	189	176	176
query46	919	1093	685	685
query47	1998	2037	1939	1939
query48	399	426	333	333
query49	710	486	399	399
query50	677	683	402	402
query51	7290	7150	7227	7150
query52	105	103	88	88
query53	230	256	184	184
query54	495	531	422	422
query55	86	79	80	79
query56	269	259	245	245
query57	1253	1258	1160	1160
query58	235	225	225	225
query59	3019	3303	3197	3197
query60	272	257	244	244
query61	118	125	107	107
query62	869	821	780	780
query63	241	201	217	201
query64	3912	1042	660	660
query65	3298	3266	3291	3266
query66	759	412	307	307
query67	16310	15919	15598	15598
query68	10069	750	520	520
query69	493	293	248	248
query70	1221	1139	1171	1139
query71	437	275	272	272
query72	6092	3788	3836	3788
query73	672	736	368	368
query74	10597	9117	9110	9110
query75	4544	3159	2658	2658
query76	5439	1168	757	757
query77	1029	349	270	270
query78	10087	10105	9500	9500
query79	5599	868	671	671
query80	765	520	408	408
query81	483	259	233	233
query82	345	151	115	115
query83	199	156	158	156
query84	284	93	72	72
query85	743	357	297	297
query86	352	307	258	258
query87	4459	4411	4434	4411
query88	3676	2227	2195	2195
query89	448	334	298	298
query90	2085	187	184	184
query91	184	127	111	111
query92	63	54	49	49
query93	3618	893	513	513
query94	688	370	278	278
query95	326	260	249	249
query96	489	617	284	284
query97	2742	2787	2704	2704
query98	222	194	192	192
query99	1663	1591	1459	1459
Total cold run time: 305898 ms
Total hot run time: 197019 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Dec 27, 2024

ClickBench: Total hot run time: 31.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae361996fc04a81f11c0bb9df5cb401c4c829af5, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.61	0.10	0.10
query5	0.41	0.42	0.43
query6	1.14	0.65	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.50
query10	0.56	0.58	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	2.84	2.82	2.82
query15	0.89	0.83	0.83
query16	0.38	0.39	0.36
query17	1.05	1.01	1.06
query18	0.22	0.20	0.20
query19	1.94	1.78	2.01
query20	0.01	0.01	0.01
query21	15.36	0.99	0.60
query22	0.77	0.74	0.65
query23	15.36	1.47	0.58
query24	2.95	0.97	1.70
query25	0.21	0.26	0.08
query26	0.31	0.14	0.13
query27	0.07	0.04	0.05
query28	13.96	1.56	1.05
query29	12.56	3.91	3.21
query30	0.24	0.09	0.06
query31	2.82	0.60	0.39
query32	3.23	0.54	0.45
query33	3.07	3.08	3.06
query34	16.81	5.19	4.45
query35	4.53	4.46	4.45
query36	0.68	0.52	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.36 s
Total hot run time: 31.18 s

dataroaring
dataroaring approved these changes Dec 27, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@w41ter w41ter merged commit 314b708 into master Dec 27, 2024
@CalvinKirs CalvinKirs deleted the disable_reset_index_id branch December 28, 2024 10:21
github-actions bot pushed a commit that referenced this pull request Dec 28, 2024
@w41ter
[chore](config) disable restore_reset_index_id by default (#46104)
9b0dd81 
otherwise, the restored inverted index files are not readable.
@github-actions github-actions bot mentioned this pull request Dec 28, 2024
Merged
github-actions bot pushed a commit that referenced this pull request Dec 28, 2024
@w41ter
[chore](config) disable restore_reset_index_id by default (#46104)
e9d8143 
otherwise, the restored inverted index files are not readable.
@github-actions github-actions bot mentioned this pull request Dec 28, 2024
Merged
w41ter added a commit that referenced this pull request Dec 30, 2024
@github-actions @w41ter
branch-3.0: [chore](config) disable restore_reset_index_id by default #…
3369a27 
…46104 (#46126)

Cherry-picked from #46104

Co-authored-by: walter <maochuan@selectdb.com>
w41ter added a commit that referenced this pull request Dec 30, 2024
@github-actions @w41ter
branch-2.1: [chore](config) disable restore_reset_index_id by default #…
c8fb934 
…46104 (#46127)

Cherry-picked from #46104

Co-authored-by: walter <maochuan@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@w41ter @hello-stephen @doris-robot @airborne12 @dataroaring @CalvinKirs