Skip to content

[chore](persist) save base index id & index names in TableAddOrDropColumnsInfo#47881

Merged
gavinchou merged 1 commit intoapache:masterfrom
w41ter:save_base_index_id
Feb 14, 2025
Merged

[chore](persist) save base index id & index names in TableAddOrDropColumnsInfo#47881
gavinchou merged 1 commit intoapache:masterfrom
w41ter:save_base_index_id

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Feb 13, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: https://github.com/selectdb/ccr-syncer

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter w41ter changed the title [chore](persist) save base index id in TableAddOrDropColumnsInfo [chore](persist) save base index id & index names in TableAddOrDropColumnsInfo Feb 13, 2025
@w41ter
Copy link
Contributor Author

w41ter commented Feb 13, 2025

run buildall

@w41ter w41ter force-pushed the save_base_index_id branch from cce43f3 to c730882 Compare February 13, 2025 09:37
@w41ter w41ter force-pushed the save_base_index_id branch from c730882 to 95fd863 Compare February 13, 2025 09:40
@w41ter
Copy link
Contributor Author

w41ter commented Feb 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 13, 2025

TPC-H: Total hot run time: 31654 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 95fd8636d80f2b7c9965ed6707b1c3c294f0479e, data reload: false

------ Round 1 ----------------------------------
q1	18143	5244	5072	5072
q2	2052	291	168	168
q3	10428	1253	740	740
q4	10200	1030	534	534
q5	7531	2388	2355	2355
q6	187	168	137	137
q7	905	762	609	609
q8	9322	1321	1075	1075
q9	5039	4736	4759	4736
q10	6869	2332	1875	1875
q11	505	289	259	259
q12	350	354	224	224
q13	17779	3674	3124	3124
q14	221	235	211	211
q15	519	464	451	451
q16	630	623	587	587
q17	560	869	342	342
q18	6967	6179	6154	6154
q19	1568	967	562	562
q20	326	323	185	185
q21	2825	2133	1946	1946
q22	366	341	308	308
Total cold run time: 103292 ms
Total hot run time: 31654 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5150	5336	5142	5142
q2	241	340	229	229
q3	2165	2664	2272	2272
q4	1438	1822	1358	1358
q5	4218	4146	4179	4146
q6	206	166	125	125
q7	1916	1837	1759	1759
q8	2697	2715	2617	2617
q9	7185	7139	7176	7139
q10	3028	3250	2819	2819
q11	585	521	499	499
q12	677	762	621	621
q13	3475	3944	3311	3311
q14	274	309	283	283
q15	516	469	466	466
q16	673	701	641	641
q17	1157	1632	1390	1390
q18	7489	7345	7380	7345
q19	821	890	913	890
q20	1989	2021	1908	1908
q21	5407	5101	4683	4683
q22	638	585	613	585
Total cold run time: 51945 ms
Total hot run time: 50228 ms

@doris-robot
Copy link

doris-robot commented Feb 13, 2025

TPC-DS: Total hot run time: 189826 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 95fd8636d80f2b7c9965ed6707b1c3c294f0479e, data reload: false

query1	1328	946	938	938
query2	6213	1815	1824	1815
query3	11100	4540	4618	4540
query4	25743	23792	23029	23029
query5	4434	642	491	491
query6	307	198	187	187
query7	3992	501	313	313
query8	299	243	236	236
query9	8518	2564	2569	2564
query10	467	310	268	268
query11	15505	15329	14909	14909
query12	166	106	106	106
query13	1573	516	400	400
query14	9690	6437	6139	6139
query15	218	195	178	178
query16	7641	634	456	456
query17	1195	775	610	610
query18	2039	424	329	329
query19	198	188	186	186
query20	127	122	132	122
query21	207	126	108	108
query22	4627	4859	4480	4480
query23	34196	33740	33328	33328
query24	7848	2448	2402	2402
query25	508	464	403	403
query26	1221	273	157	157
query27	2330	487	343	343
query28	4440	2455	2413	2413
query29	661	566	428	428
query30	222	193	168	168
query31	904	904	826	826
query32	75	64	73	64
query33	546	351	297	297
query34	781	870	519	519
query35	804	862	790	790
query36	1012	1027	910	910
query37	120	100	78	78
query38	4329	4253	4267	4253
query39	1492	1433	1454	1433
query40	200	118	105	105
query41	51	51	50	50
query42	125	98	101	98
query43	484	508	478	478
query44	1352	810	798	798
query45	176	169	175	169
query46	862	1047	693	693
query47	1804	1842	1818	1818
query48	397	422	314	314
query49	746	553	441	441
query50	703	742	426	426
query51	4245	4348	4220	4220
query52	104	101	101	101
query53	223	264	190	190
query54	487	508	420	420
query55	86	82	77	77
query56	287	265	271	265
query57	1167	1193	1138	1138
query58	256	247	235	235
query59	2711	2720	2547	2547
query60	294	288	264	264
query61	119	120	116	116
query62	783	723	694	694
query63	228	186	211	186
query64	4093	1055	702	702
query65	3333	3188	3233	3188
query66	974	398	293	293
query67	16317	15523	15450	15450
query68	8274	763	508	508
query69	522	292	254	254
query70	1191	1141	1118	1118
query71	467	287	257	257
query72	5134	3601	3773	3601
query73	750	720	348	348
query74	8922	9173	9034	9034
query75	3798	3125	2695	2695
query76	3676	1238	748	748
query77	796	358	274	274
query78	9872	10094	9362	9362
query79	2681	798	610	610
query80	645	526	451	451
query81	491	280	232	232
query82	643	126	94	94
query83	193	236	149	149
query84	286	88	71	71
query85	768	356	300	300
query86	340	314	294	294
query87	4500	4401	4246	4246
query88	3313	2207	2192	2192
query89	413	310	282	282
query90	1929	188	189	188
query91	133	149	108	108
query92	71	60	55	55
query93	1214	1005	583	583
query94	651	413	295	295
query95	346	271	261	261
query96	477	567	269	269
query97	2825	2820	2735	2735
query98	235	212	202	202
query99	1439	1403	1318	1318
Total cold run time: 276678 ms
Total hot run time: 189826 ms

@doris-robot
Copy link

doris-robot commented Feb 13, 2025

ClickBench: Total hot run time: 30.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 95fd8636d80f2b7c9965ed6707b1c3c294f0479e, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.40	0.41	0.39
query6	1.17	0.67	0.66
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.52
query10	0.57	0.57	0.57
query11	0.15	0.10	0.10
query12	0.15	0.12	0.12
query13	0.62	0.59	0.60
query14	2.67	2.69	2.71
query15	0.93	0.86	0.84
query16	0.38	0.37	0.40
query17	0.98	1.06	1.02
query18	0.22	0.20	0.20
query19	1.89	1.84	1.99
query20	0.01	0.01	0.01
query21	15.34	0.90	0.57
query22	0.78	1.08	0.72
query23	14.96	1.41	0.62
query24	7.98	4.88	0.59
query25	0.33	0.19	0.20
query26	1.10	0.17	0.13
query27	0.05	0.06	0.04
query28	6.23	0.79	0.42
query29	12.56	3.91	3.26
query30	0.25	0.09	0.06
query31	2.82	0.57	0.38
query32	3.21	0.55	0.45
query33	3.03	3.04	3.04
query34	15.79	5.17	4.53
query35	4.53	4.53	4.61
query36	0.67	0.50	0.48
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 102.88 s
Total hot run time: 30.51 s

dataroaring
dataroaring approved these changes Feb 14, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 14, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Feb 14, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 14, 2025

PR approved by anyone and no changes requested.

@gavinchou gavinchou merged commit b7ace78 into apache:master Feb 14, 2025
28 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Feb 14, 2025
@w41ter
[chore](persist) save base index id & index names in TableAddOrDropCo…
0f2b35d 
…lumnsInfo (#47881)

Related PR: https://github.com/selectdb/ccr-syncer
@github-actions github-actions bot mentioned this pull request Feb 14, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Feb 14, 2025
@w41ter
[chore](persist) save base index id & index names in TableAddOrDropCo…
abb49be 
…lumnsInfo (#47881)

Related PR: https://github.com/selectdb/ccr-syncer
@github-actions github-actions bot mentioned this pull request Feb 14, 2025
Merged
@w41ter w41ter deleted the save_base_index_id branch February 14, 2025 13:21
yiguolei pushed a commit that referenced this pull request Feb 16, 2025
@github-actions @w41ter
branch-2.1: [chore](persist) save base index id & index names in Tabl…
056022b 
…eAddOrDropColumnsInfo #47881 (#47936)

Cherry-picked from #47881

Co-authored-by: walter <maochuan@selectdb.com>
dataroaring pushed a commit that referenced this pull request Feb 19, 2025
@github-actions @w41ter
branch-3.0: [chore](persist) save base index id & index names in Tabl…
0a461e4 
…eAddOrDropColumnsInfo #47881 (#47935)

Cherry-picked from #47881

Co-authored-by: walter <maochuan@selectdb.com>
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@w41ter @lzyy2024
[chore](persist) save base index id & index names in TableAddOrDropCo…
0f26385 
…lumnsInfo (apache#47881)

Related PR: https://github.com/selectdb/ccr-syncer
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@w41ter
[chore](persist) save base index id & index names in TableAddOrDropCo…
af66b6c 
…lumnsInfo (apache#47881)

Related PR: https://github.com/selectdb/ccr-syncer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@w41ter @Thearas @doris-robot @gavinchou @dataroaring @yiguolei