Skip to content

Comments

[feat](binlog) filter the temp table related binlogs#49755

Merged
dataroaring merged 1 commit intoapache:masterfrom
koarz:filter_temp_table_binlog
Apr 15, 2025
Merged

[feat](binlog) filter the temp table related binlogs#49755
dataroaring merged 1 commit intoapache:masterfrom
koarz:filter_temp_table_binlog

Conversation

@koarz
Copy link
Contributor

@koarz koarz commented Apr 2, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@koarz
Copy link
Contributor Author

koarz commented Apr 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 2, 2025

TPC-H: Total hot run time: 34119 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a3eb0e7bb090f101dc4de0e92931be9b66c93aaf, data reload: false

------ Round 1 ----------------------------------
q1	25976	5092	5072	5072
q2	2060	294	186	186
q3	9139	1260	712	712
q4	10082	996	529	529
q5	7651	2400	2316	2316
q6	191	162	130	130
q7	930	732	598	598
q8	9278	1300	1063	1063
q9	6952	5138	5137	5137
q10	6888	2313	1883	1883
q11	477	286	277	277
q12	347	370	223	223
q13	3677	3720	3113	3113
q14	235	232	214	214
q15	529	483	475	475
q16	651	623	588	588
q17	587	854	353	353
q18	7451	7347	7065	7065
q19	1189	937	558	558
q20	339	328	228	228
q21	4000	3320	2418	2418
q22	1110	1022	981	981
Total cold run time: 99739 ms
Total hot run time: 34119 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5185	5163	5155	5155
q2	249	323	231	231
q3	2139	2621	2316	2316
q4	1447	1813	1384	1384
q5	4195	4094	4158	4094
q6	209	163	121	121
q7	1835	1899	1739	1739
q8	2577	2577	2481	2481
q9	7410	7299	7273	7273
q10	3025	3231	2762	2762
q11	579	512	487	487
q12	683	734	561	561
q13	3546	3942	3240	3240
q14	287	318	269	269
q15	531	495	485	485
q16	650	692	648	648
q17	1155	1563	1392	1392
q18	7753	7487	7498	7487
q19	800	831	850	831
q20	1989	2054	1887	1887
q21	5513	4868	4745	4745
q22	1140	1064	1009	1009
Total cold run time: 52897 ms
Total hot run time: 50597 ms

@doris-robot
Copy link

doris-robot commented Apr 2, 2025

TPC-DS: Total hot run time: 194228 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a3eb0e7bb090f101dc4de0e92931be9b66c93aaf, data reload: false

query1	1407	1083	1048	1048
query2	6262	1973	1913	1913
query3	11154	4710	4567	4567
query4	54056	24569	22900	22900
query5	5111	581	488	488
query6	361	212	200	200
query7	4873	507	293	293
query8	326	256	246	246
query9	5577	2599	2593	2593
query10	461	324	271	271
query11	15110	14969	14989	14969
query12	176	114	109	109
query13	1046	527	414	414
query14	10171	6401	6514	6401
query15	208	192	182	182
query16	7139	682	483	483
query17	1068	745	600	600
query18	1541	412	329	329
query19	193	199	158	158
query20	128	129	124	124
query21	219	125	120	120
query22	4468	4611	4489	4489
query23	34150	33412	33442	33412
query24	6649	2475	2462	2462
query25	474	470	400	400
query26	684	275	158	158
query27	2317	521	339	339
query28	3275	2474	2469	2469
query29	597	570	445	445
query30	276	220	193	193
query31	861	896	813	813
query32	75	68	70	68
query33	482	364	308	308
query34	767	863	542	542
query35	798	846	777	777
query36	964	1015	911	911
query37	121	107	86	86
query38	4072	4217	4278	4217
query39	1495	1512	1415	1415
query40	222	135	126	126
query41	61	60	62	60
query42	138	117	109	109
query43	524	529	497	497
query44	1349	850	821	821
query45	181	175	170	170
query46	843	1036	659	659
query47	1843	1877	1821	1821
query48	383	423	312	312
query49	654	508	478	478
query50	683	700	415	415
query51	4253	4260	4458	4260
query52	120	113	106	106
query53	242	275	191	191
query54	582	583	516	516
query55	88	89	80	80
query56	347	311	308	308
query57	1205	1219	1131	1131
query58	281	275	266	266
query59	2747	2797	2757	2757
query60	353	350	315	315
query61	133	129	153	129
query62	727	745	663	663
query63	233	200	202	200
query64	1647	1051	723	723
query65	4526	4440	4442	4440
query66	729	397	307	307
query67	16023	15536	15308	15308
query68	7022	886	519	519
query69	541	302	264	264
query70	1206	1124	1075	1075
query71	481	333	282	282
query72	5925	4859	4893	4859
query73	1310	688	351	351
query74	9142	9199	8937	8937
query75	3720	3239	2743	2743
query76	4107	1194	771	771
query77	699	395	286	286
query78	10047	10031	9319	9319
query79	2645	834	581	581
query80	694	535	453	453
query81	484	261	229	229
query82	532	121	99	99
query83	260	257	241	241
query84	299	102	90	90
query85	839	430	398	398
query86	378	308	282	282
query87	4759	4423	4332	4332
query88	3528	2250	2309	2250
query89	417	326	285	285
query90	1774	224	226	224
query91	147	157	114	114
query92	79	64	57	57
query93	2225	979	582	582
query94	687	396	316	316
query95	374	306	294	294
query96	480	582	273	273
query97	3187	3249	3169	3169
query98	237	208	205	205
query99	1429	1402	1268	1268
Total cold run time: 299883 ms
Total hot run time: 194228 ms

@doris-robot
Copy link

doris-robot commented Apr 2, 2025

ClickBench: Total hot run time: 30.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a3eb0e7bb090f101dc4de0e92931be9b66c93aaf, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.10
query3	0.24	0.19	0.19
query4	1.60	0.19	0.11
query5	0.54	0.54	0.55
query6	1.18	0.72	0.71
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.57	0.53	0.50
query10	0.60	0.60	0.57
query11	0.16	0.10	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.68	2.81	2.69
query15	0.92	0.84	0.84
query16	0.38	0.39	0.38
query17	1.02	1.02	1.05
query18	0.21	0.20	0.21
query19	2.08	1.83	1.81
query20	0.01	0.01	0.02
query21	15.38	0.89	0.55
query22	0.76	1.33	0.67
query23	14.76	1.40	0.58
query24	7.35	2.11	0.52
query25	0.52	0.19	0.06
query26	0.67	0.16	0.14
query27	0.05	0.05	0.05
query28	9.13	0.89	0.45
query29	12.55	4.01	3.30
query30	0.26	0.09	0.07
query31	2.83	0.58	0.39
query32	3.22	0.55	0.49
query33	3.07	3.11	3.09
query34	15.73	5.14	4.50
query35	4.52	4.54	4.54
query36	0.67	0.50	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.12 s
Total hot run time: 30.75 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2025

PR approved by anyone and no changes requested.

@w41ter w41ter changed the title [feat](binlog)filter the temporary binlogs [feat](binlog) filter the temp table related binlogs Apr 3, 2025
@w41ter w41ter requested a review from dataroaring April 11, 2025 07:40
dataroaring
dataroaring approved these changes Apr 15, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 991eaec into apache:master Apr 15, 2025
26 of 28 checks passed
@koarz koarz deleted the filter_temp_table_binlog branch April 16, 2025 02:33
seawinde pushed a commit to seawinde/doris that referenced this pull request Apr 17, 2025
@seawinde
[feat](binlog) filter the temp table related binlogs (apache#49755)
76841c3
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
[feat](binlog) filter the temp table related binlogs (apache#49755)
7cac1c8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@w41ter w41ter w41ter approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@koarz @hello-stephen @doris-robot @w41ter @dataroaring