Skip to content

[log](mow) optimize some log for mow table #51093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 21, 2025

Conversation

zhannngchen
Copy link
Contributor

@zhannngchen zhannngchen commented May 20, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented May 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhannngchen
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 100.00% (7/7) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.31% (1113/1336)
Line Coverage 66.27% (18681/28191)
Region Coverage 65.81% (9263/14076)
Branch Coverage 55.67% (4990/8964)

@doris-robot
Copy link

TPC-H: Total hot run time: 33659 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1808eb8a59d488ac31776634ecb5959708985fb9, data reload: false

------ Round 1 ----------------------------------
q1	26259	5052	4996	4996
q2	2083	289	177	177
q3	10388	1240	677	677
q4	10235	982	525	525
q5	7490	2380	2324	2324
q6	175	162	135	135
q7	909	730	613	613
q8	9314	1226	1104	1104
q9	6852	5062	5085	5062
q10	6865	2303	1855	1855
q11	486	284	292	284
q12	348	344	207	207
q13	17769	3652	3071	3071
q14	229	237	219	219
q15	536	479	486	479
q16	431	434	372	372
q17	595	834	363	363
q18	7473	7287	7184	7184
q19	1608	942	543	543
q20	324	343	224	224
q21	3632	2497	2278	2278
q22	1067	1005	967	967
Total cold run time: 115068 ms
Total hot run time: 33659 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5093	5020	5081	5020
q2	229	320	230	230
q3	2149	2594	2328	2328
q4	1350	1795	1320	1320
q5	4361	4291	4397	4291
q6	217	170	126	126
q7	2006	1936	1746	1746
q8	2578	2569	2521	2521
q9	7234	7293	6979	6979
q10	3018	3216	2791	2791
q11	571	521	495	495
q12	666	782	628	628
q13	3478	3915	3268	3268
q14	285	307	275	275
q15	522	479	487	479
q16	459	471	480	471
q17	1138	1551	1349	1349
q18	7644	7415	7548	7415
q19	797	824	995	824
q20	1973	2040	1942	1942
q21	4889	4379	4388	4379
q22	1098	1053	1014	1014
Total cold run time: 51755 ms
Total hot run time: 49891 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194006 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1808eb8a59d488ac31776634ecb5959708985fb9, data reload: false

query1	1415	1055	1052	1052
query2	6210	1804	1789	1789
query3	11019	4598	4436	4436
query4	53658	26219	23413	23413
query5	5252	517	444	444
query6	391	208	215	208
query7	5032	508	300	300
query8	327	258	240	240
query9	6042	2633	2598	2598
query10	452	331	271	271
query11	15314	15108	14967	14967
query12	168	112	114	112
query13	1122	535	415	415
query14	10081	6406	6345	6345
query15	212	208	183	183
query16	7532	677	505	505
query17	2389	782	638	638
query18	1533	432	324	324
query19	211	197	170	170
query20	142	127	136	127
query21	212	130	112	112
query22	4462	4461	4409	4409
query23	34525	34531	34432	34432
query24	7565	2431	2460	2431
query25	529	466	422	422
query26	748	275	152	152
query27	2604	515	357	357
query28	3410	2204	2156	2156
query29	653	577	449	449
query30	338	224	194	194
query31	934	875	804	804
query32	212	62	64	62
query33	894	373	332	332
query34	1659	862	527	527
query35	820	816	727	727
query36	942	988	872	872
query37	108	107	75	75
query38	4249	4308	4257	4257
query39	1518	1466	1455	1455
query40	217	116	105	105
query41	54	53	53	53
query42	133	119	112	112
query43	506	520	486	486
query44	1340	846	848	846
query45	189	170	169	169
query46	871	1055	675	675
query47	1851	1892	1802	1802
query48	399	452	329	329
query49	920	546	444	444
query50	663	706	408	408
query51	4249	4322	4195	4195
query52	104	111	101	101
query53	221	259	198	198
query54	610	587	535	535
query55	85	87	86	86
query56	320	294	288	288
query57	1152	1212	1126	1126
query58	282	280	269	269
query59	2649	2722	2682	2682
query60	348	330	323	323
query61	157	119	123	119
query62	738	757	707	707
query63	234	193	189	189
query64	2382	1079	703	703
query65	4462	4362	4210	4210
query66	807	394	307	307
query67	15885	15637	15308	15308
query68	8611	897	520	520
query69	541	306	264	264
query70	1227	1147	1067	1067
query71	500	323	292	292
query72	5355	4867	4866	4866
query73	1303	715	353	353
query74	8864	9057	8802	8802
query75	3774	3210	2695	2695
query76	4302	1195	748	748
query77	641	359	321	321
query78	10206	10283	9380	9380
query79	2163	787	569	569
query80	607	519	447	447
query81	503	248	218	218
query82	448	125	92	92
query83	250	253	234	234
query84	298	101	88	88
query85	774	359	313	313
query86	384	308	306	306
query87	4405	4382	4328	4328
query88	3672	2336	2317	2317
query89	410	323	285	285
query90	1818	210	215	210
query91	146	146	113	113
query92	73	61	55	55
query93	1699	945	585	585
query94	672	412	305	305
query95	380	291	282	282
query96	507	576	282	282
query97	2708	2774	2643	2643
query98	237	208	202	202
query99	1713	1417	1287	1287
Total cold run time: 305903 ms
Total hot run time: 194006 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.01 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1808eb8a59d488ac31776634ecb5959708985fb9, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.11	0.12
query3	0.25	0.20	0.20
query4	1.59	0.20	0.19
query5	0.46	0.45	0.45
query6	1.18	0.68	0.66
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.59	0.52	0.52
query10	0.57	0.57	0.57
query11	0.16	0.10	0.11
query12	0.16	0.12	0.11
query13	0.62	0.60	0.60
query14	0.79	0.81	0.80
query15	0.89	0.85	0.87
query16	0.39	0.38	0.38
query17	1.02	1.00	1.02
query18	0.22	0.21	0.21
query19	1.89	1.79	1.76
query20	0.02	0.01	0.01
query21	15.40	0.87	0.53
query22	0.77	1.18	0.64
query23	14.98	1.41	0.65
query24	6.84	1.99	0.76
query25	0.49	0.17	0.20
query26	0.61	0.16	0.14
query27	0.05	0.05	0.04
query28	9.60	0.90	0.44
query29	12.55	3.90	3.23
query30	0.25	0.09	0.07
query31	2.80	0.60	0.40
query32	3.23	0.55	0.47
query33	3.02	3.08	3.06
query34	15.78	5.07	4.56
query35	4.50	4.53	4.47
query36	0.66	0.50	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.14
query41	0.08	0.02	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 103.04 s
Total hot run time: 29.01 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.94% (14923/26676)
Line Coverage 44.76% (132403/295784)
Region Coverage 43.86% (66611/151883)
Branch Coverage 38.45% (34132/88774)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.40% (20850/26261)
Line Coverage 72.61% (214769/295782)
Region Coverage 70.77% (126290/178439)
Branch Coverage 64.56% (65470/101410)

@zhannngchen
Copy link
Contributor Author

run buildall

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 21, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 100.00% (7/7) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.25% (1113/1337)
Line Coverage 66.13% (18665/28223)
Region Coverage 65.81% (9264/14077)
Branch Coverage 55.57% (4981/8964)

@doris-robot
Copy link

TPC-H: Total hot run time: 33994 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85a9f6592fca6d680d1d3aa366bdfcd679001530, data reload: false

------ Round 1 ----------------------------------
q1	26861	5011	4984	4984
q2	2079	288	196	196
q3	10593	1250	711	711
q4	10231	995	522	522
q5	7806	2373	2369	2369
q6	182	168	133	133
q7	930	747	612	612
q8	9321	1285	1148	1148
q9	6747	5115	5108	5108
q10	6877	2339	1912	1912
q11	498	294	275	275
q12	359	349	220	220
q13	17777	3727	3063	3063
q14	236	224	211	211
q15	546	487	481	481
q16	432	436	384	384
q17	627	932	378	378
q18	7546	7124	7149	7124
q19	1912	986	567	567
q20	338	343	223	223
q21	3943	3185	2427	2427
q22	1064	1014	946	946
Total cold run time: 116905 ms
Total hot run time: 33994 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5188	5065	5061	5061
q2	241	331	233	233
q3	2214	2625	2321	2321
q4	1364	1829	1488	1488
q5	4470	4413	4441	4413
q6	217	171	133	133
q7	2065	1938	1747	1747
q8	2623	2580	2615	2580
q9	7331	7179	7079	7079
q10	3029	3181	2751	2751
q11	587	499	497	497
q12	698	810	601	601
q13	3521	3949	3377	3377
q14	271	293	263	263
q15	515	491	466	466
q16	453	518	449	449
q17	1207	1640	1336	1336
q18	7706	7532	7341	7341
q19	820	789	881	789
q20	1964	1967	1836	1836
q21	4944	4528	4475	4475
q22	1057	1070	989	989
Total cold run time: 52485 ms
Total hot run time: 50225 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193281 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85a9f6592fca6d680d1d3aa366bdfcd679001530, data reload: false

query1	1433	1103	1057	1057
query2	6208	1948	1857	1857
query3	11052	4544	4560	4544
query4	53126	25776	23371	23371
query5	5170	508	491	491
query6	377	222	209	209
query7	5005	504	303	303
query8	328	258	241	241
query9	6081	2658	2668	2658
query10	445	334	271	271
query11	14992	15159	14810	14810
query12	163	112	111	111
query13	1111	545	412	412
query14	10124	6239	6397	6239
query15	206	194	180	180
query16	7101	704	465	465
query17	1089	741	595	595
query18	1586	423	332	332
query19	207	206	189	189
query20	134	130	134	130
query21	259	128	111	111
query22	4337	4562	4242	4242
query23	34364	33619	33535	33535
query24	6564	2469	2432	2432
query25	475	489	392	392
query26	704	272	177	177
query27	2305	524	362	362
query28	2925	2167	2174	2167
query29	588	587	441	441
query30	275	225	197	197
query31	887	864	795	795
query32	75	76	64	64
query33	463	365	319	319
query34	791	880	533	533
query35	821	847	758	758
query36	954	1006	913	913
query37	115	108	81	81
query38	4307	4378	4320	4320
query39	1520	1657	1464	1464
query40	217	122	107	107
query41	61	86	56	56
query42	131	120	118	118
query43	527	527	513	513
query44	1359	839	866	839
query45	186	176	172	172
query46	870	1049	661	661
query47	1855	1898	1775	1775
query48	393	429	363	363
query49	697	515	465	465
query50	668	700	411	411
query51	4231	4275	4263	4263
query52	118	124	113	113
query53	261	267	194	194
query54	612	595	517	517
query55	90	88	88	88
query56	314	334	320	320
query57	1166	1244	1150	1150
query58	286	280	267	267
query59	2859	2879	2793	2793
query60	348	342	315	315
query61	122	122	124	122
query62	746	751	669	669
query63	227	187	188	187
query64	1734	1055	666	666
query65	4357	4255	4182	4182
query66	714	414	303	303
query67	15953	15688	15427	15427
query68	7277	903	514	514
query69	548	318	280	280
query70	1214	1099	1098	1098
query71	507	335	314	314
query72	6121	4858	4952	4858
query73	1381	720	358	358
query74	8983	8845	8834	8834
query75	3777	3189	2654	2654
query76	4283	1203	770	770
query77	622	376	280	280
query78	10072	10217	9413	9413
query79	1973	792	576	576
query80	588	520	447	447
query81	472	263	217	217
query82	454	131	99	99
query83	259	242	241	241
query84	286	109	88	88
query85	812	363	310	310
query86	413	313	316	313
query87	4364	4433	4311	4311
query88	3460	2319	2308	2308
query89	413	310	275	275
query90	1924	207	204	204
query91	146	146	114	114
query92	77	58	54	54
query93	1121	932	575	575
query94	692	400	287	287
query95	370	298	289	289
query96	510	572	282	282
query97	2785	2735	2595	2595
query98	234	208	205	205
query99	1404	1396	1294	1294
Total cold run time: 297311 ms
Total hot run time: 193281 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85a9f6592fca6d680d1d3aa366bdfcd679001530, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.11	0.11
query3	0.26	0.20	0.20
query4	1.59	0.21	0.19
query5	0.48	0.45	0.46
query6	1.14	0.68	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.61	0.51	0.53
query10	0.57	0.58	0.57
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.59	0.59
query14	0.80	0.82	0.80
query15	0.88	0.89	0.86
query16	0.37	0.39	0.39
query17	1.04	1.02	1.05
query18	0.23	0.22	0.22
query19	1.94	1.79	1.79
query20	0.01	0.01	0.01
query21	15.41	0.91	0.56
query22	0.76	1.13	0.70
query23	14.96	1.38	0.63
query24	7.01	1.42	0.48
query25	0.51	0.24	0.10
query26	0.65	0.17	0.14
query27	0.05	0.06	0.04
query28	9.57	0.92	0.45
query29	12.77	4.02	3.25
query30	0.25	0.09	0.06
query31	2.82	0.63	0.39
query32	3.23	0.55	0.47
query33	3.00	3.01	3.08
query34	15.78	5.11	4.48
query35	4.50	4.54	4.52
query36	0.67	0.52	0.48
query37	0.09	0.06	0.07
query38	0.06	0.04	0.03
query39	0.03	0.03	0.03
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 103.53 s
Total hot run time: 28.73 s

@zhannngchen zhannngchen changed the title (log)(mow) optimize some log for mow table [log](mow) optimize some log for mow table May 21, 2025
@zhannngchen zhannngchen merged commit e9adc01 into apache:master May 21, 2025
26 of 32 checks passed
zhannngchen added a commit to zhannngchen/incubator-doris that referenced this pull request May 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.6-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants