Skip to content

[enhancement](statistics) Add segment footer open stats to profile #50326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Apr 25, 2025
Merged

[enhancement](statistics) Add segment footer open stats to profile #50326

merged 1 commit into from
Apr 25, 2025

Conversation

TangSiyang2001
Copy link
Collaborator

@TangSiyang2001 TangSiyang2001 commented Apr 23, 2025 •
edited
Loading

What problem does this PR solve?

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
      Before: 
       Scanner:
        - ReadColumns: [id]
        - PerScannerRunningTime: [3s903ms, ]
        - PerScannerRowsRead: [90, ]
        - PerScannerWaitTime: [96.832us, ]
        - PerScannerProjectionTime: [1.616ms, ]
        - EOS: True
       SegmentIterator:
          - FileCache: 0ns
            - BytesScannedFromCache: 0.00 
            - BytesScannedFromRemote: 2.90 KB
            - BytesWriteIntoCache: 0.00 
            - CacheGetOrSetTimer: 715.475us
            - GetTimer: 54.503us
            - LocalIOUseTimer: 0ns
            - LockWaitTimer: 31.919us
            - NumInvertedIndexRemoteIOTotal: 0
            - NumLocalIOTotal: 0
            - NumRemoteIOTotal: 90
            - NumSkipCacheIOTotal: 90
            - ReadCacheFileDirectlyTimer: 0ns
            - RemoteIOUseTimer: 1sec235ms
            - SetTimer: 453.935us
            - WriteCacheIOUseTimer: 0n
      After: 
      Scanner:
               - ReadColumns: [id]
               - PerScannerRunningTime: [3s828ms, ]
               - PerScannerRowsRead: [90, ]
               - PerScannerWaitTime: [70.718us, ]
               - PerScannerProjectionTime: [1.073ms, ]
               - EOS: True
              SegmentIterator:
                 - FileCache: 0ns
                   - BytesScannedFromCache: 0.00 
                   - BytesScannedFromRemote: 20.68 KB
                   - BytesWriteIntoCache: 0.00 
                   - CacheGetOrSetTimer: 1.951ms
                   - GetTimer: 115.892us
                   - LocalIOUseTimer: 0ns
                   - LockWaitTimer: 93.375us
                   - NumInvertedIndexRemoteIOTotal: 0
                   - NumLocalIOTotal: 0
                   - NumRemoteIOTotal: 270
                   - NumSkipCacheIOTotal: 270
                   - ReadCacheFileDirectlyTimer: 0ns
                   - RemoteIOUseTimer: 3sec869ms
                   - SetTimer: 1.53ms
                   - WriteCacheIOUseTimer: 0ns
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@dataroaring dataroaring added dev/3.0.x usercase Important user case type label labels Apr 23, 2025
@TangSiyang2001
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34106 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6fced06e632f060b85dbb6ab78dea67011efbd4b, data reload: false

------ Round 1 ----------------------------------
q1	25923	5083	5088	5083
q2	2064	304	196	196
q3	10345	1279	699	699
q4	10219	1012	545	545
q5	7552	2411	2345	2345
q6	188	166	130	130
q7	896	750	628	628
q8	9306	1286	1063	1063
q9	6758	5131	5115	5115
q10	6806	2309	1911	1911
q11	501	288	282	282
q12	360	381	233	233
q13	17768	3648	3057	3057
q14	240	238	216	216
q15	528	471	500	471
q16	452	448	422	422
q17	595	863	355	355
q18	7575	7205	7138	7138
q19	1677	949	557	557
q20	333	334	212	212
q21	4357	2670	2473	2473
q22	1068	1048	975	975
Total cold run time: 115511 ms
Total hot run time: 34106 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5156	5100	5078	5078
q2	240	331	227	227
q3	2158	2624	2247	2247
q4	1449	1839	1371	1371
q5	4398	4384	4428	4384
q6	226	170	126	126
q7	2013	1940	1766	1766
q8	2622	2616	2492	2492
q9	7286	7292	7069	7069
q10	3022	3187	2765	2765
q11	583	498	512	498
q12	691	780	590	590
q13	3541	3921	3259	3259
q14	272	298	296	296
q15	535	501	478	478
q16	478	506	459	459
q17	1148	1497	1426	1426
q18	7779	7677	7473	7473
q19	867	823	855	823
q20	1986	2010	1877	1877
q21	5182	4953	4878	4878
q22	1114	1053	1032	1032
Total cold run time: 52746 ms
Total hot run time: 50614 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192625 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6fced06e632f060b85dbb6ab78dea67011efbd4b, data reload: false

query1	1387	1098	1077	1077
query2	6047	1852	1826	1826
query3	11154	4532	4742	4532
query4	53451	24991	23362	23362
query5	4947	560	450	450
query6	339	195	191	191
query7	4872	490	288	288
query8	321	257	239	239
query9	5767	2586	2603	2586
query10	441	313	272	272
query11	15033	14970	15283	14970
query12	157	108	108	108
query13	1051	520	393	393
query14	10254	6314	6354	6314
query15	207	204	170	170
query16	7084	694	532	532
query17	1130	748	615	615
query18	1572	414	321	321
query19	213	196	175	175
query20	128	129	126	126
query21	209	137	107	107
query22	4442	4558	4310	4310
query23	34048	33548	33648	33548
query24	6609	2394	2427	2394
query25	462	469	408	408
query26	697	282	163	163
query27	2487	500	336	336
query28	3172	2185	2155	2155
query29	583	565	444	444
query30	274	229	198	198
query31	870	887	784	784
query32	75	65	65	65
query33	438	360	328	328
query34	764	881	556	556
query35	782	838	736	736
query36	958	972	912	912
query37	136	103	75	75
query38	4296	4219	4161	4161
query39	1479	1415	1440	1415
query40	208	131	112	112
query41	54	55	55	55
query42	125	111	110	110
query43	506	516	491	491
query44	1353	813	819	813
query45	176	174	172	172
query46	841	1017	639	639
query47	1862	1918	1853	1853
query48	392	409	320	320
query49	703	506	417	417
query50	641	685	404	404
query51	4184	4193	4176	4176
query52	110	117	102	102
query53	226	262	192	192
query54	591	587	511	511
query55	86	77	85	77
query56	313	302	288	288
query57	1149	1199	1129	1129
query58	255	257	264	257
query59	2747	2831	2657	2657
query60	328	314	315	314
query61	135	130	129	129
query62	731	734	677	677
query63	220	197	187	187
query64	1746	1043	726	726
query65	4352	4243	4242	4242
query66	760	388	308	308
query67	15846	15473	15255	15255
query68	6574	862	517	517
query69	535	292	262	262
query70	1179	1111	1069	1069
query71	504	315	281	281
query72	6006	4631	4604	4604
query73	1235	570	344	344
query74	9162	9069	8612	8612
query75	4054	3208	2726	2726
query76	4130	1174	763	763
query77	718	374	275	275
query78	10177	10050	9381	9381
query79	2798	800	559	559
query80	761	508	450	450
query81	475	256	225	225
query82	442	119	101	101
query83	299	260	240	240
query84	289	110	89	89
query85	781	350	310	310
query86	364	295	310	295
query87	4426	4399	4295	4295
query88	3422	2182	2170	2170
query89	409	307	341	307
query90	1821	207	208	207
query91	144	140	109	109
query92	78	56	60	56
query93	2235	930	571	571
query94	663	415	318	318
query95	372	288	340	288
query96	482	563	272	272
query97	3206	3331	3161	3161
query98	231	212	205	205
query99	1435	1413	1308	1308
Total cold run time: 298383 ms
Total hot run time: 192625 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6fced06e632f060b85dbb6ab78dea67011efbd4b, data reload: false

query1	0.03	0.05	0.03
query2	0.13	0.10	0.12
query3	0.25	0.20	0.20
query4	1.61	0.20	0.19
query5	0.59	0.59	0.59
query6	1.18	0.71	0.72
query7	0.02	0.01	0.02
query8	0.03	0.04	0.04
query9	0.57	0.53	0.53
query10	0.57	0.56	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	1.16	1.16	1.16
query15	0.87	0.84	0.85
query16	0.38	0.37	0.38
query17	1.06	1.01	1.05
query18	0.22	0.19	0.19
query19	1.87	1.76	1.81
query20	0.02	0.01	0.02
query21	15.40	0.90	0.55
query22	0.76	1.25	0.64
query23	14.87	1.40	0.59
query24	6.50	2.27	1.26
query25	0.55	0.21	0.05
query26	0.61	0.17	0.13
query27	0.05	0.05	0.06
query28	9.89	0.82	0.45
query29	12.54	3.99	3.26
query30	0.25	0.09	0.06
query31	2.83	0.59	0.37
query32	3.23	0.54	0.46
query33	3.05	3.08	3.11
query34	15.73	5.10	4.50
query35	4.56	4.54	4.54
query36	0.66	0.50	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.41 s
Total hot run time: 29.89 s

@TangSiyang2001 TangSiyang2001 force-pushed the open-seg-footer-stats branch from 6fced06 to 4c11322 Compare April 23, 2025 16:09
@TangSiyang2001
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34606 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4c113221a7e8fd3dc9c67a276c7c5dca3147b1b1, data reload: false

------ Round 1 ----------------------------------
q1	25914	5051	5138	5051
q2	2066	275	186	186
q3	10407	1231	694	694
q4	10225	1018	533	533
q5	7659	2390	2379	2379
q6	179	166	130	130
q7	913	740	626	626
q8	9341	1304	1096	1096
q9	6812	5101	5073	5073
q10	6823	2280	1888	1888
q11	481	285	256	256
q12	332	357	220	220
q13	17765	3645	3090	3090
q14	239	222	210	210
q15	536	504	495	495
q16	443	442	392	392
q17	583	854	360	360
q18	7555	7049	7142	7049
q19	1210	965	537	537
q20	336	345	229	229
q21	3972	3372	3151	3151
q22	1009	1010	961	961
Total cold run time: 114800 ms
Total hot run time: 34606 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5054	5092	5102	5092
q2	243	330	230	230
q3	2121	2606	2290	2290
q4	1454	1822	1397	1397
q5	4573	4414	4401	4401
q6	222	164	125	125
q7	1975	1908	1720	1720
q8	2554	2592	2535	2535
q9	7245	7114	7156	7114
q10	2922	3032	2628	2628
q11	560	497	511	497
q12	626	732	616	616
q13	3344	3716	3107	3107
q14	271	276	276	276
q15	536	481	481	481
q16	434	482	444	444
q17	1106	1561	1288	1288
q18	7352	7199	7016	7016
q19	806	799	834	799
q20	1938	1942	1826	1826
q21	5092	4674	4629	4629
q22	1042	1017	1009	1009
Total cold run time: 51470 ms
Total hot run time: 49520 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186181 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4c113221a7e8fd3dc9c67a276c7c5dca3147b1b1, data reload: false

query1	1002	473	492	473
query2	6555	1916	1848	1848
query3	6745	221	215	215
query4	27059	23407	23370	23370
query5	4790	624	457	457
query6	300	218	203	203
query7	4630	484	272	272
query8	286	245	243	243
query9	8625	2587	2600	2587
query10	512	336	276	276
query11	15384	15210	14853	14853
query12	169	110	106	106
query13	1643	500	372	372
query14	9534	6032	6160	6032
query15	228	195	167	167
query16	7263	632	495	495
query17	1156	744	580	580
query18	1966	431	306	306
query19	199	185	162	162
query20	121	119	116	116
query21	211	122	109	109
query22	4132	4304	4021	4021
query23	33838	32855	33152	32855
query24	8446	2399	2394	2394
query25	546	446	387	387
query26	1234	264	146	146
query27	2758	498	340	340
query28	4322	2092	2091	2091
query29	746	548	418	418
query30	280	220	186	186
query31	941	821	758	758
query32	73	69	66	66
query33	587	391	312	312
query34	788	836	532	532
query35	779	815	737	737
query36	953	973	882	882
query37	124	103	83	83
query38	4194	4176	4165	4165
query39	1456	1405	1402	1402
query40	210	119	117	117
query41	58	53	56	53
query42	115	105	105	105
query43	507	505	459	459
query44	1300	800	793	793
query45	177	184	161	161
query46	821	1013	614	614
query47	1740	1782	1723	1723
query48	399	408	305	305
query49	796	525	415	415
query50	625	667	391	391
query51	4100	4111	4069	4069
query52	113	109	98	98
query53	227	250	181	181
query54	585	584	493	493
query55	84	82	77	77
query56	313	319	284	284
query57	1130	1137	1078	1078
query58	257	254	249	249
query59	2605	2724	2574	2574
query60	332	331	304	304
query61	130	132	127	127
query62	791	724	657	657
query63	217	188	198	188
query64	4343	1116	789	789
query65	4314	4228	4264	4228
query66	1157	437	346	346
query67	15598	15480	15452	15452
query68	7738	885	520	520
query69	474	324	277	277
query70	1178	1130	1117	1117
query71	513	326	306	306
query72	5538	4771	4870	4771
query73	695	627	343	343
query74	9087	9074	8852	8852
query75	3839	3181	2737	2737
query76	3670	1191	753	753
query77	790	396	294	294
query78	10020	10183	9335	9335
query79	2001	814	562	562
query80	628	539	464	464
query81	456	251	218	218
query82	450	128	95	95
query83	247	250	232	232
query84	246	95	84	84
query85	788	364	315	315
query86	332	313	295	295
query87	4350	4329	4301	4301
query88	3680	2213	2202	2202
query89	388	317	283	283
query90	1942	216	212	212
query91	145	141	112	112
query92	77	62	59	59
query93	1679	936	580	580
query94	668	398	312	312
query95	371	292	285	285
query96	485	562	274	274
query97	3146	3231	3139	3139
query98	232	213	204	204
query99	1471	1393	1301	1301
Total cold run time: 274943 ms
Total hot run time: 186181 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4c113221a7e8fd3dc9c67a276c7c5dca3147b1b1, data reload: false

query1	0.03	0.03	0.03
query2	0.12	0.10	0.11
query3	0.26	0.20	0.19
query4	1.59	0.20	0.19
query5	0.61	0.60	0.59
query6	1.18	0.73	0.71
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.52	0.52
query10	0.58	0.58	0.56
query11	0.16	0.11	0.10
query12	0.15	0.11	0.12
query13	0.61	0.60	0.60
query14	1.17	1.18	1.20
query15	0.87	0.86	0.85
query16	0.38	0.39	0.36
query17	1.05	1.03	1.05
query18	0.21	0.20	0.20
query19	1.86	1.81	1.80
query20	0.02	0.02	0.01
query21	15.42	0.96	0.54
query22	0.74	1.17	0.64
query23	15.00	1.37	0.66
query24	6.81	1.36	1.21
query25	0.50	0.17	0.07
query26	0.64	0.16	0.14
query27	0.05	0.06	0.05
query28	9.95	0.90	0.44
query29	12.59	4.10	3.41
query30	0.24	0.09	0.06
query31	2.83	0.59	0.39
query32	3.23	0.55	0.47
query33	2.98	3.03	3.03
query34	15.62	5.19	4.45
query35	4.53	4.50	4.48
query36	0.66	0.50	0.48
query37	0.09	0.06	0.07
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.76 s
Total hot run time: 30.01 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.53% (14526/27138)
Line Coverage 42.34% (125969/297509)
Region Coverage 41.16% (64389/156421)
Branch Coverage 35.75% (32374/90564)

@hello-stephen
Copy link
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage
Line Coverage
Region Coverage
Branch Coverage

1 similar comment
@hello-stephen
Copy link
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage
Line Coverage
Region Coverage
Branch Coverage

dataroaring
dataroaring reviewed Apr 24, 2025
View reviewed changes
@TangSiyang2001 TangSiyang2001 force-pushed the open-seg-footer-stats branch from 4c11322 to 74696c8 Compare April 24, 2025 07:25
@TangSiyang2001
Copy link
Collaborator Author

run buildall

dataroaring
dataroaring previously approved these changes Apr 24, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

TPC-H: Total hot run time: 33951 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 74696c8b88269e042cd9980d7295b39d777d1ca9, data reload: false

------ Round 1 ----------------------------------
q1	26221	5175	5100	5100
q2	2103	287	201	201
q3	10429	1235	683	683
q4	10239	1015	518	518
q5	7582	2462	2360	2360
q6	194	163	132	132
q7	923	740	613	613
q8	9317	1281	1094	1094
q9	6857	5125	5053	5053
q10	6813	2336	1898	1898
q11	475	293	265	265
q12	344	353	219	219
q13	17758	3675	3073	3073
q14	223	226	203	203
q15	545	494	487	487
q16	449	455	400	400
q17	608	873	354	354
q18	7684	7276	7065	7065
q19	1556	954	551	551
q20	337	325	208	208
q21	4014	3299	2529	2529
q22	1011	993	945	945
Total cold run time: 115682 ms
Total hot run time: 33951 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5161	5103	5118	5103
q2	234	327	229	229
q3	2156	2644	2316	2316
q4	1483	1855	1405	1405
q5	4571	4453	4426	4426
q6	227	165	128	128
q7	1975	1892	1754	1754
q8	2630	2443	2488	2443
q9	7253	7163	7215	7163
q10	2990	3160	2778	2778
q11	582	503	497	497
q12	698	747	624	624
q13	3516	3910	3273	3273
q14	276	310	286	286
q15	515	481	477	477
q16	457	487	470	470
q17	1124	1526	1440	1440
q18	7671	7474	7485	7474
q19	824	819	1010	819
q20	1936	2019	1858	1858
q21	5157	4636	4574	4574
q22	1019	1015	983	983
Total cold run time: 52455 ms
Total hot run time: 50520 ms

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 24, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-DS: Total hot run time: 185179 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 74696c8b88269e042cd9980d7295b39d777d1ca9, data reload: false

query1	1028	475	503	475
query2	6556	1828	1790	1790
query3	6747	217	217	217
query4	26591	23423	23357	23357
query5	4352	615	457	457
query6	296	208	190	190
query7	4644	477	281	281
query8	282	229	230	229
query9	8608	2590	2593	2590
query10	452	341	270	270
query11	15297	15189	14999	14999
query12	158	117	111	111
query13	1656	519	462	462
query14	9152	6064	6025	6025
query15	201	185	169	169
query16	7138	609	498	498
query17	1197	700	562	562
query18	1942	392	286	286
query19	178	182	150	150
query20	116	114	117	114
query21	220	124	110	110
query22	4002	4255	3952	3952
query23	33748	33025	32772	32772
query24	8433	2407	2340	2340
query25	547	482	386	386
query26	1233	268	152	152
query27	2764	490	327	327
query28	4386	2110	2105	2105
query29	769	551	427	427
query30	286	226	186	186
query31	934	828	740	740
query32	72	72	65	65
query33	554	385	305	305
query34	779	838	518	518
query35	766	825	742	742
query36	957	1002	887	887
query37	116	107	76	76
query38	4246	4213	4016	4016
query39	1451	1381	1412	1381
query40	222	124	111	111
query41	63	61	59	59
query42	119	111	105	105
query43	495	490	466	466
query44	1302	813	811	811
query45	180	183	175	175
query46	832	1043	608	608
query47	1755	1791	1734	1734
query48	366	412	293	293
query49	775	493	403	403
query50	649	683	407	407
query51	4109	4110	4000	4000
query52	107	107	100	100
query53	226	248	183	183
query54	582	569	503	503
query55	83	80	80	80
query56	289	294	282	282
query57	1140	1159	1084	1084
query58	257	292	252	252
query59	2581	2722	2602	2602
query60	327	324	298	298
query61	153	127	125	125
query62	812	732	655	655
query63	223	183	186	183
query64	4387	1011	675	675
query65	4344	4238	4226	4226
query66	1165	417	308	308
query67	15805	15413	15050	15050
query68	7577	877	506	506
query69	469	304	272	272
query70	1145	1121	1125	1121
query71	419	323	303	303
query72	5843	4757	4885	4757
query73	691	652	343	343
query74	8862	8978	8763	8763
query75	3142	3173	2690	2690
query76	3145	1188	762	762
query77	483	380	366	366
query78	9873	9941	9242	9242
query79	2205	806	572	572
query80	661	511	428	428
query81	498	271	218	218
query82	186	127	97	97
query83	255	246	226	226
query84	250	107	87	87
query85	755	404	315	315
query86	395	298	293	293
query87	4391	4352	4450	4352
query88	3205	2208	2213	2208
query89	382	300	281	281
query90	1944	211	212	211
query91	147	141	116	116
query92	78	64	57	57
query93	2206	923	581	581
query94	680	408	311	311
query95	379	291	289	289
query96	490	558	275	275
query97	3078	3183	3152	3152
query98	240	209	207	207
query99	1332	1432	1281	1281
Total cold run time: 271608 ms
Total hot run time: 185179 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 74696c8b88269e042cd9980d7295b39d777d1ca9, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.20	0.19
query4	1.62	0.20	0.11
query5	0.57	0.57	0.55
query6	1.22	0.72	0.74
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.54	0.53
query10	0.58	0.58	0.57
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	1.15	1.18	1.16
query15	0.89	0.86	0.85
query16	0.39	0.40	0.38
query17	1.04	1.05	1.08
query18	0.22	0.20	0.19
query19	1.95	1.76	1.80
query20	0.01	0.01	0.01
query21	15.39	0.92	0.56
query22	0.75	1.18	0.71
query23	14.88	1.40	0.65
query24	6.83	0.90	0.90
query25	0.52	0.25	0.07
query26	0.58	0.16	0.14
query27	0.05	0.05	0.05
query28	9.38	0.98	0.43
query29	12.54	4.20	3.47
query30	0.26	0.10	0.06
query31	2.82	0.61	0.38
query32	3.24	0.55	0.48
query33	3.01	3.05	3.06
query34	15.77	5.10	4.52
query35	4.51	4.54	4.48
query36	0.67	0.50	0.49
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.07	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.3 s
Total hot run time: 29.84 s

@TangSiyang2001 TangSiyang2001 force-pushed the open-seg-footer-stats branch from 74696c8 to 7029d5d Compare April 24, 2025 10:16
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 24, 2025
@TangSiyang2001
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34009 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7029d5d85a85e8a1aca580ba2ff41afc9ca689c6, data reload: false

------ Round 1 ----------------------------------
q1	25959	5091	5074	5074
q2	2069	287	184	184
q3	10474	1267	731	731
q4	10282	1038	569	569
q5	8988	2425	2424	2424
q6	189	165	137	137
q7	937	742	612	612
q8	9324	1312	1091	1091
q9	6796	5048	5110	5048
q10	6814	2320	1871	1871
q11	489	277	277	277
q12	353	354	219	219
q13	17753	3753	3098	3098
q14	228	233	208	208
q15	540	475	501	475
q16	451	460	406	406
q17	586	865	357	357
q18	7881	7154	7212	7154
q19	1381	961	561	561
q20	339	329	222	222
q21	4139	3351	2376	2376
q22	1036	1024	915	915
Total cold run time: 117008 ms
Total hot run time: 34009 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5209	5157	5158	5157
q2	237	327	243	243
q3	2161	2682	2277	2277
q4	1441	1888	1505	1505
q5	4609	4442	4405	4405
q6	217	163	129	129
q7	1983	1960	1746	1746
q8	2597	2512	2599	2512
q9	7187	7142	7187	7142
q10	3061	3218	2731	2731
q11	582	514	499	499
q12	671	779	635	635
q13	3477	3910	3279	3279
q14	293	315	259	259
q15	511	484	497	484
q16	481	508	460	460
q17	1151	1562	1373	1373
q18	7647	7564	7334	7334
q19	787	765	813	765
q20	1966	2000	1796	1796
q21	5139	4764	4690	4690
q22	1073	1039	1009	1009
Total cold run time: 52480 ms
Total hot run time: 50430 ms

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@gavinchou gavinchou changed the title [enhancement](statistics) Add segment segment footer open stats to profile [enhancement](statistics) Add segment footer open stats to profile Apr 24, 2025
dataroaring
dataroaring previously approved these changes Apr 24, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TangSiyang2001 TangSiyang2001 dismissed stale reviews from dataroaring and gavinchou via ecb29cb April 24, 2025 12:31
@TangSiyang2001 TangSiyang2001 force-pushed the open-seg-footer-stats branch from 7c877d3 to ecb29cb Compare April 24, 2025 12:31
@TangSiyang2001
Copy link
Collaborator Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 24, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 34056 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ecb29cb2880b1b8e5e276794cdae669043d5e3f6, data reload: false

------ Round 1 ----------------------------------
q1	26104	5135	5096	5096
q2	2069	276	180	180
q3	10408	1228	705	705
q4	10233	1010	539	539
q5	7552	2360	2324	2324
q6	180	170	135	135
q7	914	765	615	615
q8	9325	1295	1101	1101
q9	6770	5067	5010	5010
q10	6882	2303	1913	1913
q11	483	282	289	282
q12	364	350	232	232
q13	17775	3839	3073	3073
q14	233	222	217	217
q15	543	478	504	478
q16	453	450	406	406
q17	616	838	379	379
q18	7893	7168	7119	7119
q19	1322	963	567	567
q20	328	332	218	218
q21	4289	2686	2503	2503
q22	1064	964	966	964
Total cold run time: 115800 ms
Total hot run time: 34056 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5283	5206	5160	5160
q2	241	325	237	237
q3	2145	2811	2256	2256
q4	1395	1824	1366	1366
q5	4573	4428	4398	4398
q6	212	164	125	125
q7	1960	1936	1749	1749
q8	2570	2496	2519	2496
q9	7186	7221	7058	7058
q10	3013	3166	2819	2819
q11	568	512	492	492
q12	682	755	593	593
q13	3517	3938	3257	3257
q14	271	312	268	268
q15	522	480	475	475
q16	453	500	460	460
q17	1202	1544	1373	1373
q18	7812	7463	7422	7422
q19	829	820	861	820
q20	1948	2005	1854	1854
q21	5304	4726	4711	4711
q22	1020	985	983	983
Total cold run time: 52706 ms
Total hot run time: 50372 ms

dataroaring
dataroaring approved these changes Apr 24, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 24, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-DS: Total hot run time: 183389 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ecb29cb2880b1b8e5e276794cdae669043d5e3f6, data reload: false

query1	1026	486	516	486
query2	6566	1864	1799	1799
query3	6736	224	211	211
query4	25925	23410	23045	23045
query5	4352	641	476	476
query6	320	214	204	204
query7	4634	494	293	293
query8	300	248	236	236
query9	8601	2565	2600	2565
query10	480	304	275	275
query11	15377	15011	14802	14802
query12	164	115	109	109
query13	1672	526	407	407
query14	8924	6612	6584	6584
query15	207	185	166	166
query16	7329	658	506	506
query17	1210	737	579	579
query18	1985	417	314	314
query19	194	188	156	156
query20	123	120	129	120
query21	220	124	107	107
query22	4076	4263	3964	3964
query23	33848	32989	32856	32856
query24	8448	2377	2429	2377
query25	538	449	399	399
query26	1231	267	165	165
query27	2742	496	333	333
query28	4299	2086	2074	2074
query29	785	552	431	431
query30	279	216	185	185
query31	906	862	761	761
query32	81	67	67	67
query33	558	385	315	315
query34	808	847	520	520
query35	780	810	729	729
query36	940	1025	905	905
query37	113	99	75	75
query38	4194	4257	4094	4094
query39	1489	1411	1396	1396
query40	217	124	108	108
query41	58	54	56	54
query42	119	111	106	106
query43	501	507	469	469
query44	1329	810	801	801
query45	184	178	170	170
query46	826	1026	623	623
query47	1760	1772	1706	1706
query48	371	414	294	294
query49	765	518	400	400
query50	647	712	398	398
query51	4159	4130	4011	4011
query52	117	108	99	99
query53	231	257	187	187
query54	593	581	497	497
query55	83	79	79	79
query56	317	303	299	299
query57	1126	1131	1091	1091
query58	271	257	255	255
query59	2625	2713	2589	2589
query60	331	337	313	313
query61	130	133	129	129
query62	834	740	683	683
query63	229	201	196	196
query64	4412	1006	660	660
query65	4329	4261	4257	4257
query66	1142	406	310	310
query67	15647	15566	15282	15282
query68	8165	890	513	513
query69	471	309	265	265
query70	1169	1107	1080	1080
query71	500	322	307	307
query72	5703	2372	5309	2372
query73	753	754	367	367
query74	8915	9007	8654	8654
query75	3998	3249	2722	2722
query76	3756	1196	772	772
query77	781	396	292	292
query78	10004	10052	9315	9315
query79	3906	807	535	535
query80	632	569	446	446
query81	475	256	218	218
query82	472	128	99	99
query83	287	255	248	248
query84	297	102	92	92
query85	791	354	317	317
query86	344	300	282	282
query87	4411	4391	4349	4349
query88	3247	2241	2239	2239
query89	443	313	280	280
query90	1955	221	217	217
query91	141	143	114	114
query92	75	66	59	59
query93	2484	928	567	567
query94	678	409	289	289
query95	376	299	292	292
query96	487	575	279	279
query97	3152	3200	3143	3143
query98	242	212	209	209
query99	1426	1394	1304	1304
Total cold run time: 276349 ms
Total hot run time: 183389 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ecb29cb2880b1b8e5e276794cdae669043d5e3f6, data reload: false

query1	0.04	0.03	0.03
query2	0.13	0.10	0.11
query3	0.25	0.18	0.19
query4	1.60	0.19	0.19
query5	0.60	0.59	0.59
query6	1.22	0.72	0.72
query7	0.03	0.01	0.02
query8	0.04	0.04	0.03
query9	0.58	0.52	0.51
query10	0.56	0.58	0.56
query11	0.16	0.10	0.11
query12	0.14	0.12	0.11
query13	0.61	0.59	0.60
query14	1.17	1.22	1.20
query15	0.88	0.85	0.86
query16	0.38	0.39	0.39
query17	1.02	1.03	1.07
query18	0.21	0.19	0.20
query19	1.87	1.73	1.81
query20	0.02	0.01	0.01
query21	15.42	0.91	0.53
query22	0.75	1.16	0.99
query23	14.64	1.37	0.65
query24	7.00	1.04	0.74
query25	0.50	0.19	0.16
query26	0.61	0.16	0.14
query27	0.06	0.05	0.04
query28	9.28	0.88	0.44
query29	12.56	4.02	3.29
query30	0.25	0.10	0.07
query31	2.81	0.61	0.40
query32	3.23	0.55	0.48
query33	2.99	3.08	3.06
query34	15.96	5.10	4.58
query35	4.54	4.55	4.56
query36	0.68	0.48	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.27 s
Total hot run time: 30.04 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 90.00% (18/20) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.75% (14595/27151)
Line Coverage 42.57% (126659/297523)
Region Coverage 41.38% (64759/156503)
Branch Coverage 35.93% (32558/90608)

@hello-stephen
Copy link
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 90.00% (18/20) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage
Line Coverage
Region Coverage
Branch Coverage

1 similar comment
@hello-stephen
Copy link
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 90.00% (18/20) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage
Line Coverage
Region Coverage
Branch Coverage

@dataroaring dataroaring merged commit 56b3ffe into apache:master Apr 25, 2025
27 of 29 checks passed
@hello-stephen
Copy link
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage
Line Coverage
Region Coverage
Branch Coverage

@TangSiyang2001 TangSiyang2001 mentioned this pull request Apr 25, 2025
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request Apr 25, 2025
@TangSiyang2001
[enhancement](statistics) Add segment footer open stats to profile (#…
92b9fbe 
…50326) (#50416)

### What problem does this PR solve?

pick: #50326
dataroaring pushed a commit that referenced this pull request Apr 25, 2025
@TangSiyang2001
[enhancement](statistics) Add segment footer open stats to profile (#…
460460a 
…50326) (#50416)

### What problem does this PR solve?

pick: #50326
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@TangSiyang2001 @koarz
[enhancement](statistics) Add segment footer open stats to profile (a…
7c87ce3 
…pache#50326)
@gavinchou gavinchou mentioned this pull request Jun 11, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.6-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@TangSiyang2001 @doris-robot @hello-stephen @gavinchou @dataroaring