Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](http action)Add http action to show nested inverted index file #38272

Merged
merged 6 commits into from
Jul 27, 2024
Merged

[feature](http action)Add http action to show nested inverted index file #38272

merged 6 commits into from
Jul 27, 2024

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Jul 23, 2024 •
edited
Loading

Proposed changes

Add show_nested_index_file http action to show contents of nested inverted index files.
Support inverted index storage format v1 and v2.
Add http_client to execute http command when the response body is too large.

Usage:

# format v2
curl  http://10.16.10.6:8147/api/show_nested_index_file?tablet_id=10090
{
    "tablet_id": 11479,
    "rowsets": [
        {
            "rowset_id": "02000000000001993e4f197d4f45cf7c58a5d70f01990d84",
            "index_storage_format": "V2",
            "segments": []
        },
        {
            "rowset_id": "020000000000019b3e4f197d4f45cf7c58a5d70f01990d84",
            "index_storage_format": "V2",
            "segments": [
                {
                    "segment_id": 0,
                    "idx_file_path": "/mnt/disk2/luen/mydoris/be/storage/data/46/11479/777485197/020000000000019b3e4f197d4f45cf7c58a5d70f01990d84_0.idx",
                    "idx_file_size": 585,
                    "indices": [
                        {
                            "index_id": 11474,
                            "index_suffix": "",
                            "files": [
                                {
                                    "name": "_0.fnm",
                                    "size": 5
                                },
                                {
                                    "name": "_0.frq",
                                    "size": 3
                                },
                                {
                                    "name": "_0.tii",
                                    "size": 51
                                },
                                {
                                    "name": "_0.tis",
                                    "size": 42
                                },
                                {
                                    "name": "null_bitmap",
                                    "size": 5
                                },
                                {
                                    "name": "segments.gen",
                                    "size": 20
                                },
                                {
                                    "name": "segments_2",
                                    "size": 47
                                }
                            ]
                        },
                        {
                            "index_id": 11475,
                            "index_suffix": "",
                            "files": [
                                {
                                    "name": "bkd",
                                    "size": 18
                                },
                                {
                                    "name": "bkd_index",
                                    "size": 29
                                },
                                {
                                    "name": "bkd_meta",
                                    "size": 12
                                },
                                {
                                    "name": "null_bitmap",
                                    "size": 5
                                }
                            ]
                        }
                    ]
                }
            ]
        }
    ]
}


# format v1
curl  http://10.16.10.6:8147/api/show_nested_index_file?tablet_id=10373
{
    "tablet_id": 11460,
    "rowsets": [
        {
            "rowset_id": "02000000000001923e4f197d4f45cf7c58a5d70f01990d84",
            "index_storage_format": "V1",
            "segments": []
        },
        {
            "rowset_id": "02000000000001933e4f197d4f45cf7c58a5d70f01990d84",
            "index_storage_format": "V1",
            "segments": [
                {
                    "segment_id": 0,
                    "indices": [
                        {
                            "index_id": 11455,
                            "index_suffix": "",
                            "idx_file_path": "/mnt/disk2/luen/mydoris/be/storage/data/45/11460/1080325693/02000000000001933e4f197d4f45cf7c58a5d70f01990d84_0_11455.idx",
                            "idx_file_size": 353,
                            "files": [
                                {
                                    "name": "_0.fnm",
                                    "size": 8
                                },
                                {
                                    "name": "_0.frq",
                                    "size": 3
                                },
                                {
                                    "name": "_0.tii",
                                    "size": 51
                                },
                                {
                                    "name": "_0.tis",
                                    "size": 42
                                },
                                {
                                    "name": "null_bitmap",
                                    "size": 5
                                },
                                {
                                    "name": "segments.gen",
                                    "size": 20
                                },
                                {
                                    "name": "segments_2",
                                    "size": 47
                                }
                            ]
                        },
                        {
                            "index_id": 11456,
                            "index_suffix": "",
                            "idx_file_path": "/mnt/disk2/luen/mydoris/be/storage/data/45/11460/1080325693/02000000000001933e4f197d4f45cf7c58a5d70f01990d84_0_11456.idx",
                            "idx_file_size": 164,
                            "files": [
                                {
                                    "name": "bkd",
                                    "size": 18
                                },
                                {
                                    "name": "bkd_index",
                                    "size": 29
                                },
                                {
                                    "name": "bkd_meta",
                                    "size": 12
                                },
                                {
                                    "name": "null_bitmap",
                                    "size": 5
                                }
                            ]
                        }
                    ]
                }
            ]
        }
    ]
}

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Jul 23, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Jul 23, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/http/action/show_nested_index_file_action.h Show resolved Hide resolved
be/src/olap/rowset/beta_rowset.cpp Show resolved Hide resolved
@doris-robot
Copy link

TPC-H: Total hot run time: 40196 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5dc7c691c42b5a587a1017b5b6ecf98d648890b7, data reload: false

------ Round 1 ----------------------------------
q1	17628	5091	4336	4336
q2	2023	193	193	193
q3	10456	1242	1064	1064
q4	10192	877	757	757
q5	7585	2706	2718	2706
q6	221	139	143	139
q7	966	604	604	604
q8	9226	2089	2099	2089
q9	8749	6557	6581	6557
q10	8791	3768	3837	3768
q11	462	233	234	233
q12	452	226	236	226
q13	17907	3005	2969	2969
q14	270	240	243	240
q15	518	483	503	483
q16	511	404	394	394
q17	980	661	705	661
q18	8112	7556	7580	7556
q19	6902	1447	1466	1447
q20	688	329	326	326
q21	4900	3162	3230	3162
q22	350	287	286	286
Total cold run time: 117889 ms
Total hot run time: 40196 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4438	4267	4314	4267
q2	370	271	271	271
q3	3046	2929	2978	2929
q4	2003	1756	1717	1717
q5	5539	5555	5472	5472
q6	227	136	134	134
q7	2268	1872	1837	1837
q8	3252	3465	3396	3396
q9	8775	8869	8819	8819
q10	4107	3735	3847	3735
q11	617	501	510	501
q12	808	645	634	634
q13	17079	3168	3169	3168
q14	329	310	309	309
q15	524	497	495	495
q16	495	451	443	443
q17	1829	1528	1535	1528
q18	8225	7912	7965	7912
q19	1725	1618	1594	1594
q20	2099	1887	1851	1851
q21	5121	4852	4810	4810
q22	572	500	506	500
Total cold run time: 73448 ms
Total hot run time: 56322 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174222 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5dc7c691c42b5a587a1017b5b6ecf98d648890b7, data reload: false

query1	912	373	368	368
query2	6450	1945	1811	1811
query3	6650	206	223	206
query4	25875	17638	17492	17492
query5	3652	480	497	480
query6	246	169	156	156
query7	4588	284	281	281
query8	232	186	187	186
query9	8510	2419	2395	2395
query10	454	290	274	274
query11	11824	10131	10069	10069
query12	119	90	86	86
query13	1632	376	376	376
query14	10251	7773	7725	7725
query15	216	170	165	165
query16	7756	439	428	428
query17	1605	561	546	546
query18	1937	284	298	284
query19	209	150	153	150
query20	95	84	84	84
query21	213	134	139	134
query22	4259	4048	4042	4042
query23	34647	33676	33482	33482
query24	11193	2918	2918	2918
query25	674	394	383	383
query26	711	155	155	155
query27	2179	279	284	279
query28	6123	2098	2097	2097
query29	901	654	637	637
query30	257	160	151	151
query31	987	804	754	754
query32	90	54	56	54
query33	752	329	325	325
query34	943	502	500	500
query35	860	739	728	728
query36	1152	989	980	980
query37	139	87	90	87
query38	2945	2830	2881	2830
query39	905	846	852	846
query40	213	123	124	123
query41	50	46	50	46
query42	115	103	106	103
query43	529	497	463	463
query44	1230	724	736	724
query45	193	164	161	161
query46	1108	747	740	740
query47	1855	1781	1742	1742
query48	367	291	297	291
query49	836	403	427	403
query50	785	397	399	397
query51	6770	6695	6768	6695
query52	109	91	95	91
query53	354	301	298	298
query54	893	448	449	448
query55	78	74	77	74
query56	286	270	271	270
query57	1147	1070	1066	1066
query58	239	250	249	249
query59	2961	2756	2674	2674
query60	300	287	281	281
query61	97	93	92	92
query62	768	645	679	645
query63	320	298	289	289
query64	9140	2196	1663	1663
query65	3271	3120	3090	3090
query66	704	331	336	331
query67	15620	14884	14888	14884
query68	4573	569	543	543
query69	639	431	404	404
query70	1185	1146	1120	1120
query71	439	278	288	278
query72	7709	5716	5266	5266
query73	756	333	324	324
query74	6369	5706	5678	5678
query75	4291	2675	2721	2675
query76	2735	880	948	880
query77	666	312	298	298
query78	9657	10371	9143	9143
query79	4142	532	519	519
query80	1419	472	469	469
query81	593	215	229	215
query82	1044	135	133	133
query83	352	176	175	175
query84	276	91	89	89
query85	1357	364	312	312
query86	495	329	306	306
query87	3304	3143	3129	3129
query88	4820	2373	2385	2373
query89	503	414	388	388
query90	2035	195	192	192
query91	128	100	101	100
query92	64	50	51	50
query93	5181	515	516	515
query94	1423	296	295	295
query95	402	323	320	320
query96	610	277	280	277
query97	3221	3042	3002	3002
query98	221	208	203	203
query99	1520	1274	1280	1274
Total cold run time: 285610 ms
Total hot run time: 174222 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.94 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5dc7c691c42b5a587a1017b5b6ecf98d648890b7, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.09	0.07
query5	0.50	0.47	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.56	0.49	0.48
query10	0.56	0.53	0.53
query11	0.16	0.12	0.12
query12	0.15	0.13	0.13
query13	0.61	0.60	0.59
query14	0.77	0.80	0.84
query15	0.90	0.84	0.84
query16	0.37	0.38	0.38
query17	1.09	1.07	1.10
query18	0.22	0.21	0.22
query19	1.94	1.86	1.76
query20	0.02	0.01	0.01
query21	15.43	0.78	0.66
query22	4.61	6.99	1.97
query23	18.27	1.40	1.28
query24	2.17	0.22	0.22
query25	0.16	0.09	0.09
query26	0.30	0.22	0.21
query27	0.45	0.22	0.23
query28	13.25	1.03	1.01
query29	12.58	3.36	3.37
query30	0.27	0.06	0.05
query31	2.88	0.38	0.38
query32	3.30	0.49	0.47
query33	2.86	2.98	2.86
query34	17.04	4.35	4.40
query35	4.42	4.44	4.39
query36	0.66	0.47	0.46
query37	0.18	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.14	0.13	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.05
Total cold run time: 110.43 s
Total hot run time: 30.94 s

airborne12
airborne12 previously approved these changes Jul 24, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 25, 2024
github-actions[bot]
github-actions bot reviewed Jul 25, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/beta_rowset.cpp Show resolved Hide resolved
@qidaye
Copy link
Contributor Author

qidaye commented Jul 25, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Jul 25, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/beta_rowset.cpp Show resolved Hide resolved
@doris-robot
Copy link

TPC-H: Total hot run time: 39742 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 98fbd111af1301db171dd605ef0b8be9a9f445da, data reload: false

------ Round 1 ----------------------------------
q1	17708	4545	4280	4280
q2	2014	196	191	191
q3	10475	1177	1133	1133
q4	10168	773	788	773
q5	8611	2793	2638	2638
q6	228	137	136	136
q7	995	591	590	590
q8	9218	1899	1923	1899
q9	9263	6937	6953	6937
q10	8804	3828	3803	3803
q11	491	243	245	243
q12	391	220	220	220
q13	17787	2978	2985	2978
q14	281	229	243	229
q15	535	485	503	485
q16	508	378	379	378
q17	974	693	730	693
q18	8175	7533	7417	7417
q19	6902	1015	969	969
q20	680	315	342	315
q21	4890	3154	3312	3154
q22	357	286	281	281
Total cold run time: 119455 ms
Total hot run time: 39742 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4479	4259	4256	4256
q2	365	263	265	263
q3	2994	2822	2729	2729
q4	1900	1626	1651	1626
q5	5286	5336	5302	5302
q6	226	128	131	128
q7	2127	1781	1702	1702
q8	3218	3486	3343	3343
q9	8890	8739	8839	8739
q10	3921	3715	3634	3634
q11	613	491	475	475
q12	755	587	593	587
q13	16499	2985	2996	2985
q14	304	276	270	270
q15	508	486	468	468
q16	481	412	410	410
q17	1767	1487	1489	1487
q18	7790	7499	7432	7432
q19	2389	1551	1430	1430
q20	1996	1782	1783	1782
q21	5040	4745	4635	4635
q22	574	520	504	504
Total cold run time: 72122 ms
Total hot run time: 54187 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172432 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 98fbd111af1301db171dd605ef0b8be9a9f445da, data reload: false

query1	911	367	370	367
query2	6448	1939	1878	1878
query3	6653	205	215	205
query4	28625	17585	17075	17075
query5	4189	507	467	467
query6	269	165	161	161
query7	4584	302	286	286
query8	240	191	198	191
query9	8416	2441	2378	2378
query10	448	288	275	275
query11	11534	10239	10121	10121
query12	136	93	84	84
query13	1628	370	377	370
query14	10329	7637	7669	7637
query15	213	174	164	164
query16	7707	498	468	468
query17	1590	565	522	522
query18	1898	280	272	272
query19	186	139	138	138
query20	95	85	84	84
query21	207	113	96	96
query22	4328	3999	4088	3999
query23	33924	33164	33295	33164
query24	11082	2884	2909	2884
query25	648	371	379	371
query26	1377	146	147	146
query27	2806	270	270	270
query28	7464	2021	2002	2002
query29	983	408	403	403
query30	280	148	149	148
query31	991	711	739	711
query32	96	53	51	51
query33	780	330	364	330
query34	919	461	471	461
query35	840	738	770	738
query36	1086	926	928	926
query37	151	76	77	76
query38	2844	2732	2778	2732
query39	892	811	807	807
query40	263	117	114	114
query41	46	41	49	41
query42	128	97	100	97
query43	501	468	479	468
query44	1201	723	735	723
query45	210	176	175	175
query46	1092	769	708	708
query47	1836	1747	1777	1747
query48	379	286	295	286
query49	1063	411	430	411
query50	807	403	429	403
query51	6789	6631	6583	6583
query52	92	93	95	93
query53	253	185	177	177
query54	853	435	430	430
query55	79	73	77	73
query56	292	276	273	273
query57	1148	1021	1036	1021
query58	285	256	254	254
query59	2851	2682	2636	2636
query60	309	286	281	281
query61	110	95	92	92
query62	836	648	650	648
query63	218	187	178	178
query64	9723	2257	1693	1693
query65	3174	3130	3169	3130
query66	931	339	323	323
query67	15249	14853	14962	14853
query68	4565	543	550	543
query69	440	290	300	290
query70	1127	1138	1084	1084
query71	411	271	262	262
query72	7110	5662	5724	5662
query73	761	341	327	327
query74	6129	5749	5692	5692
query75	3379	2657	2636	2636
query76	2830	911	1001	911
query77	539	310	306	306
query78	9664	9215	10438	9215
query79	2753	515	528	515
query80	1473	504	496	496
query81	575	218	225	218
query82	686	134	132	132
query83	269	177	171	171
query84	261	88	85	85
query85	1998	313	304	304
query86	477	325	309	309
query87	3272	3113	3069	3069
query88	4077	2400	2407	2400
query89	399	281	283	281
query90	1864	194	197	194
query91	129	101	100	100
query92	68	49	52	49
query93	2100	547	538	538
query94	879	360	304	304
query95	351	260	270	260
query96	596	269	266	266
query97	3183	3008	3048	3008
query98	222	200	190	190
query99	1535	1256	1246	1246
Total cold run time: 282637 ms
Total hot run time: 172432 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 98fbd111af1301db171dd605ef0b8be9a9f445da, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.06	0.08
query5	0.51	0.48	0.49
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.55	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.60	0.58	0.59
query14	0.77	0.78	0.77
query15	0.84	0.82	0.81
query16	0.36	0.36	0.38
query17	1.06	1.01	1.05
query18	0.22	0.21	0.21
query19	1.85	1.70	1.83
query20	0.01	0.02	0.01
query21	15.42	0.78	0.66
query22	3.83	8.52	1.34
query23	18.25	1.37	1.22
query24	2.15	0.24	0.21
query25	0.15	0.10	0.09
query26	0.29	0.21	0.22
query27	0.45	0.23	0.23
query28	13.30	1.02	0.99
query29	12.58	3.30	3.30
query30	0.25	0.07	0.05
query31	2.86	0.38	0.37
query32	3.30	0.49	0.48
query33	2.94	2.86	2.93
query34	17.06	4.42	4.40
query35	4.42	4.41	4.44
query36	0.65	0.48	0.47
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.16	0.12	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.49 s
Total hot run time: 30.03 s

@qidaye
Copy link
Contributor Author

qidaye commented Jul 25, 2024

run p0

@qidaye
Copy link
Contributor Author

qidaye commented Jul 25, 2024

run cloud_p0

@qidaye
Copy link
Contributor Author

qidaye commented Jul 25, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39721 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 07c49a1ffff28d9ad82ecde93b4b9c8b7d3bcc6f, data reload: false

------ Round 1 ----------------------------------
q1	17661	4509	4330	4330
q2	2024	201	193	193
q3	10439	1270	1174	1174
q4	10135	764	826	764
q5	7666	2748	2736	2736
q6	223	139	137	137
q7	979	617	605	605
q8	9897	1907	1929	1907
q9	8653	6583	6586	6583
q10	8683	3877	3773	3773
q11	477	243	253	243
q12	393	217	217	217
q13	18711	2963	3008	2963
q14	279	233	251	233
q15	526	477	479	477
q16	511	390	391	390
q17	981	682	708	682
q18	8131	7449	7543	7449
q19	2932	994	1078	994
q20	669	315	329	315
q21	5008	3288	3275	3275
q22	351	281	282	281
Total cold run time: 115329 ms
Total hot run time: 39721 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4413	4293	4293	4293
q2	387	295	259	259
q3	2989	2775	2797	2775
q4	1885	1580	1602	1580
q5	5276	5269	5293	5269
q6	229	131	130	130
q7	2086	1720	1731	1720
q8	3179	3359	3288	3288
q9	8488	8390	8391	8390
q10	3904	3649	3682	3649
q11	592	480	476	476
q12	766	612	640	612
q13	17455	2976	2980	2976
q14	306	276	272	272
q15	516	460	485	460
q16	470	428	415	415
q17	1774	1479	1461	1461
q18	7620	7487	7364	7364
q19	3817	1596	1491	1491
q20	2024	1782	1750	1750
q21	4739	4696	4614	4614
q22	568	495	506	495
Total cold run time: 73483 ms
Total hot run time: 53739 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171622 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 07c49a1ffff28d9ad82ecde93b4b9c8b7d3bcc6f, data reload: false

query1	918	381	371	371
query2	6443	1947	1866	1866
query3	6654	208	216	208
query4	25831	17637	17477	17477
query5	4199	493	518	493
query6	281	194	168	168
query7	4533	281	282	281
query8	240	189	190	189
query9	8266	2417	2394	2394
query10	433	280	260	260
query11	10598	9970	9894	9894
query12	139	85	83	83
query13	1533	377	351	351
query14	10165	7616	7587	7587
query15	222	162	168	162
query16	7730	505	492	492
query17	1590	575	528	528
query18	1970	285	285	285
query19	190	141	137	137
query20	90	86	84	84
query21	218	97	99	97
query22	4188	3948	3973	3948
query23	33820	33011	33076	33011
query24	12197	2911	2853	2853
query25	660	364	375	364
query26	1826	152	145	145
query27	2950	272	270	270
query28	7747	2016	2007	2007
query29	1145	408	428	408
query30	296	150	148	148
query31	993	716	740	716
query32	92	52	51	51
query33	765	327	323	323
query34	894	477	471	471
query35	848	739	732	732
query36	1070	927	901	901
query37	283	81	74	74
query38	2872	2788	2753	2753
query39	856	822	801	801
query40	270	114	111	111
query41	47	46	45	45
query42	111	99	98	98
query43	499	479	480	479
query44	1210	721	727	721
query45	202	174	187	174
query46	1109	701	730	701
query47	1848	1751	1769	1751
query48	363	324	296	296
query49	1189	415	416	415
query50	801	404	404	404
query51	6745	6649	6629	6629
query52	100	96	92	92
query53	249	185	181	181
query54	874	429	444	429
query55	76	72	74	72
query56	293	273	270	270
query57	1175	1019	1045	1019
query58	272	253	261	253
query59	2843	2783	2554	2554
query60	308	289	290	289
query61	125	93	92	92
query62	845	640	659	640
query63	225	184	183	183
query64	10472	2262	1672	1672
query65	3184	3096	3146	3096
query66	1401	361	349	349
query67	15140	14889	14774	14774
query68	4503	558	556	556
query69	449	307	312	307
query70	1182	1073	1115	1073
query71	394	280	271	271
query72	7135	5395	5865	5395
query73	746	325	325	325
query74	6128	5621	5654	5621
query75	3357	2684	2682	2682
query76	2787	905	928	905
query77	478	335	312	312
query78	10282	8910	9512	8910
query79	2678	523	533	523
query80	1846	488	508	488
query81	564	221	220	220
query82	723	136	132	132
query83	277	175	181	175
query84	264	83	80	80
query85	2041	307	305	305
query86	484	302	292	292
query87	3257	3083	3038	3038
query88	4142	2428	2425	2425
query89	390	277	286	277
query90	1844	192	197	192
query91	127	113	174	113
query92	58	50	49	49
query93	2429	528	527	527
query94	931	302	295	295
query95	356	265	263	263
query96	601	274	275	274
query97	3202	3027	3025	3025
query98	235	209	200	200
query99	1620	1200	1264	1200
Total cold run time: 283231 ms
Total hot run time: 171622 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 07c49a1ffff28d9ad82ecde93b4b9c8b7d3bcc6f, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.03
query3	0.22	0.05	0.05
query4	1.67	0.08	0.07
query5	0.50	0.52	0.49
query6	1.13	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.54	0.48	0.49
query10	0.55	0.55	0.53
query11	0.15	0.11	0.11
query12	0.16	0.12	0.12
query13	0.60	0.58	0.59
query14	0.76	0.77	0.78
query15	0.84	0.82	0.82
query16	0.38	0.37	0.34
query17	0.98	0.96	0.94
query18	0.22	0.21	0.22
query19	1.76	1.68	1.64
query20	0.02	0.01	0.01
query21	15.40	0.82	0.65
query22	4.48	7.08	1.93
query23	18.26	1.36	1.27
query24	2.03	0.24	0.22
query25	0.15	0.09	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.23
query28	13.29	1.00	1.01
query29	12.66	3.29	3.27
query30	0.25	0.05	0.06
query31	2.88	0.39	0.39
query32	3.25	0.49	0.47
query33	2.86	2.93	2.88
query34	17.14	4.36	4.34
query35	4.37	4.44	4.45
query36	0.65	0.48	0.48
query37	0.20	0.16	0.16
query38	0.15	0.15	0.16
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.83 s
Total hot run time: 30.48 s

@qidaye
Copy link
Contributor Author

qidaye commented Jul 26, 2024

run cloud_p0

airborne12
airborne12 previously approved these changes Jul 26, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@qidaye
Copy link
Contributor Author

qidaye commented Jul 26, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 26, 2024
@qidaye
Copy link
Contributor Author

qidaye commented Jul 26, 2024

run perfotmance

@qidaye
Copy link
Contributor Author

qidaye commented Jul 26, 2024

run compile

@qidaye
Copy link
Contributor Author

qidaye commented Jul 26, 2024

run p0

@qidaye
Copy link
Contributor Author

qidaye commented Jul 26, 2024

run performance

@doris-robot
Copy link

TPC-H: Total hot run time: 40094 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 98ac2bcf96e303954f929268355f6833a7981e2b, data reload: false

------ Round 1 ----------------------------------
q1	17785	4614	4583	4583
q2	2595	206	204	204
q3	11073	1172	1171	1171
q4	10682	704	734	704
q5	7568	2780	2787	2780
q6	232	139	139	139
q7	992	616	611	611
q8	9299	1991	1952	1952
q9	8967	6583	6623	6583
q10	8811	3837	3835	3835
q11	451	250	253	250
q12	399	220	227	220
q13	17788	2985	2970	2970
q14	276	256	232	232
q15	516	479	484	479
q16	517	393	382	382
q17	986	732	680	680
q18	8083	7514	7565	7514
q19	3978	1094	1011	1011
q20	698	336	349	336
q21	5381	3260	3180	3180
q22	345	278	291	278
Total cold run time: 117422 ms
Total hot run time: 40094 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4421	4295	4265	4265
q2	366	260	264	260
q3	3084	2819	2753	2753
q4	1921	1623	1560	1560
q5	5349	5366	5370	5366
q6	223	129	129	129
q7	2165	1707	1771	1707
q8	3194	3427	3299	3299
q9	8511	8456	8502	8456
q10	3924	3728	3729	3728
q11	598	498	488	488
q12	766	601	623	601
q13	17293	2994	2986	2986
q14	290	282	270	270
q15	511	470	494	470
q16	467	409	410	409
q17	1793	1465	1483	1465
q18	7627	7646	7307	7307
q19	1672	1503	1510	1503
q20	1985	1802	1785	1785
q21	4806	4791	4691	4691
q22	591	484	497	484
Total cold run time: 71557 ms
Total hot run time: 53982 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173557 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 98ac2bcf96e303954f929268355f6833a7981e2b, data reload: false

query1	911	374	372	372
query2	6440	1904	1810	1810
query3	6657	211	214	211
query4	24752	17480	17276	17276
query5	4278	490	473	473
query6	268	181	162	162
query7	4592	293	285	285
query8	242	200	206	200
query9	8325	2507	2494	2494
query10	454	312	286	286
query11	11765	10087	10081	10081
query12	141	94	89	89
query13	1624	389	374	374
query14	10503	7884	7908	7884
query15	230	167	179	167
query16	7786	451	497	451
query17	1626	599	574	574
query18	1829	294	296	294
query19	200	152	157	152
query20	93	85	92	85
query21	203	104	98	98
query22	4180	3923	4044	3923
query23	33841	33178	33388	33178
query24	11907	2901	2890	2890
query25	695	396	404	396
query26	1782	158	157	157
query27	2881	281	293	281
query28	7425	2062	2035	2035
query29	1080	442	432	432
query30	301	151	156	151
query31	947	738	728	728
query32	97	57	59	57
query33	809	355	347	347
query34	899	472	499	472
query35	856	771	766	766
query36	1093	944	952	944
query37	180	83	82	82
query38	2849	2728	2755	2728
query39	854	826	814	814
query40	276	125	120	120
query41	54	48	47	47
query42	122	99	105	99
query43	499	470	455	455
query44	1255	732	741	732
query45	212	180	183	180
query46	1117	820	727	727
query47	1858	1749	1776	1749
query48	370	294	296	294
query49	1189	433	410	410
query50	806	410	423	410
query51	6853	6766	6733	6733
query52	107	89	95	89
query53	257	182	183	182
query54	973	449	452	449
query55	76	73	76	73
query56	295	292	295	292
query57	1152	1030	1055	1030
query58	276	272	271	271
query59	2804	2540	2802	2540
query60	324	297	306	297
query61	100	98	95	95
query62	845	667	644	644
query63	221	186	184	184
query64	10525	2282	1744	1744
query65	3186	3131	3142	3131
query66	1360	332	337	332
query67	15556	14780	14795	14780
query68	8838	562	565	562
query69	753	504	337	337
query70	1128	1128	1124	1124
query71	518	277	288	277
query72	8732	5817	6101	5817
query73	1550	340	331	331
query74	6133	5622	5724	5622
query75	4646	2676	2646	2646
query76	4856	941	906	906
query77	742	325	308	308
query78	10568	8924	8971	8924
query79	8626	535	533	533
query80	1200	543	530	530
query81	577	224	225	224
query82	274	135	138	135
query83	306	175	175	175
query84	297	79	86	79
query85	986	323	301	301
query86	359	326	318	318
query87	3299	3083	3071	3071
query88	4564	2519	2531	2519
query89	445	284	295	284
query90	2090	199	201	199
query91	130	106	103	103
query92	70	52	52	52
query93	5494	566	533	533
query94	1096	309	307	307
query95	364	263	265	263
query96	622	284	275	275
query97	3194	3042	3047	3042
query98	224	198	197	197
query99	1527	1275	1285	1275
Total cold run time: 302471 ms
Total hot run time: 173557 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.7 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 98ac2bcf96e303954f929268355f6833a7981e2b, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.47	0.48
query6	1.13	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.54	0.48	0.50
query10	0.54	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.60	0.58	0.58
query14	0.78	0.78	0.78
query15	0.86	0.83	0.81
query16	0.35	0.36	0.36
query17	0.96	0.96	1.00
query18	0.23	0.22	0.22
query19	1.83	1.71	1.70
query20	0.01	0.01	0.01
query21	15.40	0.80	0.67
query22	4.18	7.47	1.99
query23	18.25	1.34	1.27
query24	2.14	0.23	0.22
query25	0.15	0.08	0.08
query26	0.30	0.20	0.21
query27	0.45	0.24	0.22
query28	13.28	1.03	1.01
query29	12.68	3.33	3.37
query30	0.25	0.07	0.06
query31	2.86	0.40	0.39
query32	3.26	0.49	0.48
query33	2.88	2.87	2.93
query34	17.15	4.38	4.32
query35	4.42	4.44	4.45
query36	0.65	0.46	0.47
query37	0.19	0.16	0.15
query38	0.15	0.16	0.15
query39	0.04	0.04	0.03
query40	0.16	0.12	0.12
query41	0.10	0.04	0.04
query42	0.06	0.05	0.06
query43	0.04	0.04	0.04
Total cold run time: 109.75 s
Total hot run time: 30.7 s

@qidaye
Copy link
Contributor Author

qidaye commented Jul 26, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Jul 26, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/beta_rowset.cpp Show resolved Hide resolved
@doris-robot
Copy link

TPC-H: Total hot run time: 39629 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3e516b921ba0b83b2bd29ec928bd388d731bc3a, data reload: false

------ Round 1 ----------------------------------
q1	18036	4499	4393	4393
q2	2209	201	195	195
q3	10436	1225	1089	1089
q4	10141	763	696	696
q5	7542	2746	2690	2690
q6	222	137	140	137
q7	972	608	609	608
q8	9224	1957	1964	1957
q9	8783	6600	6649	6600
q10	8830	3845	3791	3791
q11	453	254	247	247
q12	400	235	232	232
q13	18330	2986	2961	2961
q14	289	250	236	236
q15	537	497	498	497
q16	497	389	382	382
q17	984	745	698	698
q18	8230	7478	7382	7382
q19	6367	1096	970	970
q20	712	324	332	324
q21	5038	3292	3260	3260
q22	353	284	286	284
Total cold run time: 118585 ms
Total hot run time: 39629 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4449	4266	4290	4266
q2	367	257	268	257
q3	3009	2746	2766	2746
q4	1895	1642	1610	1610
q5	5317	5322	5332	5322
q6	216	131	133	131
q7	2129	1720	1777	1720
q8	3222	3411	3347	3347
q9	8462	8378	8435	8378
q10	3943	3695	3731	3695
q11	595	473	479	473
q12	753	627	576	576
q13	16778	2975	3013	2975
q14	302	278	267	267
q15	525	473	473	473
q16	489	415	433	415
q17	1815	1517	1467	1467
q18	7678	7586	7272	7272
q19	1685	1536	1571	1536
q20	2005	1817	1792	1792
q21	4880	4633	4661	4633
q22	575	509	474	474
Total cold run time: 71089 ms
Total hot run time: 53825 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172881 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e3e516b921ba0b83b2bd29ec928bd388d731bc3a, data reload: false

query1	905	377	373	373
query2	6436	2003	1855	1855
query3	6645	207	212	207
query4	23894	17540	17212	17212
query5	4174	500	493	493
query6	277	182	163	163
query7	4592	295	289	289
query8	247	196	200	196
query9	8714	2472	2451	2451
query10	448	261	257	257
query11	10574	10117	10192	10117
query12	131	90	83	83
query13	1614	358	372	358
query14	10259	7559	7598	7559
query15	220	166	165	165
query16	7750	451	486	451
query17	1365	570	537	537
query18	1962	283	284	283
query19	204	137	140	137
query20	99	86	85	85
query21	205	96	97	96
query22	4293	4030	3973	3973
query23	33875	33167	33090	33090
query24	12295	2863	2841	2841
query25	662	376	387	376
query26	1812	149	148	148
query27	2923	279	278	278
query28	7575	2045	2038	2038
query29	1152	422	404	404
query30	280	146	148	146
query31	953	738	739	738
query32	94	53	57	53
query33	782	348	336	336
query34	912	471	490	471
query35	872	772	735	735
query36	1076	901	928	901
query37	280	83	78	78
query38	2818	2809	2786	2786
query39	855	823	830	823
query40	269	117	111	111
query41	49	43	43	43
query42	120	95	109	95
query43	523	472	451	451
query44	1200	728	725	725
query45	202	177	176	176
query46	1090	740	718	718
query47	1848	1771	1802	1771
query48	357	283	295	283
query49	1213	412	416	412
query50	811	412	438	412
query51	6700	6778	6797	6778
query52	100	97	90	90
query53	251	179	175	175
query54	958	443	448	443
query55	80	75	74	74
query56	288	270	273	270
query57	1156	1050	1054	1050
query58	315	269	271	269
query59	2907	2700	2792	2700
query60	308	284	281	281
query61	97	93	98	93
query62	824	649	636	636
query63	220	181	179	179
query64	10497	2259	1765	1765
query65	3196	3094	3100	3094
query66	1378	338	329	329
query67	15325	14790	14941	14790
query68	4619	557	557	557
query69	442	301	301	301
query70	1160	1146	1136	1136
query71	392	278	274	274
query72	7045	5693	5719	5693
query73	745	342	338	338
query74	6182	5677	5668	5668
query75	3365	2683	2679	2679
query76	2683	896	882	882
query77	471	301	307	301
query78	9481	9010	8973	8973
query79	2407	535	546	535
query80	2260	482	477	477
query81	573	213	222	213
query82	860	141	136	136
query83	300	178	174	174
query84	269	87	80	80
query85	2109	399	305	305
query86	489	319	299	299
query87	3260	3074	3096	3074
query88	4024	2506	2505	2505
query89	395	287	290	287
query90	1861	196	193	193
query91	126	98	101	98
query92	62	49	50	49
query93	2839	540	536	536
query94	945	301	289	289
query95	352	263	268	263
query96	617	284	292	284
query97	3246	3004	3038	3004
query98	207	206	206	206
query99	1679	1277	1301	1277
Total cold run time: 281971 ms
Total hot run time: 172881 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3e516b921ba0b83b2bd29ec928bd388d731bc3a, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.08
query5	0.50	0.48	0.49
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.56	0.53	0.49
query10	0.56	0.54	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.13
query13	0.60	0.59	0.60
query14	0.79	0.76	0.78
query15	0.86	0.82	0.81
query16	0.36	0.37	0.36
query17	0.98	1.00	0.97
query18	0.23	0.22	0.21
query19	1.80	1.74	1.70
query20	0.01	0.01	0.01
query21	15.42	0.74	0.65
query22	4.24	7.10	1.92
query23	18.30	1.40	1.30
query24	2.09	0.22	0.22
query25	0.15	0.09	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.22
query28	13.35	1.02	1.00
query29	12.65	3.31	3.25
query30	0.25	0.07	0.06
query31	2.88	0.41	0.38
query32	3.25	0.48	0.48
query33	2.90	2.87	2.93
query34	17.05	4.37	4.35
query35	4.46	4.46	4.43
query36	0.66	0.46	0.46
query37	0.20	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.04	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.93 s
Total hot run time: 30.56 s

zzzxl1993
zzzxl1993 approved these changes Jul 26, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

csun5285
csun5285 approved these changes Jul 27, 2024
View reviewed changes
Copy link
Contributor

@csun5285 csun5285 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

airborne12
airborne12 approved these changes Jul 27, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 27, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@qidaye qidaye merged commit 390cdcc into apache:master Jul 27, 2024
31 of 34 checks passed
@qidaye qidaye deleted the add_index_http_action branch July 27, 2024 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions left review comments

@airborne12 airborne12 airborne12 approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

@csun5285 csun5285 csun5285 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@qidaye @doris-robot @airborne12 @zzzxl1993 @csun5285