Skip to content

[fix](multi-catalog) add max compute custom odps and tunnel url#31925

Merged
xiaokang merged 2 commits intoapache:branch-2.0from
wsjz:branch-2.0
Mar 7, 2024
Merged

[fix](multi-catalog) add max compute custom odps and tunnel url#31925
xiaokang merged 2 commits intoapache:branch-2.0from
wsjz:branch-2.0

Conversation

@wsjz
Copy link
Contributor

@wsjz wsjz commented Mar 7, 2024

Proposed changes

from #31390

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@wsjz wsjz marked this pull request as ready for review March 7, 2024 06:48
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@wsjz
Copy link
Contributor Author

wsjz commented Mar 7, 2024

run buildall

@github-actions github-actions bot added the kind/docs Categorizes issue or PR as related to documentation. label Mar 7, 2024
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

const std::string region() const { return _region; }
const std::string project() const { return _project; }
const std::string table() const { return _table; }
const std::string odps_url() const { return _odps_url; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'odps_url' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
const std::string odps_url() const { return _odps_url; }
[[nodiscard]] const std::string odps_url() const { return _odps_url; }

const std::string project() const { return _project; }
const std::string table() const { return _table; }
const std::string odps_url() const { return _odps_url; }
const std::string tunnel_url() const { return _tunnel_url; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'tunnel_url' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
const std::string tunnel_url() const { return _tunnel_url; }
[[nodiscard]] const std::string tunnel_url() const { return _tunnel_url; }

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.79% (8037/21265)
Line Coverage: 29.48% (65572/222425)
Region Coverage: 28.95% (33745/116571)
Branch Coverage: 24.79% (17317/69846)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7a743e0fff7571ec18a0d98bee3eaf9b0c789738_7a743e0fff7571ec18a0d98bee3eaf9b0c789738/report/index.html

@wsjz
Copy link
Contributor Author

wsjz commented Mar 7, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.79% (8037/21265)
Line Coverage: 29.48% (65576/222425)
Region Coverage: 28.95% (33747/116571)
Branch Coverage: 24.79% (17318/69846)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7a743e0fff7571ec18a0d98bee3eaf9b0c789738_7a743e0fff7571ec18a0d98bee3eaf9b0c789738/report/index.html

@xiaokang
Copy link
Contributor

xiaokang commented Mar 7, 2024

@wsjz There is compile error

@wsjz
Copy link
Contributor Author

wsjz commented Mar 7, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49857 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1525ac803fb0df24cacdf3625017a678273cbb5b, data reload: false

------ Round 1 ----------------------------------
q1	17667	4402	4362	4362
q2	2075	151	139	139
q3	10440	2012	1930	1930
q4	10382	1231	1300	1231
q5	8396	3908	3973	3908
q6	229	120	122	120
q7	2019	1623	1631	1623
q8	9257	2695	2728	2695
q9	10901	10571	10290	10290
q10	8653	3522	3553	3522
q11	418	233	246	233
q12	473	293	300	293
q13	18483	3999	4012	3999
q14	346	324	314	314
q15	511	463	454	454
q16	688	587	599	587
q17	1120	971	958	958
q18	7165	6990	6850	6850
q19	1670	1545	1539	1539
q20	524	303	300	300
q21	4511	4112	4121	4112
q22	513	407	398	398
Total cold run time: 116441 ms
Total hot run time: 49857 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4324	4335	4275	4275
q2	325	222	219	219
q3	4156	4094	4108	4094
q4	2749	2750	2720	2720
q5	7289	7169	7243	7169
q6	231	124	114	114
q7	3196	2831	2889	2831
q8	4423	4435	4507	4435
q9	17154	17043	16853	16853
q10	4232	4245	4269	4245
q11	789	707	683	683
q12	1011	840	880	840
q13	7149	3716	3715	3715
q14	451	418	437	418
q15	493	450	460	450
q16	771	703	700	700
q17	3838	3900	3821	3821
q18	8725	8734	8777	8734
q19	1712	1703	1644	1644
q20	2415	2136	2095	2095
q21	8497	8644	8545	8545
q22	1075	978	943	943
Total cold run time: 85005 ms
Total hot run time: 79543 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.80% (8039/21267)
Line Coverage: 29.48% (65591/222456)
Region Coverage: 28.95% (33752/116578)
Branch Coverage: 24.79% (17317/69852)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1525ac803fb0df24cacdf3625017a678273cbb5b_1525ac803fb0df24cacdf3625017a678273cbb5b/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 201899 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1525ac803fb0df24cacdf3625017a678273cbb5b, data reload: false

query1	904	393	375	375
query2	6544	2112	2065	2065
query3	6923	203	195	195
query4	21260	18082	18094	18082
query5	19713	6461	6572	6461
query6	292	214	223	214
query7	4150	283	295	283
query8	244	245	256	245
query9	3074	2636	2547	2547
query10	414	297	304	297
query11	11664	11145	11049	11049
query12	114	76	74	74
query13	5578	637	627	627
query14	18167	13409	13600	13409
query15	383	229	229	229
query16	6444	262	260	260
query17	1719	1453	855	855
query18	2340	413	399	399
query19	210	148	145	145
query20	78	77	77	77
query21	184	92	96	92
query22	5329	5131	5129	5129
query23	32579	31969	31716	31716
query24	6886	6442	6599	6442
query25	504	404	415	404
query26	519	156	155	155
query27	1905	291	287	287
query28	6132	2211	2182	2182
query29	2850	2673	2628	2628
query30	239	164	161	161
query31	892	719	706	706
query32	66	56	59	56
query33	394	250	258	250
query34	850	471	471	471
query35	1112	910	920	910
query36	1730	1336	1494	1336
query37	84	64	56	56
query38	3073	2979	2939	2939
query39	1363	1296	1314	1296
query40	205	96	91	91
query41	36	31	31	31
query42	82	81	78	78
query43	651	610	617	610
query44	1106	701	724	701
query45	232	228	225	225
query46	1219	992	953	953
query47	2063	1797	1781	1781
query48	968	656	668	656
query49	619	376	352	352
query50	857	599	637	599
query51	4781	4616	4646	4616
query52	79	76	77	76
query53	448	308	312	308
query54	2631	2417	2410	2410
query55	96	76	78	76
query56	215	191	203	191
query57	1159	1128	1046	1046
query58	205	203	203	203
query59	3416	3097	3165	3097
query60	201	185	192	185
query61	82	80	79	79
query62	836	486	517	486
query63	478	332	340	332
query64	2239	1407	1459	1407
query65	3611	3545	3579	3545
query66	795	371	375	371
query67	16713	16056	15060	15060
query68	8988	649	663	649
query69	572	330	356	330
query70	1871	1681	1412	1412
query71	422	307	302	302
query72	6465	3473	3448	3448
query73	722	315	305	305
query74	6409	5917	5890	5890
query75	5368	3806	3741	3741
query76	5468	1099	1175	1099
query77	913	248	244	244
query78	13408	12282	12933	12282
query79	11353	631	624	624
query80	786	380	376	376
query81	484	232	227	227
query82	1127	98	93	93
query83	170	129	129	129
query84	258	68	66	66
query85	683	302	274	274
query86	393	363	363	363
query87	3229	3009	3029	3009
query88	4921	2272	2277	2272
query89	408	283	264	264
query90	1934	189	196	189
query91	144	115	113	113
query92	55	49	49	49
query93	5674	572	575	572
query94	650	196	207	196
query95	1084	1059	1061	1059
query96	635	314	324	314
query97	6406	6332	6470	6332
query98	173	180	170	170
query99	2728	837	886	837
Total cold run time: 316830 ms
Total hot run time: 201899 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1525ac803fb0df24cacdf3625017a678273cbb5b, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.04
query4	1.80	0.07	0.06
query5	0.54	0.51	0.52
query6	1.26	0.63	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.51	0.46	0.48
query10	0.53	0.53	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.61	0.61	0.61
query14	0.79	0.77	0.77
query15	0.78	0.77	0.77
query16	0.36	0.36	0.37
query17	1.02	1.01	1.01
query18	0.22	0.27	0.23
query19	1.93	1.88	1.80
query20	0.01	0.01	0.01
query21	15.44	0.54	0.55
query22	2.27	2.58	1.63
query23	17.27	1.08	0.92
query24	4.16	5.78	1.19
query25	0.41	0.07	0.08
query26	0.95	0.15	0.16
query27	0.04	0.04	0.03
query28	4.08	0.72	0.73
query29	12.63	2.34	2.35
query30	0.58	0.54	0.53
query31	2.81	0.38	0.37
query32	3.40	0.50	0.49
query33	3.08	3.06	3.03
query34	15.29	4.78	4.77
query35	4.85	4.86	4.85
query36	1.09	1.02	1.02
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 99.7 s
Total hot run time: 30.94 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1525ac803fb0df24cacdf3625017a678273cbb5b with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.6 seconds inserted 10000000 Rows, about 485K ops/s

@xiaokang xiaokang merged commit e2e3dfe into apache:branch-2.0 Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/docs Categorizes issue or PR as related to documentation.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants