Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pick][improvement](jdbc catalog) Optimize connection pool parameter settings #31177

Merged
merged 1 commit into from
Feb 21, 2024

Conversation

zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Feb 20, 2024

Proposed changes

Issue Number: close #xxx

pick from (#30588)

This PR makes the following changes to the connection pool of JDBC Catalog

  1. Set the maximum connection survival time, the default is 30 minutes
  • Moreover, one-half of the maximum survival time is the recyclable time,
  • One-tenth is the check interval for recycling connections
  1. Keepalive only takes effect on the connection pool on BE, and will be activated based on one-fifth of the maximum survival time.
  2. The maximum number of existing connections is changed from 100 to 10
  3. Add the connection cache recycling thread on BE, and add a parameter to control the recycling time, the default is 28800 (8 hours)
  4. Add CatalogID to the key of the connection pool cache to achieve better isolation, requires refresh catalog to take effect
  5. Upgrade druid connection pool to version 1.2.20
  6. Added JdbcResource's setting of default parameters when upgrading the FE version to avoid errors due to unset parameters.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@github-actions github-actions bot added the area/planner Issues or PRs related to the query planner label Feb 20, 2024
@zy-kkk
Copy link
Member Author

zy-kkk commented Feb 20, 2024

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -307,6 +307,7 @@ class JdbcTableDescriptor : public TableDescriptor {
public:
JdbcTableDescriptor(const TTableDescriptor& tdesc);
std::string debug_string() const override;
int64_t jdbc_catalog_id() const { return _jdbc_catalog_id; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'jdbc_catalog_id' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
int64_t jdbc_catalog_id() const { return _jdbc_catalog_id; }
[[nodiscard]] int64_t jdbc_catalog_id() const { return _jdbc_catalog_id; }

@@ -315,8 +316,14 @@
const std::string& jdbc_table_name() const { return _jdbc_table_name; }
const std::string& jdbc_user() const { return _jdbc_user; }
const std::string& jdbc_passwd() const { return _jdbc_passwd; }
int32_t connection_pool_min_size() const { return _connection_pool_min_size; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'connection_pool_min_size' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
int32_t connection_pool_min_size() const { return _connection_pool_min_size; }
[[nodiscard]] int32_t connection_pool_min_size() const { return _connection_pool_min_size; }

@@ -315,8 +316,14 @@
const std::string& jdbc_table_name() const { return _jdbc_table_name; }
const std::string& jdbc_user() const { return _jdbc_user; }
const std::string& jdbc_passwd() const { return _jdbc_passwd; }
int32_t connection_pool_min_size() const { return _connection_pool_min_size; }
int32_t connection_pool_max_size() const { return _connection_pool_max_size; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'connection_pool_max_size' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
int32_t connection_pool_max_size() const { return _connection_pool_max_size; }
[[nodiscard]] int32_t connection_pool_max_size() const { return _connection_pool_max_size; }

@@ -315,8 +316,14 @@
const std::string& jdbc_table_name() const { return _jdbc_table_name; }
const std::string& jdbc_user() const { return _jdbc_user; }
const std::string& jdbc_passwd() const { return _jdbc_passwd; }
int32_t connection_pool_min_size() const { return _connection_pool_min_size; }
int32_t connection_pool_max_size() const { return _connection_pool_max_size; }
int32_t connection_pool_max_wait_time() const { return _connection_pool_max_wait_time; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'connection_pool_max_wait_time' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
int32_t connection_pool_max_wait_time() const { return _connection_pool_max_wait_time; }
[[nodiscard]] int32_t connection_pool_max_wait_time() const { return _connection_pool_max_wait_time; }

int32_t connection_pool_min_size() const { return _connection_pool_min_size; }
int32_t connection_pool_max_size() const { return _connection_pool_max_size; }
int32_t connection_pool_max_wait_time() const { return _connection_pool_max_wait_time; }
int32_t connection_pool_max_life_time() const { return _connection_pool_max_life_time; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'connection_pool_max_life_time' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
int32_t connection_pool_max_life_time() const { return _connection_pool_max_life_time; }
[[nodiscard]] int32_t connection_pool_max_life_time() const { return _connection_pool_max_life_time; }

int32_t connection_pool_max_size() const { return _connection_pool_max_size; }
int32_t connection_pool_max_wait_time() const { return _connection_pool_max_wait_time; }
int32_t connection_pool_max_life_time() const { return _connection_pool_max_life_time; }
bool connection_pool_keep_alive() const { return _connection_pool_keep_alive; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'connection_pool_keep_alive' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
bool connection_pool_keep_alive() const { return _connection_pool_keep_alive; }
[[nodiscard]] bool connection_pool_keep_alive() const { return _connection_pool_keep_alive; }

@doris-robot
Copy link

TPC-H: Total hot run time: 50161 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 470c8060eba0be619d3597b445083f812657615e, data reload: false

------ Round 1 ----------------------------------
q1	17817	6227	4366	4366
q2	2066	156	144	144
q3	10651	2031	1954	1954
q4	10298	1235	1321	1235
q5	8432	4009	3996	3996
q6	233	120	122	120
q7	2059	1604	1593	1593
q8	9293	2737	2743	2737
q9	10821	10478	10565	10478
q10	8659	3558	3520	3520
q11	428	237	249	237
q12	471	296	295	295
q13	18387	3988	4081	3988
q14	350	337	323	323
q15	507	448	465	448
q16	699	605	601	601
q17	1131	975	948	948
q18	7224	6831	7726	6831
q19	1683	1597	1520	1520
q20	546	300	300	300
q21	4435	4174	4133	4133
q22	503	394	399	394
Total cold run time: 116693 ms
Total hot run time: 50161 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4348	4277	4336	4277
q2	322	221	223	221
q3	4202	4175	4170	4170
q4	2779	2757	2754	2754
q5	7297	7225	7272	7225
q6	235	120	117	117
q7	3181	2859	2840	2840
q8	4397	4526	4527	4526
q9	17210	17064	17155	17064
q10	4262	4270	4251	4251
q11	773	697	683	683
q12	1022	853	848	848
q13	6760	3718	3748	3718
q14	440	448	435	435
q15	496	461	455	455
q16	758	707	700	700
q17	3785	3852	3842	3842
q18	8867	8836	8920	8836
q19	1722	1727	1661	1661
q20	2399	2161	2140	2140
q21	8675	8491	8565	8491
q22	1012	947	1012	947
Total cold run time: 84942 ms
Total hot run time: 80201 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 237938 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 470c8060eba0be619d3597b445083f812657615e, data reload: false

query1	918	404	380	380
query2	6523	2307	2315	2307
query3	6910	197	202	197
query4	21244	17933	18110	17933
query5	19704	6542	6578	6542
query6	275	221	229	221
query7	4144	292	291	291
query8	259	249	220	220
query9	3090	2682	2626	2626
query10	407	287	303	287
query11	11268	10643	10800	10643
query12	127	73	73	73
query13	5587	644	609	609
query14	17367	13269	13227	13227
query15	356	240	240	240
query16	6429	265	263	263
query17	1730	1447	867	867
query18	2329	404	400	400
query19	201	148	142	142
query20	74	73	75	73
query21	190	96	95	95
query22	5255	5129	5102	5102
query23	32614	31997	31885	31885
query24	7068	6548	6467	6467
query25	522	427	439	427
query26	525	166	155	155
query27	1877	286	287	286
query28	6069	2200	2162	2162
query29	3041	2749	2673	2673
query30	239	162	159	159
query31	887	756	736	736
query32	60	58	58	58
query33	384	248	234	234
query34	847	472	483	472
query35	1095	923	886	886
query36	1565	1471	1578	1471
query37	89	63	61	61
query38	3105	2936	2935	2935
query39	1394	1332	1341	1332
query40	199	86	91	86
query41	33	31	31	31
query42	83	86	86	86
query43	616	687	644	644
query44	1099	708	710	708
query45	236	225	223	223
query46	1241	993	962	962
query47	1955	1726	1850	1726
query48	967	680	671	671
query49	605	373	360	360
query50	854	630	604	604
query51	5529	5416	5446	5416
query52	80	92	85	85
query53	439	326	314	314
query54	2629	2478	2473	2473
query55	79	77	71	71
query56	229	218	209	209
query57	1214	1127	1112	1112
query58	213	199	197	197
query59	3516	3358	3196	3196
query60	196	189	192	189
query61	83	87	83	83
query62	786	511	479	479
query63	504	342	337	337
query64	2529	1471	1450	1450
query65	3654	3596	3654	3596
query66	838	358	362	358
query67	17416	16150	17635	16150
query68	6479	648	641	641
query69	552	338	333	333
query70	1725	1748	1999	1748
query71	395	308	311	308
query72	6463	3402	3391	3391
query73	714	319	317	317
query74	6282	5908	5768	5768
query75	4832	3729	3749	3729
query76	4374	1145	1199	1145
query77	542	243	255	243
query78	32820	50168	45820	45820
query79	14798	619	625	619
query80	4476	386	375	375
query81	555	231	225	225
query82	614	90	96	90
query83	333	130	127	127
query84	250	66	68	66
query85	2268	273	274	273
query86	447	380	385	380
query87	3233	2986	2999	2986
query88	6318	2312	2308	2308
query89	395	308	293	293
query90	2510	204	205	204
query91	149	118	128	118
query92	60	48	47	47
query93	3944	572	548	548
query94	1658	208	207	207
query95	1104	1058	1051	1051
query96	642	319	320	319
query97	6473	6287	6343	6287
query98	191	173	176	173
query99	3781	846	905	846
Total cold run time: 342967 ms
Total hot run time: 237938 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.7 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 470c8060eba0be619d3597b445083f812657615e, data reload: false

query1	0.03	0.02	0.02
query2	0.06	0.02	0.03
query3	0.24	0.05	0.04
query4	1.83	0.07	0.07
query5	0.52	0.52	0.52
query6	1.23	0.64	0.62
query7	0.01	0.01	0.01
query8	0.03	0.03	0.03
query9	0.51	0.50	0.50
query10	0.52	0.53	0.53
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.63	0.61	0.62
query14	0.79	0.78	0.79
query15	0.79	0.76	0.77
query16	0.37	0.40	0.37
query17	1.01	1.04	0.98
query18	0.24	0.26	0.24
query19	1.86	1.85	1.87
query20	0.01	0.01	0.01
query21	15.45	0.55	0.57
query22	2.21	2.10	1.50
query23	17.27	0.92	0.89
query24	6.30	1.08	1.13
query25	1.50	0.11	0.11
query26	0.28	0.14	0.14
query27	0.11	0.11	0.12
query28	7.60	0.69	0.77
query29	12.74	2.13	2.28
query30	0.55	0.52	0.52
query31	2.83	0.39	0.40
query32	3.39	0.50	0.50
query33	3.07	3.08	3.07
query34	15.26	4.82	4.79
query35	4.84	4.83	4.86
query36	1.06	1.04	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.15	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.77 s
Total hot run time: 30.7 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 470c8060eba0be619d3597b445083f812657615e with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8035/21253)
Line Coverage: 29.48% (65497/222201)
Region Coverage: 28.95% (33704/116418)
Branch Coverage: 24.81% (17300/69728)
Coverage Report: http://coverage.selectdb-in.cc/coverage/470c8060eba0be619d3597b445083f812657615e_470c8060eba0be619d3597b445083f812657615e/report/index.html

@zy-kkk
Copy link
Member Author

zy-kkk commented Feb 21, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49947 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f09db516001aba162f8ef7dd7dd8986b302fc0f, data reload: false

------ Round 1 ----------------------------------
q1	18059	4445	4291	4291
q2	2037	148	141	141
q3	10530	1916	1893	1893
q4	10305	1259	1355	1259
q5	8557	3916	4002	3916
q6	228	122	122	122
q7	2022	1619	1598	1598
q8	9485	2702	2704	2702
q9	13848	10416	10412	10412
q10	8650	3503	3505	3503
q11	407	246	237	237
q12	466	303	310	303
q13	18354	4008	4052	4008
q14	346	315	317	315
q15	502	455	462	455
q16	683	604	590	590
q17	1138	923	916	916
q18	7325	6976	6972	6972
q19	1663	1571	1521	1521
q20	537	317	318	317
q21	4465	4078	4087	4078
q22	501	410	398	398
Total cold run time: 120108 ms
Total hot run time: 49947 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4510	4298	4341	4298
q2	322	220	223	220
q3	4145	4189	4158	4158
q4	2747	2741	2739	2739
q5	7230	7187	7220	7187
q6	235	118	122	118
q7	3222	2838	2793	2793
q8	4388	4515	4451	4451
q9	17132	16890	16991	16890
q10	4222	4262	4258	4258
q11	741	691	707	691
q12	1012	863	886	863
q13	6520	3735	3761	3735
q14	448	438	425	425
q15	495	446	447	446
q16	768	714	702	702
q17	3869	3834	3857	3834
q18	8872	8760	8855	8760
q19	1735	1659	1657	1657
q20	2412	2130	2147	2130
q21	8534	8549	8614	8549
q22	1031	930	971	930
Total cold run time: 84590 ms
Total hot run time: 79834 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8035/21253)
Line Coverage: 29.49% (65543/222228)
Region Coverage: 28.96% (33713/116427)
Branch Coverage: 24.82% (17304/69732)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9f09db516001aba162f8ef7dd7dd8986b302fc0f_9f09db516001aba162f8ef7dd7dd8986b302fc0f/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 240261 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f09db516001aba162f8ef7dd7dd8986b302fc0f, data reload: false

query1	928	386	380	380
query2	6534	2205	1934	1934
query3	6922	202	204	202
query4	19884	17965	17973	17965
query5	19717	6522	6503	6503
query6	281	212	228	212
query7	4151	294	302	294
query8	255	256	235	235
query9	3108	2650	2600	2600
query10	401	301	316	301
query11	11193	10672	10633	10633
query12	122	72	74	72
query13	5601	630	624	624
query14	17575	13380	13425	13380
query15	378	244	234	234
query16	6452	279	257	257
query17	1572	1464	857	857
query18	2306	404	393	393
query19	195	148	148	148
query20	75	82	71	71
query21	187	94	92	92
query22	5250	5026	4984	4984
query23	32319	31990	31967	31967
query24	6927	6508	6438	6438
query25	521	428	417	417
query26	531	166	159	159
query27	1891	285	296	285
query28	6082	2223	2180	2180
query29	2775	2736	2738	2736
query30	235	157	159	157
query31	909	734	736	734
query32	63	57	60	57
query33	392	236	246	236
query34	847	455	486	455
query35	1120	940	930	930
query36	2476	1527	1647	1527
query37	89	66	61	61
query38	3094	2981	2929	2929
query39	1363	1318	1312	1312
query40	195	94	91	91
query41	35	36	30	30
query42	94	84	81	81
query43	741	570	555	555
query44	1125	708	724	708
query45	239	230	233	230
query46	1231	961	984	961
query47	1920	1660	1767	1660
query48	980	667	676	667
query49	600	365	360	360
query50	855	586	593	586
query51	5522	5418	5422	5418
query52	89	84	79	79
query53	445	335	322	322
query54	2635	2463	2492	2463
query55	96	83	81	81
query56	215	199	202	199
query57	1192	1055	1037	1037
query58	198	200	189	189
query59	3499	3347	3096	3096
query60	206	180	204	180
query61	82	83	83	83
query62	846	482	476	476
query63	473	340	338	338
query64	2377	1471	1435	1435
query65	3617	3577	3591	3577
query66	795	372	362	362
query67	17422	17261	15637	15637
query68	7410	655	661	655
query69	556	338	347	338
query70	2040	1675	1765	1675
query71	402	308	303	303
query72	6411	3434	3471	3434
query73	717	319	309	309
query74	6329	5913	5868	5868
query75	4535	3713	3657	3657
query76	4425	1169	1147	1147
query77	531	240	250	240
query78	32333	51735	49137	49137
query79	15787	662	644	644
query80	5121	385	390	385
query81	573	228	230	228
query82	1327	95	95	95
query83	335	130	128	128
query84	262	69	68	68
query85	2079	284	276	276
query86	452	379	365	365
query87	3237	2988	3069	2988
query88	7173	2309	2293	2293
query89	466	301	285	285
query90	2523	205	210	205
query91	152	123	114	114
query92	66	50	48	48
query93	6556	565	602	565
query94	1734	214	202	202
query95	1093	1083	1053	1053
query96	654	309	319	309
query97	6515	6363	6384	6363
query98	193	173	168	168
query99	3755	864	906	864
Total cold run time: 348187 ms
Total hot run time: 240261 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f09db516001aba162f8ef7dd7dd8986b302fc0f, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.25	0.04	0.04
query4	1.82	0.07	0.07
query5	0.53	0.51	0.52
query6	1.24	0.61	0.62
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.52	0.49	0.48
query10	0.53	0.54	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.78	0.80
query15	0.77	0.76	0.77
query16	0.36	0.39	0.38
query17	1.01	1.02	1.04
query18	0.21	0.26	0.26
query19	1.94	1.82	1.83
query20	0.01	0.01	0.01
query21	15.47	0.55	0.54
query22	2.03	2.60	1.25
query23	17.24	0.93	0.94
query24	4.27	5.28	0.78
query25	1.58	0.12	0.11
query26	0.39	0.14	0.14
query27	0.11	0.11	0.11
query28	5.87	0.72	0.71
query29	12.78	2.39	2.36
query30	0.60	0.48	0.53
query31	2.82	0.38	0.40
query32	3.39	0.50	0.49
query33	3.09	3.08	3.10
query34	15.29	4.79	4.81
query35	4.89	4.83	4.82
query36	1.04	1.01	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.01	0.01	0.01
query40	0.16	0.13	0.13
query41	0.06	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 102.19 s
Total hot run time: 30.35 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9f09db516001aba162f8ef7dd7dd8986b302fc0f with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 7e7fcf1 into apache:branch-2.0 Feb 21, 2024
25 of 29 checks passed
@zy-kkk zy-kkk deleted the jdbc_pool_improve branch March 22, 2024 03:15
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/planner Issues or PRs related to the query planner
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants