Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](connections) fix connection hang after too many connections #31594

Merged
merged 1 commit into from
Mar 4, 2024
Merged

[fix](connections) fix connection hang after too many connections #31594

merged 1 commit into from
Mar 4, 2024

Conversation

cambyzju
Copy link
Contributor

@cambyzju cambyzju commented Feb 29, 2024 •
edited
Loading

Proposed changes

Issue Number: close #31569

After fix, the third SQL will return error instead of hang:
ERROR 1203 (HY000): #42000Too many connections

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@cambyzju
Copy link
Contributor Author

run buildall

@cambyzju
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37674 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f0d24091b69e3a013705f99644c5c5f716adc5c, data reload: false

------ Round 1 ----------------------------------
q1	17743	4198	4036	4036
q2	2052	140	139	139
q3	10662	938	940	938
q4	4668	928	943	928
q5	7603	2865	3031	2865
q6	180	125	121	121
q7	1213	751	742	742
q8	9264	1945	2034	1945
q9	7179	6254	6270	6254
q10	8188	2489	2488	2488
q11	445	215	196	196
q12	767	314	311	311
q13	17942	3222	3218	3218
q14	273	265	241	241
q15	534	490	495	490
q16	469	403	392	392
q17	928	896	864	864
q18	6717	6097	6005	6005
q19	1557	1501	1501	1501
q20	542	277	275	275
q21	6228	3458	3588	3458
q22	792	294	267	267
Total cold run time: 105946 ms
Total hot run time: 37674 ms

----- Round 2, with runtime_filter_mode=off -----
q1	3996	3982	3958	3958
q2	298	214	212	212
q3	2978	2949	2907	2907
q4	1831	1814	1814	1814
q5	5148	5101	5103	5101
q6	197	117	117	117
q7	2121	1720	1777	1720
q8	3129	3213	3196	3196
q9	8225	8221	8204	8204
q10	6094	3629	3658	3629
q11	502	435	432	432
q12	685	509	558	509
q13	11685	3036	3038	3036
q14	290	252	245	245
q15	544	487	492	487
q16	502	473	458	458
q17	1719	1664	1661	1661
q18	7991	7532	7339	7339
q19	1656	1630	1615	1615
q20	2099	1920	1882	1882
q21	4745	4584	4628	4584
q22	517	443	432	432
Total cold run time: 66952 ms
Total hot run time: 53538 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168358 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f0d24091b69e3a013705f99644c5c5f716adc5c, data reload: false

query1	970	344	340	340
query2	6720	1730	1654	1654
query3	6700	212	210	210
query4	23121	20550	20553	20550
query5	4273	362	361	361
query6	258	170	175	170
query7	4607	295	299	295
query8	268	187	192	187
query9	8424	2260	2242	2242
query10	412	209	219	209
query11	14591	14187	14075	14075
query12	137	84	81	81
query13	1644	438	433	433
query14	8845	6546	6276	6276
query15	201	174	182	174
query16	7093	275	273	273
query17	1010	515	492	492
query18	1896	270	265	265
query19	201	146	150	146
query20	83	82	86	82
query21	195	126	116	116
query22	4694	4525	4270	4270
query23	30437	29752	29872	29752
query24	11025	3108	3135	3108
query25	625	341	350	341
query26	1656	152	165	152
query27	3017	313	321	313
query28	7025	1844	1824	1824
query29	1106	557	548	548
query30	284	132	139	132
query31	850	690	700	690
query32	99	57	56	56
query33	714	219	215	215
query34	1058	495	495	495
query35	841	749	737	737
query36	941	858	886	858
query37	129	61	63	61
query38	3065	2997	2978	2978
query39	1306	1259	1244	1244
query40	279	102	102	102
query41	46	38	36	36
query42	99	94	98	94
query43	443	417	423	417
query44	1064	695	717	695
query45	192	184	173	173
query46	1058	816	787	787
query47	1508	1436	1471	1436
query48	408	348	341	341
query49	1183	296	288	288
query50	770	364	372	364
query51	4325	4286	4225	4225
query52	116	88	93	88
query53	347	270	259	259
query54	300	218	224	218
query55	83	85	83	83
query56	215	201	197	197
query57	958	913	854	854
query58	212	200	190	190
query59	2160	2076	2100	2076
query60	234	213	212	212
query61	86	86	85	85
query62	609	354	356	354
query63	283	254	256	254
query64	6633	2936	3121	2936
query65	3520	3219	3199	3199
query66	1448	322	316	316
query67	14210	14183	14027	14027
query68	4041	579	569	569
query69	504	383	368	368
query70	1228	1182	1170	1170
query71	334	239	254	239
query72	6036	2835	2656	2656
query73	704	321	331	321
query74	6289	6051	6007	6007
query75	3000	2420	2431	2420
query76	2368	1074	1150	1074
query77	324	239	227	227
query78	9041	8512	8362	8362
query79	1102	521	512	512
query80	1005	357	343	343
query81	470	198	195	195
query82	959	85	82	82
query83	227	127	120	120
query84	277	77	78	77
query85	1324	361	351	351
query86	361	289	295	289
query87	3249	3054	3095	3054
query88	2738	2348	2347	2347
query89	396	386	327	327
query90	1874	168	166	166
query91	152	142	123	123
query92	54	51	50	50
query93	1191	528	500	500
query94	1245	185	186	185
query95	447	343	346	343
query96	571	266	268	266
query97	4270	4077	4160	4077
query98	219	207	196	196
query99	1068	669	631	631
Total cold run time: 260378 ms
Total hot run time: 168358 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f0d24091b69e3a013705f99644c5c5f716adc5c, data reload: false

query1	0.05	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.67	0.10	0.10
query5	0.52	0.51	0.53
query6	1.30	0.66	0.69
query7	0.01	0.01	0.02
query8	0.04	0.03	0.02
query9	0.56	0.54	0.51
query10	0.58	0.55	0.56
query11	0.14	0.10	0.10
query12	0.13	0.10	0.10
query13	0.63	0.62	0.62
query14	0.81	0.81	0.78
query15	0.84	0.82	0.82
query16	0.38	0.37	0.39
query17	0.99	0.95	0.99
query18	0.25	0.25	0.26
query19	1.77	1.71	1.71
query20	0.01	0.01	0.01
query21	15.40	0.62	0.55
query22	2.76	3.74	2.76
query23	17.52	1.12	1.03
query24	2.14	0.57	0.18
query25	0.25	0.14	0.04
query26	0.16	0.15	0.13
query27	0.03	0.03	0.03
query28	12.30	0.83	0.83
query29	12.62	3.15	3.24
query30	0.54	0.54	0.52
query31	2.78	0.36	0.36
query32	3.37	0.49	0.48
query33	3.22	3.16	3.24
query34	15.33	4.74	4.77
query35	4.80	4.76	4.77
query36	1.13	1.07	1.08
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.01	0.01
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.73 s
Total hot run time: 32.23 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9f0d24091b69e3a013705f99644c5c5f716adc5c with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       19.0 seconds inserted 10000000 Rows, about 526K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

xy720
xy720 approved these changes Feb 29, 2024
View reviewed changes
Copy link
Member

@xy720 xy720 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lide-reed lide-reed self-requested a review March 4, 2024 02:31
lide-reed
lide-reed approved these changes Mar 4, 2024
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xy720 xy720 merged commit 28b15b2 into apache:master Mar 4, 2024
31 of 34 checks passed
@cambyzju cambyzju mentioned this pull request Mar 4, 2024
Merged
yiguolei pushed a commit that referenced this pull request Mar 6, 2024
@cambyzju @Doris-Extras
[fix](connections) fix connection hang after too many connections (#3…
a26b1ee 
…1594)

Issue Number: close #31569

Fix fe connection hang after too high qps

After fix, the third SQL will return error instead of hang:
ERROR 1203 (HY000): #42000Too many connections
@yiguolei yiguolei mentioned this pull request Mar 24, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lide-reed lide-reed lide-reed approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@xy720 xy720 xy720 approved these changes

@cjj2010 cjj2010 cjj2010 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.6-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] Connection hang after too high qps
8 participants
@cambyzju @doris-robot @lide-reed @CalvinKirs @xy720 @cjj2010 @xiaokang @morningman