Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](mysql) Fix Connection Attributes Parsing Logic for Compatibility #32177

Merged
merged 1 commit into from
Mar 13, 2024
Merged

[fix](mysql) Fix Connection Attributes Parsing Logic for Compatibility #32177

merged 1 commit into from
Mar 13, 2024

Conversation

zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Mar 13, 2024 •
edited
Loading

Proposed changes

Issue Number: close #31857

This PR addresses compatibility issues with the latest version of the MySQL Python Connector. The previous implementation failed to correctly handle connection attributes (conn_attrs), leading to handshake failures. By adjusting the logic for reading connection attribute data to ensure we do not exceed the actual length of the attribute section when parsing key-value pairs, we have resolved the compatibility issue. This change allows our custom MySQL protocol implementation to correctly parse connection requests from the new version of the Connector.

Thanks to @liujiwen-up for helping with this

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@zy-kkk
Copy link
Member Author

zy-kkk commented Mar 13, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 36290 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17da13cdbc4f41d46d6aa43d1a74cd32b1065fd7, data reload: false

------ Round 1 ----------------------------------
q1	17648	4228	4103	4103
q2	2016	150	145	145
q3	10593	924	883	883
q4	4653	710	674	674
q5	7602	2764	2733	2733
q6	188	122	121	121
q7	1191	808	766	766
q8	9487	2029	2033	2029
q9	7119	6404	6455	6404
q10	8115	2490	2493	2490
q11	418	222	224	222
q12	757	293	291	291
q13	17961	2995	2982	2982
q14	283	257	250	250
q15	497	458	459	458
q16	470	394	389	389
q17	935	523	608	523
q18	6584	5968	5746	5746
q19	1573	1385	1350	1350
q20	557	277	272	272
q21	7350	3170	3295	3170
q22	806	289	297	289
Total cold run time: 106803 ms
Total hot run time: 36290 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4109	4056	4029	4029
q2	326	236	235	235
q3	2904	2803	2782	2782
q4	1835	1565	1554	1554
q5	5177	5217	5225	5217
q6	199	123	115	115
q7	2216	1834	1833	1833
q8	3119	3272	3268	3268
q9	8518	8525	8559	8525
q10	6149	3659	3660	3659
q11	540	448	437	437
q12	730	578	564	564
q13	12046	2863	2873	2863
q14	278	244	263	244
q15	474	444	437	437
q16	440	403	389	389
q17	1708	1474	1453	1453
q18	7580	7160	7042	7042
q19	6192	1552	1505	1505
q20	1916	1679	1699	1679
q21	4760	4666	4638	4638
q22	519	458	485	458
Total cold run time: 71735 ms
Total hot run time: 52926 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186530 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17da13cdbc4f41d46d6aa43d1a74cd32b1065fd7, data reload: false

query1	917	353	337	337
query2	7407	2009	1978	1978
query3	6709	215	217	215
query4	30980	20707	20631	20631
query5	4334	395	390	390
query6	263	170	182	170
query7	4622	292	290	290
query8	221	167	180	167
query9	9891	2231	2221	2221
query10	503	230	237	230
query11	14697	14071	14163	14071
query12	130	88	83	83
query13	1633	435	418	418
query14	11301	10492	10624	10492
query15	260	225	206	206
query16	7969	259	254	254
query17	1949	538	525	525
query18	1882	267	263	263
query19	232	154	149	149
query20	92	80	83	80
query21	196	130	120	120
query22	4672	4456	4432	4432
query23	31768	30774	30974	30774
query24	10864	2774	2784	2774
query25	580	352	350	350
query26	951	154	151	151
query27	2512	339	346	339
query28	7018	1860	1829	1829
query29	885	638	623	623
query30	303	149	159	149
query31	957	717	711	711
query32	100	57	53	53
query33	762	246	241	241
query34	1111	464	478	464
query35	815	598	601	598
query36	1010	887	861	861
query37	113	72	75	72
query38	3259	3122	3092	3092
query39	1403	1380	1363	1363
query40	211	108	110	108
query41	48	46	43	43
query42	101	96	96	96
query43	496	443	452	443
query44	1322	697	709	697
query45	281	252	242	242
query46	1074	685	677	677
query47	1673	1580	1586	1580
query48	447	345	358	345
query49	1109	334	327	327
query50	754	364	367	364
query51	6861	6890	6857	6857
query52	110	90	90	90
query53	336	282	277	277
query54	293	237	235	235
query55	89	80	79	79
query56	237	229	216	216
query57	1136	1018	1027	1018
query58	225	199	195	195
query59	2834	2894	2736	2736
query60	249	242	252	242
query61	112	104	105	104
query62	597	415	397	397
query63	298	272	276	272
query64	5546	3850	3825	3825
query65	3033	3018	2989	2989
query66	850	309	333	309
query67	14733	14204	14389	14204
query68	6950	523	530	523
query69	619	410	381	381
query70	1210	1184	1173	1173
query71	526	289	270	270
query72	6816	2761	2601	2601
query73	715	310	321	310
query74	7124	6540	6484	6484
query75	9485	8094	8070	8070
query76	4933	921	937	921
query77	663	267	263	263
query78	10284	9622	9646	9622
query79	9858	523	527	523
query80	1677	406	390	390
query81	504	211	214	211
query82	819	203	203	203
query83	212	147	144	144
query84	292	78	79	78
query85	1438	317	300	300
query86	416	315	280	280
query87	3453	3195	3229	3195
query88	4777	2312	2294	2294
query89	508	364	366	364
query90	2035	179	174	174
query91	164	121	124	121
query92	57	46	47	46
query93	6671	504	495	495
query94	1193	178	174	174
query95	427	332	320	320
query96	616	273	272	272
query97	4421	4255	4253	4253
query98	215	203	201	201
query99	1107	762	741	741
Total cold run time: 310511 ms
Total hot run time: 186530 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.6 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17da13cdbc4f41d46d6aa43d1a74cd32b1065fd7, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.67	0.10	0.10
query5	0.52	0.51	0.57
query6	1.20	0.62	0.63
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.54	0.48	0.50
query10	0.55	0.53	0.53
query11	0.12	0.09	0.10
query12	0.12	0.09	0.09
query13	0.57	0.57	0.56
query14	0.74	0.73	0.75
query15	0.79	0.77	0.76
query16	0.40	0.38	0.37
query17	1.00	1.00	0.96
query18	0.20	0.22	0.26
query19	1.85	1.70	1.77
query20	0.01	0.01	0.01
query21	15.40	0.54	0.56
query22	2.58	1.80	1.72
query23	17.32	0.76	0.70
query24	3.14	0.38	2.09
query25	0.28	0.07	0.19
query26	0.52	0.11	0.12
query27	0.03	0.03	0.03
query28	11.06	0.88	0.86
query29	12.61	3.24	3.38
query30	0.68	0.62	0.57
query31	2.80	0.33	0.34
query32	3.40	0.44	0.42
query33	2.84	2.86	2.78
query34	16.20	4.38	4.40
query35	4.32	4.30	4.36
query36	1.08	1.00	1.01
query37	0.06	0.05	0.06
query38	0.05	0.03	0.03
query39	0.02	0.02	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 105.35 s
Total hot run time: 29.6 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 17da13cdbc4f41d46d6aa43d1a74cd32b1065fd7 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.9 seconds inserted 10000000 Rows, about 719K ops/s

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2024
@zy-kkk zy-kkk merged commit cfa403f into apache:master Mar 13, 2024
28 of 31 checks passed
@zy-kkk zy-kkk deleted the fix_mysql_conn_attrs branch March 13, 2024 13:26
@wm1581066 wm1581066 added the usercase Important user case type label label Mar 14, 2024
zy-kkk added a commit to zy-kkk/doris that referenced this pull request Mar 15, 2024
@zy-kkk
[fix](mysql) Fix Connection Attributes Parsing Logic for Compatibility (
f881893 
apache#32177)
@zy-kkk zy-kkk mentioned this pull request Mar 15, 2024
Merged
yiguolei pushed a commit that referenced this pull request Mar 15, 2024
@zy-kkk @Doris-Extras
[fix](mysql) Fix Connection Attributes Parsing Logic for Compatibility (
985c4a3 
#32177)
zy-kkk added a commit that referenced this pull request Mar 16, 2024
@zy-kkk
[fix](mysql) Fix Connection Attributes Parsing Logic for Compatibility (
ba1d8fe 
#32177) (#32302)
@methane methane mentioned this pull request Jul 9, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zy-kkk
[fix](mysql) Fix Connection Attributes Parsing Logic for Compatibility (
e5f1eb6 
apache#32177) (apache#32302)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liujiwen-up liujiwen-up liujiwen-up approved these changes

@JNSimba JNSimba JNSimba approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.7-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] ky10 aarch64 mysql-connector-python cannot connect to FE
5 participants
@zy-kkk @doris-robot @JNSimba @liujiwen-up @wm1581066