Skip to content

[fix](jdbc catalog) fix and add mysql and doris extremum test#41679

Merged
morningman merged 5 commits intoapache:masterfrom
zy-kkk:mysql_extremum_test
Oct 13, 2024
Merged

[fix](jdbc catalog) fix and add mysql and doris extremum test#41679
morningman merged 5 commits intoapache:masterfrom
zy-kkk:mysql_extremum_test

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Oct 10, 2024

1、add mysql jdbc catalog extremum test
2、mysql jdbc catalog time data outside the 24-hour range can be read
3、add doris jdbc catalog extremum test
4、fix doris jdbc catalog char type max size match
5、fix doris jdbc catalog nested array read

Legacy:
Doris float extreme values ​​are incorrect 3.4028235e+38, -3.4028235e+38
Correct values ​​should be 3.4028234e+38, -3.4028234e+38. The incorrect value will exceed the Java float range, so ignore this test and add it after it is fixed

This PR also followup #40122
Adds requiredColumnSize() and requiredDecimalDigits() methods to JdbcFieldSchema, improving error handling by throwing exceptions when values are absent.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zy-kkk
Copy link
Member Author

zy-kkk commented Oct 10, 2024

run buildall

@morningman morningman changed the title [test](jdbc catalog) add mysql and doris extremum test [fix](jdbc catalog) fix and add mysql and doris extremum test Oct 10, 2024
return resultSet.getObject(columnIndex + 1);
case STRING: {
int jdbcType = resultSetMetaData.getColumnType(columnIndex + 1);
if (jdbcType == Types.TIME) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add some comment to explain this if else

qt_select_char """select * from t_char order by 1;"""

sql """set dry_run_query=true"""
qt_select_all_types_multi_block """select * from all_types_multi_block;"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use select count(k1), count(k2), and remove dry_run_query

@zy-kkk
Copy link
Member Author

zy-kkk commented Oct 10, 2024

run buildall

1 similar comment
@zy-kkk
Copy link
Member Author

zy-kkk commented Oct 10, 2024

run buildall

@zy-kkk zy-kkk force-pushed the mysql_extremum_test branch from af6fa39 to 1eae2b8 Compare October 11, 2024 03:36
@zy-kkk
Copy link
Member Author

zy-kkk commented Oct 11, 2024

run buildall

@zy-kkk
Copy link
Member Author

zy-kkk commented Oct 11, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41010 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ee379834220c308135016136835a2607889b4ad, data reload: false

------ Round 1 ----------------------------------
q1	17700	7407	7236	7236
q2	2027	317	284	284
q3	12212	1086	1166	1086
q4	10567	810	731	731
q5	7758	2928	2872	2872
q6	237	148	151	148
q7	1020	637	621	621
q8	9379	1981	2014	1981
q9	6622	6472	6434	6434
q10	6993	2299	2306	2299
q11	435	251	249	249
q12	421	217	216	216
q13	17788	3008	2951	2951
q14	252	207	210	207
q15	572	525	514	514
q16	634	590	576	576
q17	973	562	533	533
q18	7353	6774	6750	6750
q19	1345	917	1004	917
q20	506	196	203	196
q21	4026	3217	3353	3217
q22	1135	992	1020	992
Total cold run time: 109955 ms
Total hot run time: 41010 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7239	7218	7230	7218
q2	321	232	225	225
q3	3057	3008	2888	2888
q4	2091	1781	1786	1781
q5	5758	5718	5698	5698
q6	231	141	144	141
q7	2410	1805	1806	1805
q8	3402	3613	3468	3468
q9	8965	8938	8799	8799
q10	3592	3539	3522	3522
q11	595	500	506	500
q12	845	601	600	600
q13	9677	3162	3165	3162
q14	310	273	268	268
q15	580	538	519	519
q16	690	635	654	635
q17	1855	1654	1596	1596
q18	8407	7578	7667	7578
q19	1753	1494	1499	1494
q20	2098	1877	1900	1877
q21	5557	5344	5389	5344
q22	1107	1071	1080	1071
Total cold run time: 70540 ms
Total hot run time: 60189 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191186 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8ee379834220c308135016136835a2607889b4ad, data reload: false

query1	906	403	399	399
query2	6242	2089	2007	2007
query3	8684	199	199	199
query4	34387	23466	23736	23466
query5	3513	475	455	455
query6	279	167	170	167
query7	4189	298	303	298
query8	293	222	217	217
query9	9624	2685	2687	2685
query10	476	282	279	279
query11	17820	15228	15052	15052
query12	145	102	94	94
query13	1587	462	456	456
query14	10147	7608	6860	6860
query15	259	171	168	168
query16	7652	462	477	462
query17	1640	604	581	581
query18	2078	306	315	306
query19	317	153	153	153
query20	116	119	112	112
query21	208	105	111	105
query22	4922	4516	4667	4516
query23	35370	34012	33884	33884
query24	10990	2789	2758	2758
query25	604	409	408	408
query26	1089	156	159	156
query27	2308	304	302	302
query28	7339	2421	2425	2421
query29	780	433	436	433
query30	277	154	154	154
query31	1025	789	774	774
query32	104	53	55	53
query33	760	295	307	295
query34	917	504	491	491
query35	859	729	722	722
query36	1111	947	983	947
query37	152	87	91	87
query38	3999	3889	3895	3889
query39	1476	1433	1437	1433
query40	213	97	98	97
query41	48	49	49	49
query42	123	100	98	98
query43	540	481	474	474
query44	1320	815	804	804
query45	194	164	173	164
query46	1144	714	711	711
query47	1991	1897	1869	1869
query48	434	338	341	338
query49	960	417	406	406
query50	815	434	420	420
query51	7122	6913	6906	6906
query52	102	85	87	85
query53	250	182	180	180
query54	1188	472	474	472
query55	80	79	80	79
query56	275	298	250	250
query57	1276	1144	1165	1144
query58	233	233	252	233
query59	3174	3226	3111	3111
query60	295	270	249	249
query61	107	102	120	102
query62	844	674	672	672
query63	222	186	181	181
query64	4142	637	624	624
query65	3237	3211	3189	3189
query66	876	338	296	296
query67	15829	15469	15553	15469
query68	4427	580	551	551
query69	492	284	294	284
query70	1073	1165	1134	1134
query71	341	269	261	261
query72	7331	4015	3967	3967
query73	788	337	349	337
query74	9909	9003	9036	9003
query75	3387	2625	2692	2625
query76	3036	907	1002	907
query77	418	293	282	282
query78	11016	9752	9591	9591
query79	1672	604	590	590
query80	1564	432	446	432
query81	589	249	235	235
query82	688	141	132	132
query83	281	136	132	132
query84	263	78	76	76
query85	850	301	289	289
query86	432	296	305	296
query87	4437	4250	4305	4250
query88	3051	2402	2358	2358
query89	408	291	279	279
query90	2061	179	180	179
query91	141	108	119	108
query92	65	47	44	44
query93	1905	552	543	543
query94	960	291	301	291
query95	347	249	250	249
query96	605	272	277	272
query97	3274	3083	3116	3083
query98	227	205	196	196
query99	1495	1295	1294	1294
Total cold run time: 299211 ms
Total hot run time: 191186 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.26 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8ee379834220c308135016136835a2607889b4ad, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.03	0.02
query3	0.22	0.06	0.06
query4	1.65	0.10	0.10
query5	0.52	0.50	0.52
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.56	0.50	0.48
query10	0.54	0.54	0.56
query11	0.14	0.10	0.12
query12	0.14	0.11	0.11
query13	0.63	0.60	0.60
query14	2.74	2.77	2.72
query15	0.91	0.83	0.84
query16	0.39	0.39	0.39
query17	1.03	1.05	1.02
query18	0.20	0.20	0.20
query19	1.98	1.84	2.02
query20	0.01	0.01	0.01
query21	15.37	0.58	0.58
query22	2.76	2.52	2.50
query23	16.99	1.12	0.95
query24	3.37	0.97	0.71
query25	0.28	0.27	0.07
query26	0.30	0.13	0.14
query27	0.04	0.05	0.04
query28	10.98	1.08	1.06
query29	12.54	3.26	3.21
query30	0.23	0.06	0.06
query31	2.89	0.39	0.38
query32	3.26	0.46	0.45
query33	3.02	3.06	3.03
query34	16.96	4.52	4.47
query35	4.55	4.52	4.56
query36	0.66	0.49	0.50
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.61 s
Total hot run time: 33.26 s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 13, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit 6879896 into apache:master Oct 13, 2024
qzsee pushed a commit to qzsee/incubator-doris that referenced this pull request Oct 16, 2024
…#41679)

1、add mysql jdbc catalog extremum test
2、mysql jdbc catalog time data outside the 24-hour range can be read
3、add doris jdbc catalog extremum test
4、fix doris jdbc catalog char type max size match
5、fix doris jdbc catalog nested array read

Legacy:
Doris float extreme values ​​are incorrect 3.4028235e+38, -3.4028235e+38
Correct values ​​should be 3.4028234e+38, -3.4028234e+38. The incorrect
value will exceed the Java float range, so ignore this test and add it
after it is fixed

This PR also followup apache#40122
Adds requiredColumnSize() and requiredDecimalDigits() methods to
JdbcFieldSchema, improving error handling by throwing exceptions when
values are absent.
morningman pushed a commit to morningman/doris that referenced this pull request Oct 18, 2024
…#41679)

1、add mysql jdbc catalog extremum test
2、mysql jdbc catalog time data outside the 24-hour range can be read
3、add doris jdbc catalog extremum test
4、fix doris jdbc catalog char type max size match
5、fix doris jdbc catalog nested array read

Legacy:
Doris float extreme values ​​are incorrect 3.4028235e+38, -3.4028235e+38
Correct values ​​should be 3.4028234e+38, -3.4028234e+38. The incorrect
value will exceed the Java float range, so ignore this test and add it
after it is fixed

This PR also followup apache#40122
Adds requiredColumnSize() and requiredDecimalDigits() methods to
JdbcFieldSchema, improving error handling by throwing exceptions when
values are absent.
morningman pushed a commit to morningman/doris that referenced this pull request Oct 21, 2024
…#41679)

1、add mysql jdbc catalog extremum test
2、mysql jdbc catalog time data outside the 24-hour range can be read
3、add doris jdbc catalog extremum test
4、fix doris jdbc catalog char type max size match
5、fix doris jdbc catalog nested array read

Legacy:
Doris float extreme values ​​are incorrect 3.4028235e+38, -3.4028235e+38
Correct values ​​should be 3.4028234e+38, -3.4028234e+38. The incorrect
value will exceed the Java float range, so ignore this test and add it
after it is fixed

This PR also followup apache#40122
Adds requiredColumnSize() and requiredDecimalDigits() methods to
JdbcFieldSchema, improving error handling by throwing exceptions when
values are absent.
morningman added a commit that referenced this pull request Oct 21, 2024
…42180)

cherry pick from #41679

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
morningman added a commit that referenced this pull request Oct 21, 2024
…42122)

cherry pick from #41679

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
@zy-kkk zy-kkk deleted the mysql_extremum_test branch June 25, 2025 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants