Skip to content

[branch-2.0][improvement](serde) Handle NaN values in number for MySQL result write#33230

Merged
morningman merged 2 commits intoapache:branch-2.0from
zy-kkk:float_mysql_result2
Apr 4, 2024
Merged

[branch-2.0][improvement](serde) Handle NaN values in number for MySQL result write#33230
morningman merged 2 commits intoapache:branch-2.0from
zy-kkk:float_mysql_result2

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Apr 3, 2024

Proposed changes

Issue Number: close #xxx

pick #33227

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zy-kkk
Copy link
Member Author

zy-kkk commented Apr 3, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49936 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40166ddc248f2a3d168f152dfd7b37699414fd13, data reload: false

------ Round 1 ----------------------------------
q1	17813	4373	4313	4313
q2	2111	149	142	142
q3	10289	1928	1937	1928
q4	10107	1200	1315	1200
q5	8424	3961	3982	3961
q6	232	124	122	122
q7	2078	1599	1595	1595
q8	9274	2712	2715	2712
q9	10869	10698	10498	10498
q10	8727	3547	3545	3545
q11	408	244	239	239
q12	467	294	298	294
q13	18329	3974	4017	3974
q14	355	323	328	323
q15	491	457	460	457
q16	686	592	577	577
q17	1138	951	953	951
q18	7366	6840	6828	6828
q19	1692	1533	1542	1533
q20	535	299	309	299
q21	4477	4107	4047	4047
q22	498	413	398	398
Total cold run time: 116366 ms
Total hot run time: 49936 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4350	4313	4327	4313
q2	325	226	220	220
q3	4180	4147	4093	4093
q4	2744	2746	2755	2746
q5	7276	7179	7093	7093
q6	230	118	119	118
q7	3190	2884	2824	2824
q8	4351	4445	4447	4445
q9	17092	16944	16969	16944
q10	4217	4214	4273	4214
q11	785	664	656	656
q12	1022	853	876	853
q13	7168	3708	3752	3708
q14	441	422	415	415
q15	509	461	456	456
q16	759	700	706	700
q17	3833	3884	3828	3828
q18	8726	8868	8840	8840
q19	1725	1703	1594	1594
q20	2437	2120	2111	2111
q21	8514	8504	8551	8504
q22	1087	986	979	979
Total cold run time: 84961 ms
Total hot run time: 79654 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.79% (8048/21297)
Line Coverage: 29.48% (65779/223144)
Region Coverage: 28.93% (33835/116953)
Branch Coverage: 24.78% (17373/70100)
Coverage Report: http://coverage.selectdb-in.cc/coverage/40166ddc248f2a3d168f152dfd7b37699414fd13_40166ddc248f2a3d168f152dfd7b37699414fd13/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 202595 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40166ddc248f2a3d168f152dfd7b37699414fd13, data reload: false

query1	914	391	383	383
query2	6554	2154	2135	2135
query3	6925	206	204	204
query4	20362	18070	18039	18039
query5	19720	6551	6528	6528
query6	295	214	232	214
query7	4153	312	307	307
query8	266	263	224	224
query9	3210	2773	2685	2685
query10	415	295	294	294
query11	11533	10880	10763	10763
query12	120	82	68	68
query13	5576	669	653	653
query14	18219	13642	13556	13556
query15	359	238	236	236
query16	6450	270	257	257
query17	1734	1449	881	881
query18	2288	412	401	401
query19	207	146	151	146
query20	78	76	75	75
query21	187	97	87	87
query22	5346	5177	5243	5177
query23	32470	31771	31921	31771
query24	6893	6543	6538	6538
query25	515	453	413	413
query26	520	167	166	166
query27	1876	300	308	300
query28	6134	2288	2253	2253
query29	2856	2655	2674	2655
query30	248	165	159	159
query31	915	726	770	726
query32	68	60	57	57
query33	401	244	242	242
query34	854	480	498	480
query35	1097	929	861	861
query36	1257	1328	1159	1159
query37	90	58	61	58
query38	3069	2911	2918	2911
query39	1386	1313	1301	1301
query40	211	97	97	97
query41	41	32	32	32
query42	81	83	87	83
query43	722	625	576	576
query44	1203	745	734	734
query45	242	233	233	233
query46	1251	1007	985	985
query47	1814	1852	1735	1735
query48	1001	701	696	696
query49	618	371	366	366
query50	876	680	683	680
query51	4700	4704	4691	4691
query52	96	84	80	80
query53	460	321	325	321
query54	2665	2447	2484	2447
query55	83	75	76	75
query56	215	199	197	197
query57	1164	1094	1242	1094
query58	207	186	202	186
query59	3505	3477	3248	3248
query60	219	205	197	197
query61	87	91	89	89
query62	824	454	469	454
query63	486	336	345	336
query64	2554	1486	1438	1438
query65	3604	3600	3577	3577
query66	819	372	359	359
query67	15373	15460	15172	15172
query68	8353	656	680	656
query69	555	333	356	333
query70	1638	1542	1367	1367
query71	388	297	304	297
query72	6587	3468	3434	3434
query73	743	323	319	319
query74	6290	5916	5820	5820
query75	4563	3715	3701	3701
query76	4673	1179	1220	1179
query77	630	247	249	247
query78	12730	11851	11972	11851
query79	8868	637	666	637
query80	1668	411	398	398
query81	490	238	225	225
query82	1678	95	93	93
query83	169	136	127	127
query84	257	67	69	67
query85	1039	287	281	281
query86	342	313	341	313
query87	3231	3076	3071	3071
query88	5288	2377	2395	2377
query89	390	289	284	284
query90	1831	214	214	214
query91	150	123	123	123
query92	60	48	50	48
query93	5440	599	605	599
query94	781	211	203	203
query95	1122	1057	1068	1057
query96	677	344	336	336
query97	6467	6422	6304	6304
query98	188	171	172	171
query99	2925	947	922	922
Total cold run time: 310892 ms
Total hot run time: 202595 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40166ddc248f2a3d168f152dfd7b37699414fd13, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.25	0.05	0.04
query4	1.80	0.08	0.08
query5	0.53	0.52	0.52
query6	1.26	0.63	0.62
query7	0.01	0.01	0.01
query8	0.03	0.03	0.03
query9	0.53	0.46	0.49
query10	0.53	0.54	0.54
query11	0.12	0.08	0.08
query12	0.11	0.08	0.09
query13	0.62	0.62	0.62
query14	0.78	0.80	0.79
query15	0.78	0.77	0.77
query16	0.36	0.37	0.37
query17	1.00	1.01	1.02
query18	0.23	0.25	0.24
query19	1.96	1.90	1.89
query20	0.02	0.01	0.02
query21	15.45	0.62	0.54
query22	2.23	2.83	1.51
query23	17.46	1.06	0.93
query24	5.29	0.97	0.70
query25	0.31	0.14	0.07
query26	0.56	0.15	0.16
query27	0.04	0.04	0.04
query28	8.21	0.73	0.70
query29	12.72	2.32	2.32
query30	0.61	0.53	0.53
query31	2.82	0.39	0.37
query32	3.41	0.51	0.52
query33	3.08	3.09	3.10
query34	15.26	4.78	4.81
query35	4.86	4.83	4.84
query36	1.05	1.00	1.02
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.01	0.02	0.01
query40	0.17	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 104.74 s
Total hot run time: 30.51 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 40166ddc248f2a3d168f152dfd7b37699414fd13 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.4 seconds inserted 10000000 Rows, about 490K ops/s

@zy-kkk
Copy link
Member Author

zy-kkk commented Apr 3, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.80% (8055/21307)
Line Coverage: 29.48% (65780/223155)
Region Coverage: 28.94% (33840/116945)
Branch Coverage: 24.79% (17375/70076)
Coverage Report: http://coverage.selectdb-in.cc/coverage/66cecd59b50b6aab5354d8bc1fb5497dbcc123b5_66cecd59b50b6aab5354d8bc1fb5497dbcc123b5/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 50250 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66cecd59b50b6aab5354d8bc1fb5497dbcc123b5, data reload: false

------ Round 1 ----------------------------------
q1	17573	4443	4383	4383
q2	2044	148	141	141
q3	10351	1935	1973	1935
q4	10117	1275	1322	1275
q5	8785	4230	4003	4003
q6	232	126	122	122
q7	2049	1646	1604	1604
q8	9297	2772	2743	2743
q9	10724	10582	10500	10500
q10	8634	3542	3532	3532
q11	419	245	234	234
q12	471	299	303	299
q13	18348	3973	4037	3973
q14	343	318	322	318
q15	512	457	450	450
q16	698	595	589	589
q17	1136	1010	971	971
q18	7228	6896	6844	6844
q19	1683	1547	1488	1488
q20	514	301	298	298
q21	4529	4151	4147	4147
q22	503	401	406	401
Total cold run time: 116190 ms
Total hot run time: 50250 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4352	4305	4304	4304
q2	323	225	231	225
q3	4141	4163	4150	4150
q4	2769	2746	2760	2746
q5	7291	7235	7206	7206
q6	241	123	117	117
q7	3238	2866	2866	2866
q8	4369	4436	4454	4436
q9	17019	16914	17010	16914
q10	4269	4281	4302	4281
q11	745	669	691	669
q12	1033	863	828	828
q13	7181	3731	3746	3731
q14	457	421	417	417
q15	496	451	449	449
q16	757	698	702	698
q17	3841	3765	3884	3765
q18	8867	8738	8820	8738
q19	1739	1710	1663	1663
q20	2417	2166	2146	2146
q21	8521	8552	8573	8552
q22	1049	975	968	968
Total cold run time: 85115 ms
Total hot run time: 79869 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201430 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66cecd59b50b6aab5354d8bc1fb5497dbcc123b5, data reload: false

query1	913	390	379	379
query2	6520	2290	2239	2239
query3	6925	200	202	200
query4	20996	17960	17973	17960
query5	19705	6530	6508	6508
query6	275	219	237	219
query7	4155	293	299	293
query8	265	220	242	220
query9	3132	2733	2648	2648
query10	419	287	301	287
query11	11483	10750	10832	10750
query12	122	72	69	69
query13	5569	652	644	644
query14	17616	13416	13433	13416
query15	358	229	248	229
query16	6447	274	257	257
query17	1716	1450	846	846
query18	2357	408	403	403
query19	216	144	144	144
query20	74	77	77	77
query21	185	95	89	89
query22	5354	5035	5034	5034
query23	32765	31825	31987	31825
query24	7042	6593	6482	6482
query25	530	434	421	421
query26	535	160	158	158
query27	1911	300	292	292
query28	6131	2294	2256	2256
query29	2884	2620	2910	2620
query30	241	162	161	161
query31	917	728	727	727
query32	66	63	58	58
query33	393	250	251	250
query34	853	463	467	463
query35	1082	905	903	903
query36	1431	1127	1236	1127
query37	88	64	61	61
query38	3108	2927	2903	2903
query39	1375	1314	1312	1312
query40	210	96	98	96
query41	35	34	32	32
query42	85	81	82	81
query43	648	574	600	574
query44	1111	728	727	727
query45	238	231	225	225
query46	1221	969	987	969
query47	1822	1936	1853	1853
query48	996	688	656	656
query49	622	372	379	372
query50	869	630	610	610
query51	4739	4647	4731	4647
query52	85	73	76	73
query53	444	312	316	312
query54	2815	2511	2506	2506
query55	79	69	78	69
query56	213	218	210	210
query57	1140	1197	1100	1100
query58	219	204	204	204
query59	3687	3568	3130	3130
query60	209	195	209	195
query61	88	88	91	88
query62	784	455	509	455
query63	470	328	329	328
query64	2423	1465	1389	1389
query65	3619	3560	3571	3560
query66	814	371	362	362
query67	18116	15248	15076	15076
query68	9523	660	686	660
query69	563	353	367	353
query70	1756	1456	1368	1368
query71	405	313	319	313
query72	6596	3451	3460	3451
query73	723	338	325	325
query74	6360	5910	5878	5878
query75	5389	3730	3666	3666
query76	6094	1146	1189	1146
query77	999	261	251	251
query78	12624	11557	11598	11557
query79	6282	637	656	637
query80	958	403	392	392
query81	494	226	230	226
query82	1407	100	98	98
query83	164	137	131	131
query84	255	70	72	70
query85	848	293	287	287
query86	339	301	285	285
query87	3198	3024	2992	2992
query88	4550	2384	2374	2374
query89	374	292	272	272
query90	1954	190	207	190
query91	156	114	118	114
query92	55	50	51	50
query93	3899	629	590	590
query94	698	204	208	204
query95	1112	1066	1049	1049
query96	643	340	329	329
query97	6481	6265	6362	6265
query98	197	186	179	179
query99	2942	905	899	899
Total cold run time: 311993 ms
Total hot run time: 201430 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.7 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66cecd59b50b6aab5354d8bc1fb5497dbcc123b5, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.26	0.05	0.05
query4	1.78	0.07	0.08
query5	0.52	0.52	0.52
query6	1.23	0.61	0.62
query7	0.02	0.02	0.02
query8	0.03	0.02	0.02
query9	0.53	0.47	0.48
query10	0.54	0.53	0.53
query11	0.12	0.09	0.09
query12	0.13	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.78	0.80
query15	0.79	0.76	0.76
query16	0.39	0.36	0.36
query17	0.97	1.00	0.98
query18	0.24	0.24	0.25
query19	1.90	1.86	1.86
query20	0.01	0.01	0.01
query21	15.47	0.54	0.53
query22	2.28	2.98	1.41
query23	17.29	1.03	1.06
query24	5.67	1.72	1.14
query25	0.41	0.12	0.05
query26	0.63	0.16	0.17
query27	0.04	0.05	0.04
query28	6.64	0.78	0.72
query29	12.78	2.25	2.23
query30	0.62	0.54	0.53
query31	2.82	0.39	0.37
query32	3.35	0.48	0.48
query33	3.05	3.08	3.10
query34	15.30	4.81	4.76
query35	4.88	4.82	4.84
query36	1.04	1.01	1.01
query37	0.05	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.13	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 103.59 s
Total hot run time: 30.7 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 66cecd59b50b6aab5354d8bc1fb5497dbcc123b5 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.2 seconds inserted 10000000 Rows, about 495K ops/s

@morningman morningman merged commit 71872cd into apache:branch-2.0 Apr 4, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants