[improvement](serde) Handle NaN values in number for MySQL result write #33227

zy-kkk · 2024-04-03T09:42:10Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-03T09:42:15Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zy-kkk · 2024-04-03T09:42:56Z

run buildall

github-actions · 2024-04-03T09:47:31Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

LGTM

github-actions · 2024-04-03T09:49:15Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-03T09:49:17Z

PR approved by anyone and no changes requested.

doris-robot · 2024-04-03T09:59:20Z

TeamCity be ut coverage result:
Function Coverage: 35.65% (8882/24915)
Line Coverage: 27.37% (72902/266351)
Region Coverage: 26.54% (37693/142018)
Branch Coverage: 23.34% (19207/82286)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c6ff442967eed53de2abfdff58d41663f8d144ea_c6ff442967eed53de2abfdff58d41663f8d144ea/report/index.html

doris-robot · 2024-04-03T10:00:34Z

TPC-H: Total hot run time: 38897 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c6ff442967eed53de2abfdff58d41663f8d144ea, data reload: false

------ Round 1 ----------------------------------
q1	18119	4235	4171	4171
q2	2398	195	188	188
q3	11277	1262	1448	1262
q4	10838	948	1048	948
q5	7584	2975	2931	2931
q6	215	133	133	133
q7	1089	606	616	606
q8	9401	2065	2028	2028
q9	6750	6200	6132	6132
q10	8407	3515	3581	3515
q11	410	229	229	229
q12	382	214	214	214
q13	17779	2915	2910	2910
q14	271	237	249	237
q15	534	492	478	478
q16	507	379	372	372
q17	969	904	880	880
q18	7561	6425	6456	6425
q19	1611	1532	1528	1528
q20	603	308	311	308
q21	3545	3110	3106	3106
q22	352	296	297	296
Total cold run time: 110602 ms
Total hot run time: 38897 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4047	4048	4046	4046
q2	329	218	227	218
q3	2938	2950	2967	2950
q4	1875	1871	1833	1833
q5	5254	5209	5200	5200
q6	207	124	123	123
q7	2242	1808	1816	1808
q8	3229	3279	3278	3278
q9	8512	8460	8495	8460
q10	3775	3803	3830	3803
q11	538	455	456	455
q12	734	562	557	557
q13	16835	2897	2881	2881
q14	308	259	263	259
q15	511	482	480	480
q16	459	405	396	396
q17	1733	1673	1687	1673
q18	7643	7276	7376	7276
q19	1635	1629	1633	1629
q20	1953	1718	1716	1716
q21	5020	4646	4718	4646
q22	487	422	428	422
Total cold run time: 70264 ms
Total hot run time: 54109 ms

doris-robot · 2024-04-03T10:12:15Z

TPC-DS: Total hot run time: 182096 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c6ff442967eed53de2abfdff58d41663f8d144ea, data reload: false

query1	1212	1108	1117	1108
query2	6467	1959	1929	1929
query3	6670	205	212	205
query4	24678	21503	21446	21446
query5	4154	406	399	399
query6	283	183	184	183
query7	4601	310	296	296
query8	230	181	182	181
query9	8461	2224	2218	2218
query10	587	257	255	255
query11	14862	14542	14416	14416
query12	150	100	101	100
query13	1646	379	395	379
query14	8794	6895	6853	6853
query15	208	179	176	176
query16	6791	268	270	268
query17	1001	591	577	577
query18	1802	281	279	279
query19	201	157	156	156
query20	95	92	92	92
query21	197	131	121	121
query22	5123	4934	4962	4934
query23	33575	32440	32743	32440
query24	12658	3203	3132	3132
query25	684	395	409	395
query26	1931	161	170	161
query27	3019	328	326	326
query28	6784	1846	1818	1818
query29	1122	588	597	588
query30	314	167	172	167
query31	993	729	731	729
query32	96	61	60	60
query33	713	258	258	258
query34	1053	492	494	492
query35	815	702	710	702
query36	997	873	854	854
query37	234	82	79	79
query38	3546	3393	3381	3381
query39	1563	1518	1541	1518
query40	301	139	140	139
query41	48	47	46	46
query42	118	105	106	105
query43	427	401	400	400
query44	1090	718	705	705
query45	282	248	260	248
query46	1070	804	770	770
query47	1889	1826	1827	1826
query48	389	313	302	302
query49	1177	375	393	375
query50	794	385	383	383
query51	6725	6609	6599	6599
query52	108	95	96	95
query53	351	288	286	286
query54	336	235	243	235
query55	97	82	79	79
query56	249	229	227	227
query57	1227	1151	1106	1106
query58	246	221	225	221
query59	2481	2340	2247	2247
query60	274	237	240	237
query61	110	107	122	107
query62	686	447	452	447
query63	323	284	288	284
query64	6590	3432	3454	3432
query65	3096	3001	3028	3001
query66	1459	331	332	331
query67	15483	14985	14950	14950
query68	7669	565	579	565
query69	552	340	336	336
query70	1199	1093	1088	1088
query71	499	285	278	278
query72	6367	2597	2395	2395
query73	797	320	326	320
query74	6652	6367	6408	6367
query75	3367	2313	2295	2295
query76	4755	1108	1205	1108
query77	619	257	262	257
query78	11065	10085	10247	10085
query79	11041	533	545	533
query80	1890	420	435	420
query81	502	255	242	242
query82	569	104	107	104
query83	219	163	162	162
query84	279	87	88	87
query85	1338	324	282	282
query86	374	280	305	280
query87	3652	3475	3437	3437
query88	3951	2284	2288	2284
query89	564	380	379	379
query90	1976	181	177	177
query91	133	103	101	101
query92	67	51	51	51
query93	7195	542	543	542
query94	1296	194	194	194
query95	1096	1093	1095	1093
query96	618	275	272	272
query97	2702	2478	2473	2473
query98	240	216	206	206
query99	1298	834	829	829
Total cold run time: 300277 ms
Total hot run time: 182096 ms

doris-robot · 2024-04-03T10:17:37Z

ClickBench: Total hot run time: 29.88 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c6ff442967eed53de2abfdff58d41663f8d144ea, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.04	0.04
query4	1.68	0.07	0.07
query5	0.49	0.48	0.50
query6	1.14	0.65	0.66
query7	0.02	0.01	0.00
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.58	0.56	0.55
query11	0.15	0.11	0.11
query12	0.13	0.12	0.11
query13	0.60	0.60	0.59
query14	0.77	0.79	0.78
query15	0.87	0.84	0.84
query16	0.36	0.35	0.36
query17	1.02	1.03	0.97
query18	0.26	0.25	0.26
query19	1.82	1.74	1.75
query20	0.01	0.00	0.00
query21	15.43	0.76	0.76
query22	3.48	4.51	1.42
query23	17.40	1.25	1.18
query24	1.46	0.22	0.23
query25	0.12	0.09	0.08
query26	0.28	0.17	0.19
query27	0.09	0.09	0.07
query28	13.76	0.96	0.94
query29	12.66	3.30	3.32
query30	0.26	0.06	0.06
query31	2.88	0.39	0.38
query32	3.28	0.48	0.47
query33	2.88	2.89	2.88
query34	15.52	4.33	4.35
query35	4.34	4.36	4.39
query36	0.67	0.47	0.48
query37	0.19	0.17	0.17
query38	0.17	0.16	0.16
query39	0.05	0.05	0.03
query40	0.17	0.15	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 106.15 s
Total hot run time: 29.88 s

doris-robot · 2024-04-03T10:20:44Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c6ff442967eed53de2abfdff58d41663f8d144ea with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.2 seconds inserted 10000000 Rows, about 617K ops/s

zy-kkk · 2024-04-03T16:45:32Z

run buildall

github-actions · 2024-04-03T16:50:26Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-03T17:14:26Z

TeamCity be ut coverage result:
Function Coverage: 35.64% (8882/24922)
Line Coverage: 27.37% (72925/266411)
Region Coverage: 26.55% (37708/142019)
Branch Coverage: 23.35% (19215/82280)
Coverage Report: http://coverage.selectdb-in.cc/coverage/78bd4bf27aaeff9226849adb91135c4d9a93df12_78bd4bf27aaeff9226849adb91135c4d9a93df12/report/index.html

doris-robot · 2024-04-03T17:18:27Z

TPC-H: Total hot run time: 38364 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 78bd4bf27aaeff9226849adb91135c4d9a93df12, data reload: false

------ Round 1 ----------------------------------
q1	17641	4184	4121	4121
q2	2010	191	182	182
q3	10474	1164	1332	1164
q4	10217	808	929	808
q5	7462	2969	2896	2896
q6	218	134	131	131
q7	1079	610	581	581
q8	9394	2016	1991	1991
q9	6727	6179	6161	6161
q10	8458	3501	3484	3484
q11	426	242	243	242
q12	377	204	203	203
q13	17776	2915	2925	2915
q14	265	235	237	235
q15	511	482	477	477
q16	504	392	366	366
q17	944	889	879	879
q18	7237	6580	6365	6365
q19	1792	1530	1517	1517
q20	573	321	315	315
q21	3434	3080	3038	3038
q22	373	293	295	293
Total cold run time: 107892 ms
Total hot run time: 38364 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4154	4123	4123	4123
q2	326	219	234	219
q3	2940	2958	2933	2933
q4	1843	1826	1859	1826
q5	5192	5182	5178	5178
q6	208	124	125	124
q7	2222	1776	1783	1776
q8	3192	3263	3276	3263
q9	8443	8451	8421	8421
q10	3726	3892	3973	3892
q11	562	451	463	451
q12	751	588	596	588
q13	16922	3107	3145	3107
q14	310	278	305	278
q15	535	489	477	477
q16	492	428	421	421
q17	1756	1720	1729	1720
q18	8167	7796	7554	7554
q19	1708	1665	1671	1665
q20	2018	1846	1859	1846
q21	5072	4978	4803	4803
q22	495	438	433	433
Total cold run time: 71034 ms
Total hot run time: 55098 ms

doris-robot · 2024-04-03T17:30:12Z

TPC-DS: Total hot run time: 181732 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 78bd4bf27aaeff9226849adb91135c4d9a93df12, data reload: false

query1	1211	1129	1119	1119
query2	6303	1889	1885	1885
query3	6678	204	201	201
query4	24475	21489	21454	21454
query5	4203	396	400	396
query6	266	179	177	177
query7	4604	298	292	292
query8	224	177	170	170
query9	8448	2213	2228	2213
query10	444	240	234	234
query11	15100	14571	14468	14468
query12	143	103	86	86
query13	1645	389	379	379
query14	8440	6846	6828	6828
query15	207	181	180	180
query16	6779	260	261	260
query17	1009	572	571	571
query18	1852	300	285	285
query19	195	158	155	155
query20	95	93	92	92
query21	194	130	128	128
query22	4966	4938	4804	4804
query23	33419	32953	32672	32672
query24	11262	3165	3182	3165
query25	703	411	452	411
query26	1855	165	159	159
query27	3277	365	377	365
query28	7493	1868	1861	1861
query29	1299	589	588	588
query30	304	178	171	171
query31	975	729	744	729
query32	92	53	56	53
query33	628	234	235	234
query34	1236	519	516	516
query35	839	714	733	714
query36	997	861	891	861
query37	265	75	72	72
query38	3727	3672	3602	3602
query39	1619	1577	1596	1577
query40	240	127	126	126
query41	44	44	44	44
query42	115	99	108	99
query43	481	416	434	416
query44	1173	731	723	723
query45	293	290	286	286
query46	1125	833	778	778
query47	1965	1862	1882	1862
query48	372	296	308	296
query49	978	371	368	368
query50	829	402	415	402
query51	6906	6720	6704	6704
query52	104	96	90	90
query53	358	300	299	299
query54	286	246	230	230
query55	81	80	78	78
query56	249	242	240	240
query57	1271	1187	1220	1187
query58	246	218	226	218
query59	2760	2542	2357	2357
query60	265	240	242	240
query61	109	106	109	106
query62	666	457	452	452
query63	302	283	281	281
query64	5894	3270	3247	3247
query65	3069	3025	3060	3025
query66	1350	342	320	320
query67	16273	15041	14818	14818
query68	9276	576	565	565
query69	596	326	298	298
query70	1440	1066	1050	1050
query71	514	266	264	264
query72	6601	2590	2411	2411
query73	1596	318	325	318
query74	6767	6280	6373	6280
query75	3890	2274	2301	2274
query76	6267	1162	1230	1162
query77	647	247	237	237
query78	10884	10166	10074	10074
query79	12479	540	528	528
query80	1660	418	408	408
query81	496	244	237	237
query82	406	94	92	92
query83	208	163	164	163
query84	266	93	83	83
query85	991	285	276	276
query86	358	313	303	303
query87	3691	3499	3478	3478
query88	4391	2274	2268	2268
query89	553	369	364	364
query90	2029	182	179	179
query91	135	100	108	100
query92	60	47	47	47
query93	6688	537	531	531
query94	1251	182	182	182
query95	434	308	311	308
query96	615	275	272	272
query97	2722	2522	2503	2503
query98	260	215	209	209
query99	1217	853	849	849
Total cold run time: 304663 ms
Total hot run time: 181732 ms

doris-robot · 2024-04-03T17:35:39Z

ClickBench: Total hot run time: 30.17 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 78bd4bf27aaeff9226849adb91135c4d9a93df12, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.04
query4	1.69	0.07	0.07
query5	0.49	0.48	0.50
query6	1.13	0.65	0.66
query7	0.02	0.01	0.01
query8	0.05	0.03	0.04
query9	0.55	0.51	0.51
query10	0.57	0.58	0.56
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.77	0.79	0.79
query15	0.85	0.84	0.85
query16	0.36	0.36	0.35
query17	0.99	0.99	1.02
query18	0.25	0.24	0.28
query19	1.83	1.71	1.75
query20	0.02	0.01	0.02
query21	15.42	0.63	0.63
query22	4.34	5.53	1.82
query23	17.99	1.22	1.23
query24	1.56	0.22	0.19
query25	0.17	0.08	0.07
query26	0.27	0.16	0.16
query27	0.08	0.07	0.07
query28	13.79	0.97	0.96
query29	12.54	3.30	3.23
query30	0.26	0.06	0.06
query31	2.85	0.39	0.39
query32	3.30	0.48	0.48
query33	2.84	2.89	2.89
query34	15.50	4.39	4.35
query35	4.42	4.41	4.39
query36	0.68	0.47	0.47
query37	0.18	0.15	0.16
query38	0.16	0.14	0.14
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.58 s
Total hot run time: 30.17 s

doris-robot · 2024-04-03T17:38:47Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 78bd4bf27aaeff9226849adb91135c4d9a93df12 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.1 seconds inserted 10000000 Rows, about 621K ops/s

…L result write (#33230) pick #33227

morningman

LGTM

github-actions · 2024-04-07T03:10:58Z

PR approved by at least one committer and no changes requested.

LemonLiTree

LGTM

…te (apache#33227)

…te (#33227)

…te (apache#33227)

[improvement](serde) Handle NaN values in number for MySQL result write

c6ff442

morningman previously approved these changes Apr 3, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 3, 2024

github-actions bot added the reviewed label Apr 3, 2024

xiaokang added the dev/2.0.x label Apr 3, 2024

zy-kkk mentioned this pull request Apr 3, 2024

[branch-2.0][improvement](serde) Handle NaN values in number for MySQL result write #33230

Merged

fix test case

78bd4bf

zy-kkk dismissed morningman’s stale review via 78bd4bf April 3, 2024 16:45

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 3, 2024

morningman pushed a commit that referenced this pull request Apr 4, 2024

[branch-2.0][improvement](serde) Handle NaN values in number for MySQ…

71872cd

…L result write (#33230) pick #33227

morningman added dev/2.0.8-merged and removed dev/2.0.x labels Apr 4, 2024

morningman approved these changes Apr 7, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 7, 2024

LemonLiTree approved these changes Apr 7, 2024

View reviewed changes

AshinGau approved these changes Apr 7, 2024

View reviewed changes

zy-kkk merged commit e4e763a into apache:master Apr 7, 2024
28 of 32 checks passed

zy-kkk deleted the float_mysql_result branch April 7, 2024 03:20

morningman added the dev/2.1.2-merged label Apr 7, 2024

morningman pushed a commit to morningman/doris that referenced this pull request Apr 7, 2024

[improvement](serde) Handle NaN values in number for MySQL result wri…

a2e76e7

…te (apache#33227)

morningman pushed a commit that referenced this pull request Apr 7, 2024

[improvement](serde) Handle NaN values in number for MySQL result wri…

1b3e432

…te (#33227)

seawinde pushed a commit to seawinde/doris that referenced this pull request Apr 10, 2024

[improvement](serde) Handle NaN values in number for MySQL result wri…

5e8f6ff

…te (apache#33227)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improvement](serde) Handle NaN values in number for MySQL result write #33227

[improvement](serde) Handle NaN values in number for MySQL result write #33227

zy-kkk commented Apr 3, 2024

doris-robot commented Apr 3, 2024

zy-kkk commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

morningman left a comment

github-actions bot commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

zy-kkk commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

morningman left a comment

github-actions bot commented Apr 7, 2024

LemonLiTree left a comment

[improvement](serde) Handle NaN values in number for MySQL result write #33227

[improvement](serde) Handle NaN values in number for MySQL result write #33227

Conversation

zy-kkk commented Apr 3, 2024

Proposed changes

Further comments

doris-robot commented Apr 3, 2024

zy-kkk commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

zy-kkk commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

doris-robot commented Apr 3, 2024

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 7, 2024

LemonLiTree left a comment

Choose a reason for hiding this comment