Skip to content

[branch-2.0](function) fix date_format and from_unixtime core when meet long format string (#35883) #37178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
HappenLee merged 1 commit into from
Jul 3, 2024
Merged

[branch-2.0](function) fix date_format and from_unixtime core when meet long format string (#35883) #37178

HappenLee merged 1 commit into from
Jul 3, 2024

Conversation

@zclllyybb
Copy link
Contributor

@zclllyybb zclllyybb commented Jul 2, 2024

Proposed changes

pick #35883

before:
crash

now:

mysql> SELECT  DATE_FORMAT('2011-06-24', '%f %V %f %l %V %I %S %p %w %r %j %f %l %I %D %w %j %D %e %s %V %f %D %M %s %X %U %v %c %u %x %r %j %a %h %s %m %a %v %u %b');
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| date_format(cast('2011-06-24' as DATETIMEV2(0)), '%f %V %f %l %V %I %S %p %w %r %j %f %l %I %D %w %j %D %e %s %V %f %D %M %s %X %U %v %c %u %x %r %j %a %h %s %m %a %v %u %b') |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| NULL                                                                                                                                                                           |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.80 sec)

@doris-robot
Copy link

doris-robot commented Jul 2, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb
Copy link
Contributor Author

zclllyybb commented Jul 2, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@zclllyybb
[fix](function) fix date_format and from_unixtime core when meet long…
04b7fc5 
… format string (apache#35883)

before:
crash

now:
```sql
mysql> SELECT  DATE_FORMAT('2011-06-24', '%f %V %f %l %V %I %S %p %w %r %j %f %l %I %D %w %j %D %e %s %V %f %D %M %s %X %U %v %c %u %x %r %j %a %h %s %m %a %v %u %b');
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| date_format(cast('2011-06-24' as DATETIMEV2(0)), '%f %V %f %l %V %I %S %p %w %r %j %f %l %I %D %w %j %D %e %s %V %f %D %M %s %X %U %v %c %u %x %r %j %a %h %s %m %a %v %u %b') |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| NULL                                                                                                                                                                           |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.80 sec)
```
@zclllyybb zclllyybb force-pushed the pick_date_format_20 branch from b54ad9d to 04b7fc5 Compare July 2, 2024 14:18
@zclllyybb
Copy link
Contributor Author

zclllyybb commented Jul 2, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 2, 2024

TeamCity be ut coverage result:
Function Coverage: 37.87% (8111/21417)
Line Coverage: 29.54% (66437/224878)
Region Coverage: 29.01% (34232/118018)
Branch Coverage: 24.88% (17581/70662)
Coverage Report: http://coverage.selectdb-in.cc/coverage/04b7fc586d0c913f15a9b04eeaf9cbc6eb025176_04b7fc586d0c913f15a9b04eeaf9cbc6eb025176/report/index.html

@doris-robot
Copy link

doris-robot commented Jul 2, 2024

TPC-H: Total hot run time: 50468 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 04b7fc586d0c913f15a9b04eeaf9cbc6eb025176, data reload: false

------ Round 1 ----------------------------------
q1	17698	4422	4389	4389
q2	2070	156	149	149
q3	10465	1917	1921	1917
q4	10246	1295	1320	1295
q5	8642	4215	3959	3959
q6	232	147	125	125
q7	2030	1591	1600	1591
q8	9320	2759	2733	2733
q9	11035	10580	10639	10580
q10	8622	3508	3539	3508
q11	422	243	257	243
q12	472	302	295	295
q13	18314	3993	4025	3993
q14	356	325	330	325
q15	512	475	466	466
q16	691	571	572	571
q17	1151	988	965	965
q18	7312	6944	7018	6944
q19	1827	1664	1574	1574
q20	553	316	306	306
q21	4492	4204	4099	4099
q22	521	449	441	441
Total cold run time: 116983 ms
Total hot run time: 50468 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4358	4304	4324	4304
q2	326	225	236	225
q3	4211	4178	4168	4168
q4	2779	2748	2783	2748
q5	7210	7070	7063	7063
q6	241	119	121	119
q7	3250	2830	2852	2830
q8	4341	4475	4510	4475
q9	17386	17012	17173	17012
q10	4248	4267	4257	4257
q11	730	675	667	667
q12	1017	856	842	842
q13	6820	3736	3760	3736
q14	439	434	427	427
q15	505	450	458	450
q16	748	679	695	679
q17	3863	3697	3894	3697
q18	9000	8781	8730	8730
q19	1731	1747	1659	1659
q20	2370	2142	2132	2132
q21	8568	8478	8475	8475
q22	1085	965	962	962
Total cold run time: 85226 ms
Total hot run time: 79657 ms

@doris-robot
Copy link

doris-robot commented Jul 2, 2024

TPC-DS: Total hot run time: 205120 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 04b7fc586d0c913f15a9b04eeaf9cbc6eb025176, data reload: false

query1	961	406	381	381
query2	6526	2808	2794	2794
query3	6924	207	200	200
query4	20289	18139	18256	18139
query5	19732	6593	6602	6593
query6	296	220	220	220
query7	4154	312	327	312
query8	418	405	440	405
query9	3161	2692	2629	2629
query10	427	308	308	308
query11	11380	10783	10648	10648
query12	128	74	75	74
query13	5601	694	706	694
query14	18254	13220	13404	13220
query15	359	242	256	242
query16	6470	296	269	269
query17	1717	1463	888	888
query18	2296	417	421	417
query19	209	161	158	158
query20	84	76	81	76
query21	193	95	97	95
query22	5244	5015	5141	5015
query23	32715	32095	32063	32063
query24	6937	6555	6560	6555
query25	514	448	434	434
query26	538	169	168	168
query27	1860	301	298	298
query28	6102	2370	2353	2353
query29	2876	2715	2750	2715
query30	244	169	166	166
query31	923	726	754	726
query32	73	64	58	58
query33	401	258	247	247
query34	848	488	482	482
query35	1157	909	971	909
query36	1310	1444	1306	1306
query37	92	61	62	61
query38	3058	2949	2955	2949
query39	1392	1329	1345	1329
query40	202	99	97	97
query41	47	44	44	44
query42	88	96	84	84
query43	788	691	703	691
query44	1166	725	727	725
query45	250	243	238	238
query46	1221	983	1018	983
query47	1737	1996	1773	1773
query48	1038	724	726	724
query49	628	385	372	372
query50	885	608	600	600
query51	4733	4699	4673	4673
query52	97	83	84	83
query53	451	330	332	330
query54	2681	2479	2471	2471
query55	98	84	81	81
query56	231	225	204	204
query57	1229	1070	1071	1070
query58	223	187	199	187
query59	4263	4024	4119	4024
query60	217	205	216	205
query61	101	99	97	97
query62	779	441	495	441
query63	494	345	351	345
query64	2598	1527	1438	1438
query65	3654	3598	3557	3557
query66	752	374	379	374
query67	16430	15636	15366	15366
query68	8922	654	666	654
query69	571	364	359	359
query70	1507	1327	1347	1327
query71	410	319	312	312
query72	6563	3517	3538	3517
query73	741	323	319	319
query74	6304	5920	5832	5832
query75	5299	3711	3788	3711
query76	5382	1153	1160	1153
query77	907	257	265	257
query78	12887	11823	11919	11823
query79	7456	671	645	645
query80	1005	414	409	409
query81	498	234	234	234
query82	1522	103	104	103
query83	171	134	145	134
query84	263	72	72	72
query85	878	326	329	326
query86	324	341	326	326
query87	3286	3021	3032	3021
query88	5066	2321	2312	2312
query89	359	304	271	271
query90	1939	218	210	210
query91	187	139	138	138
query92	59	53	50	50
query93	4658	582	591	582
query94	712	222	216	216
query95	1119	1072	1076	1072
query96	648	326	320	320
query97	6642	6409	6493	6409
query98	190	169	169	169
query99	2862	971	964	964
Total cold run time: 312281 ms
Total hot run time: 205120 ms

@doris-robot
Copy link

doris-robot commented Jul 2, 2024

ClickBench: Total hot run time: 31.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04b7fc586d0c913f15a9b04eeaf9cbc6eb025176, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.04	0.04
query4	1.79	0.06	0.07
query5	0.53	0.52	0.52
query6	1.24	0.63	0.62
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.50	0.47
query10	0.54	0.54	0.52
query11	0.12	0.09	0.09
query12	0.12	0.09	0.10
query13	0.63	0.61	0.61
query14	0.80	0.77	0.80
query15	0.78	0.78	0.76
query16	0.37	0.37	0.37
query17	1.02	1.00	1.02
query18	0.20	0.27	0.23
query19	1.92	1.88	1.87
query20	0.01	0.00	0.00
query21	15.45	0.57	0.56
query22	2.11	2.45	1.73
query23	17.45	0.95	1.09
query24	6.49	1.19	1.16
query25	0.38	0.09	0.05
query26	0.78	0.17	0.16
query27	0.04	0.04	0.04
query28	6.31	0.76	0.77
query29	12.66	2.37	2.24
query30	0.64	0.59	0.56
query31	2.82	0.39	0.38
query32	3.37	0.50	0.50
query33	3.06	3.08	3.10
query34	15.24	4.84	4.81
query35	4.89	4.87	4.84
query36	1.07	1.01	1.02
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.08	0.01	0.02
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.14 s
Total hot run time: 31.17 s

@doris-robot
Copy link

doris-robot commented Jul 2, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 04b7fc586d0c913f15a9b04eeaf9cbc6eb025176 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@HappenLee HappenLee merged commit e2caee6 into apache:branch-2.0 Jul 3, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zclllyybb
[branch-2.0](function) fix date_format and from_unixtime core when me…
26de735 
…et long format string (apache#35883) (apache#37178)

pick apache#35883
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zclllyybb @doris-robot @HappenLee