Skip to content

[fix](nereids) fix parse date time exception #50810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
May 14, 2025
Merged

[fix](nereids) fix parse date time exception #50810

merged 4 commits into from
May 14, 2025

Conversation

yujun777
Copy link
Contributor

@yujun777 yujun777 commented May 12, 2025 •
edited
Loading

What problem does this PR solve?

before this pr:

MySQL root@127.0.0.1:(none)> select date '2025年1月20日';
(1105, 'errCode = 2, detailMessage = date/datetime literal [2025年1月20日] is invalid')

after this pr:

MySQL root@127.0.0.1:(none)> select date '2025年1月20日';
+----------------------+
| date '2025年1月20日' |
+----------------------+
| 2025-01-20           |
+----------------------+

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yujun777
Copy link
Contributor Author

run buidall

morrySnow
morrySnow previously approved these changes May 12, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 12, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@yujun777
Copy link
Contributor Author

run buildall

@yujun777
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 13, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 33812 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f9407b5aac8024ad8c3701992109fe2289f6bf9, data reload: false

------ Round 1 ----------------------------------
q1	26319	5178	4993	4993
q2	2076	291	194	194
q3	10380	1286	688	688
q4	10223	1007	511	511
q5	7542	2398	2324	2324
q6	181	166	130	130
q7	944	731	616	616
q8	9324	1304	1011	1011
q9	6767	5098	5070	5070
q10	6866	2340	1890	1890
q11	497	281	286	281
q12	355	352	210	210
q13	17783	3702	3139	3139
q14	228	222	207	207
q15	532	499	482	482
q16	420	431	372	372
q17	596	861	348	348
q18	7652	7159	7221	7159
q19	1475	950	561	561
q20	318	329	219	219
q21	3857	3369	2442	2442
q22	1036	1008	965	965
Total cold run time: 115371 ms
Total hot run time: 33812 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5128	5120	5045	5045
q2	246	327	241	241
q3	2194	2693	2310	2310
q4	1335	1749	1413	1413
q5	4487	4413	4438	4413
q6	220	167	124	124
q7	1997	1964	1843	1843
q8	2609	2576	2560	2560
q9	7270	7262	6959	6959
q10	3050	3195	2758	2758
q11	569	514	502	502
q12	674	789	642	642
q13	3522	3841	3229	3229
q14	283	290	291	290
q15	527	492	478	478
q16	452	482	445	445
q17	1137	1544	1392	1392
q18	7853	7638	7591	7591
q19	821	865	1091	865
q20	2006	2111	1858	1858
q21	4967	4790	4596	4596
q22	1079	1063	1034	1034
Total cold run time: 52426 ms
Total hot run time: 50588 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194200 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f9407b5aac8024ad8c3701992109fe2289f6bf9, data reload: false

query1	1398	1079	1067	1067
query2	6397	1846	1813	1813
query3	11013	4621	4331	4331
query4	52663	26131	23513	23513
query5	5011	506	454	454
query6	355	220	208	208
query7	4900	504	300	300
query8	320	262	235	235
query9	5345	2683	2702	2683
query10	448	322	265	265
query11	15145	15152	14778	14778
query12	153	108	99	99
query13	1024	538	402	402
query14	10212	6285	6427	6285
query15	209	196	195	195
query16	7142	672	484	484
query17	1079	748	592	592
query18	1682	408	338	338
query19	199	193	168	168
query20	136	137	131	131
query21	211	132	111	111
query22	4384	4412	4356	4356
query23	34365	33819	33602	33602
query24	6581	2407	2448	2407
query25	467	480	414	414
query26	701	283	152	152
query27	2193	510	345	345
query28	3110	2208	2181	2181
query29	611	585	473	473
query30	269	218	196	196
query31	885	891	778	778
query32	77	65	62	62
query33	464	364	320	320
query34	778	864	542	542
query35	797	837	773	773
query36	956	1020	901	901
query37	115	103	76	76
query38	4339	4407	4247	4247
query39	1532	1468	1446	1446
query40	201	125	106	106
query41	54	59	52	52
query42	125	117	142	117
query43	491	518	502	502
query44	1342	848	837	837
query45	179	177	169	169
query46	851	1055	661	661
query47	1893	1871	1804	1804
query48	394	437	347	347
query49	683	500	440	440
query50	718	710	418	418
query51	4345	4308	4240	4240
query52	105	112	107	107
query53	231	253	193	193
query54	590	609	546	546
query55	87	88	79	79
query56	321	301	292	292
query57	1173	1239	1116	1116
query58	279	283	259	259
query59	2718	2746	2687	2687
query60	357	341	329	329
query61	158	161	147	147
query62	749	763	693	693
query63	238	209	198	198
query64	1860	1197	717	717
query65	4396	4219	4233	4219
query66	726	412	308	308
query67	16154	15827	15563	15563
query68	7585	901	526	526
query69	538	313	273	273
query70	1208	1134	1114	1114
query71	499	330	298	298
query72	5896	4928	4942	4928
query73	1520	706	362	362
query74	9266	9113	8957	8957
query75	3805	3235	2692	2692
query76	4170	1196	777	777
query77	616	384	288	288
query78	10126	10160	9381	9381
query79	2292	825	570	570
query80	636	516	453	453
query81	503	251	227	227
query82	427	209	98	98
query83	378	250	239	239
query84	295	110	85	85
query85	796	361	315	315
query86	398	302	294	294
query87	4466	4514	4439	4439
query88	3649	2297	2311	2297
query89	404	322	284	284
query90	1895	214	217	214
query91	148	147	114	114
query92	73	58	61	58
query93	1703	948	588	588
query94	653	368	300	300
query95	371	298	287	287
query96	490	573	280	280
query97	3181	3246	3107	3107
query98	235	222	203	203
query99	1456	1409	1283	1283
Total cold run time: 298762 ms
Total hot run time: 194200 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f9407b5aac8024ad8c3701992109fe2289f6bf9, data reload: false

query1	0.03	0.03	0.03
query2	0.12	0.10	0.11
query3	0.24	0.19	0.20
query4	1.60	0.19	0.12
query5	0.56	0.57	0.56
query6	1.17	0.71	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.54	0.51
query10	0.57	0.58	0.56
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.59
query14	0.79	0.80	0.82
query15	0.87	0.85	0.87
query16	0.38	0.39	0.39
query17	1.09	1.07	1.08
query18	0.22	0.21	0.21
query19	2.02	1.81	1.79
query20	0.01	0.01	0.01
query21	15.41	0.88	0.55
query22	0.76	1.12	0.88
query23	14.75	1.39	0.60
query24	7.38	1.68	0.34
query25	0.28	0.20	0.09
query26	0.59	0.16	0.14
query27	0.05	0.06	0.05
query28	8.97	0.89	0.43
query29	12.56	4.00	3.30
query30	0.25	0.10	0.06
query31	2.83	0.59	0.38
query32	3.23	0.56	0.47
query33	2.99	3.04	3.06
query34	15.58	5.04	4.48
query35	4.48	4.50	4.52
query36	0.65	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.02
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 102.48 s
Total hot run time: 28.89 s

@yujun777 yujun777 closed this May 14, 2025
@yujun777 yujun777 reopened this May 14, 2025
englefly
englefly reviewed May 14, 2025
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java Outdated
try {
return Config.enable_date_conversion ? new DateV2Literal(value) : new DateLiteral(value);
} catch (Exception e) {
return new Cast(Literal.of(value),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Using "new StringLiteral(value)" is more explicit than "Literal.of(value)".
  2. why not handle this pattern on FE side? if we do it on fe side, it is good for const-fold and some other expression rewrite rules

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. fix;
  2. be can handle many pattern of date time, including fault tolerance cases, and fe can handle only some of them, if fe cannot handle, it will parse to be

englefly
englefly previously approved these changes May 14, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 14, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@yujun777
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 14, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 35046 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 12237a06f6be695e60325407b32896936f277ec7, data reload: false

------ Round 1 ----------------------------------
q1	25971	5052	5043	5043
q2	2442	280	189	189
q3	10529	1260	715	715
q4	10430	1035	544	544
q5	7938	2389	2388	2388
q6	188	173	142	142
q7	939	776	620	620
q8	9324	1350	1133	1133
q9	7287	5154	5169	5154
q10	6904	2380	1906	1906
q11	490	301	275	275
q12	350	361	226	226
q13	18219	3742	3386	3386
q14	251	253	217	217
q15	559	501	494	494
q16	435	442	405	405
q17	618	868	367	367
q18	10180	7909	7448	7448
q19	4031	1210	624	624
q20	369	318	215	215
q21	4474	3556	2586	2586
q22	1059	1003	969	969
Total cold run time: 122987 ms
Total hot run time: 35046 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5242	5083	5070	5070
q2	233	324	228	228
q3	2210	2795	2451	2451
q4	1409	1916	1440	1440
q5	4461	4438	4347	4347
q6	217	173	128	128
q7	1986	1882	1810	1810
q8	2650	2542	2514	2514
q9	7064	7128	7039	7039
q10	2997	3180	2761	2761
q11	608	515	511	511
q12	673	774	631	631
q13	3566	3910	3375	3375
q14	299	318	284	284
q15	554	477	477	477
q16	436	487	431	431
q17	1175	1666	1347	1347
q18	7730	7665	7186	7186
q19	835	906	1086	906
q20	2006	2058	1931	1931
q21	5204	4831	4771	4771
q22	1100	1053	1043	1043
Total cold run time: 52655 ms
Total hot run time: 50681 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193855 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 12237a06f6be695e60325407b32896936f277ec7, data reload: false

query1	1408	1108	1055	1055
query2	6188	1889	1895	1889
query3	11027	4649	4357	4357
query4	57927	24566	23131	23131
query5	4731	489	452	452
query6	355	216	191	191
query7	4917	512	282	282
query8	327	257	263	257
query9	6359	2684	2674	2674
query10	434	339	277	277
query11	15005	15037	14858	14858
query12	168	110	114	110
query13	1069	534	423	423
query14	10140	6499	6352	6352
query15	208	207	182	182
query16	7055	670	462	462
query17	1083	771	596	596
query18	1559	428	342	342
query19	208	201	188	188
query20	128	120	123	120
query21	207	128	115	115
query22	4320	4487	4568	4487
query23	34006	33646	33679	33646
query24	6941	2471	2451	2451
query25	452	464	415	415
query26	687	269	152	152
query27	2312	520	376	376
query28	3115	2173	2159	2159
query29	575	579	457	457
query30	275	225	194	194
query31	861	899	796	796
query32	73	62	64	62
query33	476	367	349	349
query34	774	874	536	536
query35	820	839	745	745
query36	918	1003	926	926
query37	114	103	76	76
query38	4332	4276	4242	4242
query39	1540	1461	1452	1452
query40	220	122	109	109
query41	55	55	50	50
query42	126	107	117	107
query43	517	549	504	504
query44	1353	841	841	841
query45	187	182	177	177
query46	869	1049	662	662
query47	1818	1906	1808	1808
query48	397	443	335	335
query49	704	534	449	449
query50	668	710	418	418
query51	4256	4344	4224	4224
query52	112	115	103	103
query53	236	272	199	199
query54	606	610	543	543
query55	96	91	81	81
query56	297	290	308	290
query57	1140	1153	1088	1088
query58	260	271	262	262
query59	2679	2708	2636	2636
query60	335	327	309	309
query61	142	126	140	126
query62	727	711	668	668
query63	225	186	188	186
query64	1466	1026	665	665
query65	4329	4242	4253	4242
query66	701	412	305	305
query67	16232	15548	15565	15548
query68	6978	848	545	545
query69	562	314	273	273
query70	1244	1121	1127	1121
query71	537	324	290	290
query72	5762	4811	4932	4811
query73	1323	663	357	357
query74	9055	8997	8918	8918
query75	3882	3252	2801	2801
query76	4455	1202	766	766
query77	609	393	298	298
query78	10095	10046	9298	9298
query79	3303	814	587	587
query80	622	542	465	465
query81	485	250	219	219
query82	244	130	97	97
query83	351	254	230	230
query84	291	106	95	95
query85	792	356	324	324
query86	331	313	296	296
query87	4470	4527	4277	4277
query88	2946	2307	2308	2307
query89	428	316	282	282
query90	2052	211	215	211
query91	144	140	111	111
query92	69	61	56	56
query93	2105	935	581	581
query94	711	422	285	285
query95	374	290	283	283
query96	496	576	287	287
query97	3137	3226	3120	3120
query98	240	201	211	201
query99	1392	1416	1294	1294
Total cold run time: 303332 ms
Total hot run time: 193855 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 12237a06f6be695e60325407b32896936f277ec7, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.12
query3	0.24	0.20	0.19
query4	1.60	0.18	0.18
query5	0.60	0.60	0.59
query6	1.19	0.72	0.72
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.51	0.51
query10	0.57	0.58	0.57
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.61
query14	0.79	0.81	0.80
query15	0.87	0.87	0.83
query16	0.38	0.40	0.38
query17	1.05	1.03	1.03
query18	0.22	0.21	0.21
query19	1.95	1.83	1.80
query20	0.01	0.01	0.02
query21	15.39	0.93	0.56
query22	0.76	1.16	0.60
query23	15.02	1.37	0.66
query24	7.04	1.93	0.48
query25	0.51	0.24	0.14
query26	0.60	0.16	0.13
query27	0.06	0.05	0.05
query28	9.38	0.91	0.44
query29	12.55	3.97	3.27
query30	0.26	0.11	0.07
query31	2.84	0.59	0.38
query32	3.22	0.54	0.48
query33	3.05	3.01	3.03
query34	15.75	5.09	4.48
query35	4.51	4.51	4.50
query36	0.67	0.49	0.48
query37	0.09	0.07	0.07
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.05	0.04	0.03
Total cold run time: 103.28 s
Total hot run time: 28.88 s

@englefly englefly added the p0_r label May 14, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 14, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@englefly englefly merged commit d71ca91 into apache:master May 14, 2025
30 checks passed
github-actions bot pushed a commit that referenced this pull request May 14, 2025
@yujun777
[fix](nereids) fix parse date time exception (#50810)
3c6ebc2 
### What problem does this PR solve?

before this pr:

```
MySQL root@127.0.0.1:(none)> select date '2025年1月20日';
(1105, 'errCode = 2, detailMessage = date/datetime literal [2025年1月20日] is invalid')
```

after this pr:

```
MySQL root@127.0.0.1:(none)> select date '2025年1月20日';
+----------------------+
| date '2025年1月20日' |
+----------------------+
| 2025-01-20           |
+----------------------+
```
@github-actions github-actions bot mentioned this pull request May 14, 2025
Merged
@yujun777 yujun777 mentioned this pull request May 14, 2025
Merged
dataroaring pushed a commit that referenced this pull request May 14, 2025
@github-actions @yujun777
branch-3.0: [fix](nereids) fix parse date time exception #50810 (#50889)
8dbde04 
Cherry-picked from #50810

Co-authored-by: yujun <yujun@selectdb.com>
yiguolei pushed a commit that referenced this pull request May 14, 2025
@yujun777
branch-2.1: [fix](nereids) fix parse date time exception #50810 (#50900)
01f70de 
cherry pick from #50810
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@yujun777 @koarz
[fix](nereids) fix parse date time exception (apache#50810)
14e15d6 
### What problem does this PR solve?

before this pr:

```
MySQL root@127.0.0.1:(none)> select date '2025年1月20日';
(1105, 'errCode = 2, detailMessage = date/datetime literal [2025年1月20日] is invalid')
```

after this pr:

```
MySQL root@127.0.0.1:(none)> select date '2025年1月20日';
+----------------------+
| date '2025年1月20日' |
+----------------------+
| 2025-01-20           |
+----------------------+
```
@gavinchou gavinchou mentioned this pull request Jun 11, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@englefly englefly englefly approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.6-merged p0_r reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@yujun777 @hello-stephen @doris-robot @englefly @morrySnow @yiguolei @dataroaring