Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-43485][SQL] Fix the error message for the unit argument of the datetime add/diff functions #41143

Closed
wants to merge 3 commits into from

Conversation

MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented May 11, 2023

What changes were proposed in this pull request?

In the PR, I propose to extend the grammar rule of the DATEADD/TIMESTAMPADD and DATEDIFF/TIMESTAMPDIFF, and catch wrong type of the first argument unit when an user pass a string instead of an identifier like YEAR, ..., MICROSECOND. In that case, Spark raised an error of new error class INVALID_PARAMETER_VALUE.DATETIME_UNIT.

Why are the changes needed?

To make the error message clear for the case when a literal string instead of an identifier is passed to the datetime ADD/DIFF functions:

spark-sql (default)> select dateadd('MONTH', 1, date'2023-05-11');
[WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `dateadd` requires 2 parameters but the actual number is 3. Please, refer to 'https://spark.apache.org/docs/latest/sql-ref-functions.html' for a fix.; line 1 pos 7

Does this PR introduce any user-facing change?

Yes, it changes the error class.

After the changes:

spark-sql (default)> select dateadd('MONTH', 1, date'2023-05-11');

[INVALID_PARAMETER_VALUE.DATETIME_UNIT] The value of parameter(s) `unit` in `dateadd` is invalid: expects one of the units without quotes YEAR, QUARTER, MONTH, WEEK, DAY, DAYOFYEAR, HOUR, MINUTE, SECOND, MILLISECOND, MICROSECOND, but got the string literal 'MONTH'.(line 1, pos 7)

== SQL ==
select dateadd('MONTH', 1, date'2023-05-11')
-------^^^

How was this patch tested?

By running the existing test suites:

$ PYSPARK_PYTHON=python3 build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite"

@MaxGekk MaxGekk changed the title [WIP][SQL] Improve the error message for the DATEADD/DIFF functions [WIP][SPARK-43485][SQL] Improve the error message for the DATEADD/DIFF functions May 12, 2023
@MaxGekk MaxGekk changed the title [WIP][SPARK-43485][SQL] Improve the error message for the DATEADD/DIFF functions [SPARK-43485][SQL] Fix the error message for the unit argument of the datetime add/diff functions May 12, 2023
@MaxGekk MaxGekk marked this pull request as ready for review May 12, 2023 06:50
@MaxGekk
Copy link
Member Author

MaxGekk commented May 15, 2023

I do believe the failed test suite HealthTrackerIntegrationSuite is not related to my changes. Though, I have checked it locally.

Merging to master. Thank you, @HyukjinKwon and @cloud-fan for review.

@MaxGekk MaxGekk closed this in e06275c May 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants