Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding CONVERT_TZ and DATETIME functions to SQL and PPL #848

Merged

Conversation

MitchellGale
Copy link
Contributor

Description

PR adds support for convert_tz and Datetime for PPL and SQL languages.

convert_tz

Syntax for convert_tz:
convert_tz(datetime string, from timezone, to timezone)
It converts the datetime string from the from timezone to the to timezone. Converts from negative to positive timezone, from positive to negative, from positive to positive or negative to negative time zones. Works across year roll-over and leap years.
Example:

opensearchsql> SELECT convert_tz('2021-05-12 00:00:00','-12:00','+13:00'); 
fetched rows / total rows = 1/1
+-------------------------------------------------------+
| convert_tz('2021-05-12 00:00:00','-12:00','+13:00')   |
|-------------------------------------------------------|
| 2021-05-13 01:00:00                                   |
+-------------------------------------------------------+
opensearchsql> SELECT convert_tz('2021-05-12 12:34:56','-00:00','+01:00');
fetched rows / total rows = 1/1
+-------------------------------------------------------+
| convert_tz('2021-05-12 12:34:56','-00:00','+01:00')   |
|-------------------------------------------------------|
| 2021-05-12 13:34:56                                   |
+-------------------------------------------------------+

nulls

It returns null for time zones outside of the allowed range (allow range is -13:59 -> +14:00)

opensearchsql> SELECT convert_tz('2021-05-12 12:34:56','-00:00','+15:00');
fetched rows / total rows = 1/1
+-------------------------------------------------------+
| convert_tz('2021-05-12 12:34:56','-00:00','+14:01')   |
|-------------------------------------------------------|
| null                                                  |
+-------------------------------------------------------+
opensearchsql> SELECT convert_tz('2021-05-12 12:34:56','-13:00','+10:00');
fetched rows / total rows = 1/1
+-------------------------------------------------------+
| convert_tz('2021-05-12 12:34:56','-14:00','+10:00')   |
|-------------------------------------------------------|
| null                                                  |
+-------------------------------------------------------+

Leap year

opensearchsql> SELECT convert_tz('2005-02-28 20:00:00','-10:00','+10:00');
fetched rows / total rows = 1/1
+-------------------------------------------------------+
| convert_tz('2005-02-28 20:00:00','-10:00','+10:00')   |
|-------------------------------------------------------|
| 2005-03-01 16:00:00                                   |
+-------------------------------------------------------+
opensearchsql> SELECT convert_tz('2004-02-28 20:00:00','-10:00','+10:00');
fetched rows / total rows = 1/1
+-------------------------------------------------------+
| convert_tz('2004-02-28 20:00:00','-10:00','+10:00')   |
|-------------------------------------------------------|
| 2004-02-29 16:00:00                                   |
+-------------------------------------------------------+

DateTime

DATETIME(timestamp_expression [, time_zone])

fetched rows / total rows = 1/1
+---------------------------------------------------+
| DATETIME('2008-12-25 05:30:00-05:00', '+05:00')   |
|---------------------------------------------------|
| 2008-12-25 15:30:00                               |
+---------------------------------------------------+
opensearchsql> SELECT DATETIME('2008-12-25 05:30:00', '+05:00');
fetched rows / total rows = 1/1
+---------------------------------------------+
| DATETIME('2008-12-25 05:30:00', '+05:00')   |
|---------------------------------------------|
| 2008-12-25 10:30:00                         |
+---------------------------------------------+
opensearchsql> SELECT DATETIME('2008-12-25 05:30:00+10:00'); 
fetched rows / total rows = 1/1
+-----------------------------------------+
| DATETIME('2008-12-25 05:30:00+10:00')   |
|-----------------------------------------|
| 2008-12-25 05:30:00                     |
+-----------------------------------------+
opensearchsql> SELECT DATETIME('2008-12-25 05:30:00');
fetched rows / total rows = 1/1
+-----------------------------------+
| DATETIME('2008-12-25 05:30:00')   |
|-----------------------------------|
| 2008-12-25 05:30:00               |
+-----------------------------------+

Issues Resolved

#703
#46

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@MitchellGale MitchellGale requested a review from a team as a code owner September 19, 2022 23:19
@Yury-Fridlyand Yury-Fridlyand changed the title Adding convert_tz and datetime functions to SQL and PPL Adding CONVERT_TZ and DATETIME functions to SQL and PPL Sep 20, 2022
@dai-chen
Copy link
Collaborator

Could you point me to the documentation for DATETIME function? For some reason I didn't find it in MySQL...

@dai-chen dai-chen added enhancement New feature or request SQL PPL Piped processing language labels Sep 21, 2022
@MitchellGale
Copy link
Contributor Author

@dai-chen DATETIME doesn't exist in MySQL, but it's implemented like a constructor, similar to the TIMESTATMP but for datetime.
Also listed here #722.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
…esolver convert_tz and ExprValue exprConvert_TZ. It implements the functionality for converting between time zones.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
…s that are outside the existing range (consistent with MySQL standard). Added implementation for convert_tz consistent with MySQL implementation.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
…nctionTest.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
…unction call conevrt_tz function.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Copy link
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the changes!

@dai-chen dai-chen merged commit 3ca6450 into opensearch-project:2.x Sep 28, 2022
MitchellGale added a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Oct 3, 2022
…ch-project#848)

* Added `CONVERT_TZ` to the PPL lexer/parser and the SQL lexer/parser.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added `convert_tz` to the BuiltinFunctionName.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added convert_tz to DateTimeFunction.java register, private FunctionResolver convert_tz and ExprValue exprConvert_TZ. It implements the functionality for converting between time zones.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added IT PPL and SQL tests for various conditions including time zones that are outside the existing range (consistent with MySQL standard). Added implementation for convert_tz consistent with MySQL implementation.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added DATETIME to OpenSearchPPLParser.g4 and OpenSearchSQLParser.g4.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Rebase merge conflict resolution.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added ppl doctest for convert_tz

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Completed implementation for datetime and convert_Tz.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed SQL test from PPL IT test

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed redundant convert to string.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed doctests and reverted changes to adddate function in DateTimeFunctionTest.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed doctest.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed doctest.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added additional integration tests for PPL and SQL tests for convert_tz.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added null for timezones outside of basic range for DATETIME.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added test cases for null with the datetime function. Made DateTime function call conevrt_tz function.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Seperated out test from DateTimeFunction.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Updated tests. Fixed exception to be less general.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed changes.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed rel timezone issue.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added support for non valid datetime to return null for convert_tz.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Made exception more verbose.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed unneeded format changes in DateTimeFunction.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added more doctests.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed formatting changes.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Reverting sql/ppl DateTimeFunctionsIT.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Reverted changes to DateTimeFunctionTest.java

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added more information about invalid date for convert_tz

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Converted "from Field" and "To Field" to use "Fieldn" where n is the field number.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added date validation. Added test cases in IT to cover cases. Added test in ConvertTZTest.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed date validation function. Broke up some unit tests.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed formatting.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added DateTime tests, broke up functions.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added space in DateTimeTest.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Tidied up code and tests.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed local date time rst test.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed nested try/catch exceptions.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* removed exprConvertTZ function call from within try catch statement.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Reverted change from parse localdate

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed extra casting around fromTz variable.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Updated wording for functions.rst

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Updated wording for datetime.rst to describe null for conert_tz

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added more test cases for functions.rst

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed doctests.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed extra import.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Renamed isValidTimeZone function to isValidMySqlTimeZoneId.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Has ExprDatetimeValue doing work for exprConvertTZ call.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Deleted fromTZ

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Moved fixed variables to top of class.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added Null to support of exprDateTime.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added missing expr functions for makedate/time

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* cleaning up after rebase merge.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Rebase merge conflict resolution.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed missing DATETIME in SQL Parser.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed IT test.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Addressed PR comments

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added missing variables after rebase

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed checkstyle errors after rebase.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Moved formatter for date time over.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Changed function resolved to default

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed unneeded code

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request PPL Piped processing language SQL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants