Fix to address bug in the SQL query with timestamp field format

[manasvinibs]

Description

We have this issue in SQL query when date in timestamp format in the SQL query is erroring out due to mismatch in the timestamp format between field mapping and the request query -

Sample query:

select count(*) from ami where State = 'available' and  Public = 'true' and CallerVersion = "2016-11-15 00:00:00";

select count(*) from ami where State = 'available' and  Public = 'true' and CallerVersion < "2016-11-15 00:00:00";

Error -

[INFO] 10f0cef3-4159-4c39-84e1-f4ba8b88be97 (Bobcat-29) com.amazonaws.fleetinsightsopensearchcommons.helper.ResponseConvertor: Original SQL response : {
  "error": {
    "reason": "Error occurred in OpenSearch engine: all shards failed",
    "details": "Shard[0]: OpenSearchParseException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];\nShard[1]: OpenSearchParseException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];\nShard[2]: OpenSearchParseException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];\nShard[3]: OpenSearchParseException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];\nShard[4]: OpenSearchParseException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];\n\nFor more details, please send request for Json format to see the raw response from OpenSearch engine.",
    "type": "SearchPhaseExecutionException"
  },
  "status": 400
}

Root cause -
Currently for response JDBC format, when term/range query has timestamp field in the sql query, by default we are converting timestamps to epoch time in milliseconds when building query for Opensearch DSL which is causing the above exceptions. Instead for timestamp query string in epoch, we should send parsed formatted timestamp string as it is as part of the query similar to Date format.

Issues Resolved

#2700

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[rupal-bq]

looks like CI failed due to spotless violations.

[rupal-bq]

Can you add test with different format? We should support all formats mentioned in https://opensearch.org/docs/latest/field-types/supported-field-types/date/

[rupal-bq]

Can you also add IT to cover this case? IIRC we don't have IT with date field in where clause.

[penghuo]

what is the issue with epoch value? why we want cast to ISO 8601 format?

[manasvinibs]

My intention with this PR was to remove additional formatting to specifically epoch on literal value during query build. If we expect literal value to be the right formatted value, then this additional default formatting can cause issues.
Having said that today these literal values today are not honoring all the right date/time formats. See the formatter we are using today to format timestamp string in ExprTimeStamp.value class

sql/core/src/main/java/org/opensearch/sql/data/model/ExprTimestampValue.java

Line 34 in 8eae36f

LocalDateTime.parse(timestamp, DATE_TIME_FORMATTER_VARIABLE_NANOS)

Its always yyyy-MM-dd HH:mm:ss for now

sql/core/src/main/java/org/opensearch/sql/utils/DateTimeFormatters.java

Line 125 in 8eae36f

.appendPattern("uuuu-MM-dd HH:mm:ss")

So in this case literal value is ISO 8601. But if we fix the format while parsing the date string in ExprTimeStampValue/ExprDateValue before using it in query builders, we do not have to format to epoch or any other default values.

[manasvinibs]

looks like CI failed due to spotless violations.

Thanks for catching. Fixed and updated.

[manasvinibs]

Closing this in context of this PR #2762 as this PR addresses the larger issue with custom date formats and opensearch date formats support for lucene queries.