Date values are not correctly parsed in queries

[weberhofer]

Elasticsearch version
Version: 7.4.0, Build: default/rpm/22e1767283e61a198cb4db791ea66e3f11ab9910/2019-09-27T08:36:48.569419Z, JVM: 1.8.0_222

Java version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (IcedTea 3.13.0) (build 1.8.0_222-b10 suse-lp151.2.3.1-x86_64)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

OS version
Linux herkules 4.12.14-lp151.28.13-default

Description of the problem including expected versus actual behavior:
I have defined the following field:

"alertDate": {
  "type": "date",
  "format": "epoch_second||basic_date_time_no_millis||yyyy-MM-dd HH:mm:ss||strict_date||strict_date_hour_minute_second",
  "store": false
}

I can search in this field by epoch-values but by none of the other date formats like:

{
  "from":0,
  "size":10,
  "query":
    {"bool":
      {"must": [
        { "range": {"alertDate": {"from":"2019-09-01"}}}
      ]}
    }
}

Provide logs (if relevant):

[2019-10-02T10:14:51,370][DEBUG][o.e.a.s.TransportSearchAction] [node-1] All shards failed for phase: [query]
org.elasticsearch.ElasticsearchParseException: failed to parse date field [2019-01-01] with format [epoch_second||basic_date_time_no_millis||yyyy-MM-dd HH:mm:ss||strict_date||strict_date_hour_minute_second]: [Text '2019-01-01' could not be parsed, unparsed text found at index 4]
        at org.elasticsearch.common.time.JavaDateMathParser.parseDateTime(JavaDateMathParser.java:235) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.time.JavaDateMathParser.parse(JavaDateMathParser.java:76) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parseToLong(DateFieldMapper.java:401) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.isFieldWithinQuery(DateFieldMapper.java:415) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.index.query.RangeQueryBuilder.getRelation(RangeQueryBuilder.java:447) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.index.query.RangeQueryBuilder.doRewrite(RangeQueryBuilder.java:464) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.index.query.AbstractQueryBuilder.rewrite(AbstractQueryBuilder.java:265) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.index.query.BoolQueryBuilder.rewriteClauses(BoolQueryBuilder.java:458) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.index.query.BoolQueryBuilder.doRewrite(BoolQueryBuilder.java:423) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.index.query.AbstractQueryBuilder.rewrite(AbstractQueryBuilder.java:265) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.search.builder.SearchSourceBuilder.rewrite(SearchSourceBuilder.java:962) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.search.builder.SearchSourceBuilder.rewrite(SearchSourceBuilder.java:82) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.index.query.Rewriteable.rewrite(Rewriteable.java:68) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.index.query.Rewriteable.rewrite(Rewriteable.java:51) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.search.internal.ShardSearchLocalRequest$RequestRewritable.rewrite(ShardSearchLocalRequest.java:271) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.search.internal.ShardSearchLocalRequest$RequestRewritable.rewrite(ShardSearchLocalRequest.java:261) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.index.query.Rewriteable.rewrite(Rewriteable.java:68) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:638) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:617) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:580) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:545) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:348) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.search.SearchService.lambda$rewriteShardRequest$7(SearchService.java:1043) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:45) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: java.time.format.DateTimeParseException: Text '2019-01-01' could not be parsed, unparsed text found at index 4
        at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2052) ~[?:?]
        at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1877) ~[?:?]
        at org.elasticsearch.common.time.JavaDateMathParser.parseDateTime(JavaDateMathParser.java:223) ~[elasticsearch-7.4.0.jar:7.4.0]
        ... 33 more

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[dliappis]

Possibly related to #45284?

[weberhofer]

Possibly related to #45284?

Maybe related, but #45284 is about partial parsing of data.

[pgomulka]

it reminds me of #39916 but the version is latest
@weberhofer are you 100% that the version is 7.4?

[pgomulka]

I see - that's the one #46654
this will be fixed in 7.5
you can work around by ordering your patterns from the most specific to least specific as suggested here https://bugs.openjdk.java.net/browse/JDK-8188771
so in this case I would go for this one
"yyyy-MM-dd HH:mm:ss||basic_date_time_no_millis||strict_date_hour_minute_second||strict_date||epoch_second";

[weberhofer]

Thanks a lot for your competent support. I'll wait for 7.5 as it's not a show-stopper for me.