Date type has not enough precision for the logging use case.

[jordansissel]

At present, the 'date' type is millisecond precision. For many log use cases, higher precision time is valuable - microsecond, nanosecond, etc.

The biggest impact of this is during sorting of search results. If you sort chronologically, newest-first, by a date field, documents with the same date will probably be sorted incorrectly (because they match). This is often reported by users seeing events "out of order" when they have the same timestamp. Specific example being sorting by date and seeing events in newest-first order, unless there is a tie, in which case oldest-first (or first-written?) appears. This causes a bit of confusion for the ELK use case.

Related: logstash-plugins/logstash-filter-date#8

I don't have any firm proposals, but I have two different implementation ideas:

• Proposal 1, use a separate field: Store our own custom-precision time in a separate field as a long. This allows us to do correct sorting (because we have higher precision), but it makes any date-related functionality in Elasticsearch not usable (searching now-1h or doing date_histogram, etc)
• Proposal 2, date type has tunable precision: Have the date type have configurable precision, with the default (backwards compatible) precision being milliseconds. This would let us choose, for example, nanosecond precision for the logging use case, and year precision for an archaeological use case (billions of years ago, or something). Benefit here is date histogram and other date-related features could still work. Further, having the precision configurable would allow us to keep the underlying data structure a 64bit long and users could choose their most appropriate precision.

[jordansissel]

I know Joda's got a precision limit (the Instant class is millisecond precision) and a year limit ("year must be in the range [-292275054,292278993]"). I'm open to helping explore solutions in this area.

[synhershko]

What about consequences to field_date size? even with docvalues in place, cardinality will be ridiculously high. Even for those scenarios which need this, this could be an overkill, no?

[nikonyrh]

Couldn't you just store the decimal part of the second in a secondary field (as a float or long) and sort by these two fields when needed? You could still aggregate based on the standard date field but not at a microsecond resolution.

[markwalkom]

I've been speaking to a few networking firms lately and it's dawned on me that microsecond level is going to be critical for IDS/network analytics.

[markwalkom]

Another request from the community - https://discuss.elastic.co/t/increase-the-resolution-of-elastic-timestamps-to-nanoseconds/24227

[anoinoz]

that last request is mine I believe. I would add that to monitor networking (and other) activities in our field, nanosecond support is paramount.

[abrisse]

👍 for this feature

[jack-pappas]

What about switching from Joda Time to date4j? It supports higher-precision timestamps compared to Joda and supposedly the performance is better as well.

[dadoonet]

Before looking on the technical side, is BSD License compatible with Apache2 license?

[dadoonet]

So BSD is compatible with Apache2.

[clintongormley]

I'd like to hear @jpountz's thoughts on this comment #10005 (comment) about high cardinality with regards to index size and performance.

I could imagine adding a precision parameter to date fields which defaults to ms, but also accepts s, us, ns.

We would need to move away from Joda, but I wouldn't be in favour of replacing Joda with a different dependency. Instead, we have this issue discussing replacing Joda with Java.time #12829