Support date_nanos in Kibana

[timroes]

This issue describes supporting the date_nanos type in Kibana.

What we cannot do

We can't fully support date_nanos in Kibana, due to the technical limitations of JavaScript. JavaScript stores all numeric values in a 64bit floating point number. Thus we only can represent integer (non decimal numbers) to a precision of 52 bit (the mantisse in that floating point number). Every number above that, will just not be stored incorrectly trying to use them in JavaScript, see how I am trying to store and access the current timestamp in nanoseconds in my browser and as you can see the number in the variable x is not the same as I tried to store in it:

That means that even the current timestamp in nanoseconds cannot be stored in JavaScript properly (which is also the reason behind our general lack for 64bit integer support, see #1274), without being rounded a bit (usually you'll lose at least the nano second part). Also our date formatting library moment.js cannot and will never support formatting and parsing of nanoseconds (most likely due to the same constraints, that they are not representable in a JavaScript number.

What we can do

We can enable date_nanos treat them as date fields and just ignore everything smaller then milliseconds.
This would mean:

• We show a date_nanos field in index pattern as a date field (we might want to show a warning that you are not seeing exact values here).
• You can use it as date fields all across Kibana.
• You can actually sort on that field in discover and it takes the nano second part into account (because that sorting is done by Elasticsearch, which obviously supports 64bit integer (long) numbers).
• Everywhere where we show that value (e.g. in Discover in the document) we'll just show the date the same as any other date field (to millisecond precision). I've checked that this worked technically, because we can safely parse a date string containing nanoseconds in moment.js, and it will just throw away those additional information.
• You can use it in visualization, since Elasticsearch anyway only support aggregating on the millisecond precision and return those timestamps. If ES would every allow aggregating on nanosecond precision we would run into issues, and might need additional logic to work around that or disable those fields for visualizations.

What needs to be done

After some checking and discussions with and through @flash1293 and @Bargs the following things need to be addressed to implement it:

• Map the date_nanos type to the Kibana date type in kbn_field_types.js
• Elasticsearch currently breaks on using time zones in the date histogram (date_histogram aggregation fails for date_nanos and noon utc timezone elasticsearch#39107)
• Context view is crashing, due to the rounding error in nanos (we need to find a proper workaround for handling loading "earlier/later" documents in context view
• Nano second precision is numerical returned in stored_fields and sort so we need to check if we missed some usage of those.

[elasticmachine]

Pinging @elastic/kibana-app

[flash1293]

Kibana currently doesn't recognize date_nanos as a date field, the esTypes mapping has to be extended in src/legacy/utils/kbn_field_types.js in line 56.

Non utc timezones throw an error on discover because elasticsearch doesn't handle the timezone correctly: elastic/elasticsearch#39107

The discover view should support the display of nano seconds.
Maybe there are more places in visualizations (e.g. data table) which should also support nano second formats.

Sorting in discover works correctly

[lukasolson]

Possibly related: #30420

[timroes]

I've updated the description of the issue with the current state and especially a description why we can't support date_nanos completely since we're not Java, but JavaScript :-)

cc @AlonaNadler @tbragin

[kertal]

Do we have some date_nanos testdata available, currently trying to convert our testdata to date_nanos with reindexing, which is in general no problem, but I'd need some data with nano precision. Waiting for some painless support to add nanos to existing data, but maybe we already have some data for this case?

[timroes]

@kertal I am pretty sure we don't have yet. Since we want tests on that anyway, we most likely also need to create some esarchiver data for use in functional tests that contain date_nanos fields. So I think nothing predone already.

[kertal]

@timroes I think we need 2 PRs for this:

1. adding the date_nanos + check for incompatibilities + disabling the context view for date_nanos based timestamps. It's work in progress, looking good so far, managed to convert demo log data from date to date_nanos Support Elasticsearch date_nanos datatype #36111 (comment) for testing
2. taking care of the context view, I've found a possible solution, but it's experimental so far, needs definitely more time

[timroes]

I'll go forward and close this issue, since basic date_nanos support is now implemented from 7.3 onwards in Kibana. We track additional enhancements (like a configurable nanosecond date formatter) in separate PRs.

[bmerry]

Thanks, that sounds like good news! Will it sort event timestamps correctly i.e. at the full nanosecond precision? That would cover 99% of my motivation for this request since I just need to break ties properly for log messages coming from a single process.

[timroes]

@bmerry Yes, it will sort nanosecond timestamps stored in date_nanos correctly, since the sorting is done by Elasticsearch. If they are stored in a long fields they are already sorted correctly today -- just not shown correctly in Discover.

[bmerry]

I've now installed Kibana 7.3 and created a fresh new ElasticStack environment on my dev machine for testing. I can now pick a date_nanos field as the timestamp field when adding the logstash-* index pattern, but when I go to Discover I get this error. Any idea what I'm doing wrong? The time picker is set to the default of last 15 minutes, so I'm not sure why it's querying 1944.

Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"date[1944-03-18T23:00:30Z] is before the epoch in 1970 and cannot be stored in nanosecond resolution"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-2019.08.16","node":"vQqDKdkvQSm4WIK7AYMigQ","reason":{"type":"illegal_argument_exception","reason":"date[1944-03-18T23:00:30Z] is before the epoch in 1970 and cannot be stored in nanosecond resolution"}}],"caused_by":{"type":"illegal_argument_exception","reason":"date[1944-03-18T23:00:30Z] is before the epoch in 1970 and cannot be stored in nanosecond resolution","caused_by":{"type":"illegal_argument_exception","reason":"date[1944-03-18T23:00:30Z] is before the epoch in 1970 and cannot be stored in nanosecond resolution"}}},"status":400}

KbnError@http://localhost:5601/bundles/commons.bundle.js:1:10937
RequestFailure@http://localhost:5601/bundles/commons.bundle.js:1:11430
callResponseHandlers/<@http://localhost:5601/bundles/commons.bundle.js:1:1810379
Promise.try@http://localhost:5601/bundles/commons.bundle.js:1:995957
Promise.map/<@http://localhost:5601/bundles/commons.bundle.js:1:995312
Promise.map@http://localhost:5601/bundles/commons.bundle.js:1:995273
callResponseHandlers@http://localhost:5601/bundles/commons.bundle.js:1:1809391
fetchSearchResults/<@http://localhost:5601/bundles/commons.bundle.js:1:1792163
processQueue@http://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:281:199747
scheduleProcessQueue/<@http://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:281:200710
$digest@http://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:281:210518
$evalAsync/<@http://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:281:213229
completeOutstandingRequest@http://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:281:64643
Browser/self.defer/timeoutId<@http://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:281:67485

I'm using the docker.elastic.co/kibana/kibana-oss:7.3.0 Docker image.

[bmerry]

Ah, looks like my previous comment may be the same as #40699.