Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ml] supporting date_nanos as timestamp for the input #49889

Closed
jguay opened this issue Dec 5, 2019 · 5 comments
Closed

[ml] supporting date_nanos as timestamp for the input #49889

jguay opened this issue Dec 5, 2019 · 5 comments
Labels
>enhancement :ml Machine learning

Comments

@jguay
Copy link
Contributor

jguay commented Dec 5, 2019

At present, trying to use a field of datatype date_nanos will return error :

'fieldName' cannot be used as the time-field because it's not a valid field of type 'date'.

Some indices may only use timestamps with precision requiring datatype date_nanos, these indices cannot be used for input for machine learning jobs (unless another field of datatype date is added)

Note this feature request is limited to the input - it not related to being able to process data in nano seconds or below current second bucket_span raised in 644

@jguay jguay added >enhancement :ml Machine learning labels Dec 5, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml)

@davidkyle
Copy link
Member

Using the ml APIs directly rather than kibana it is possible to create a run a job on data with a time field of type date_nanos I have tested this on version 7.5.

The error mentioned above comes from the UI, I've raised elastic/kibana#52921 in place of this.

@sophiec20 sophiec20 reopened this Jan 20, 2020
@sophiec20
Copy link
Contributor

sophiec20 commented Jan 20, 2020

Re-opening. As it stands with 7.6 and prior, using timestamps containing data with precision of date_nanos is not yet supported and will be documented as a limitation. Both UI and back end work is required to add this functionality.

@droberts195
Copy link
Contributor

Using the ml APIs directly rather than kibana it is possible to create a run a job on data with a time field of type date_nanos

It turns out this was only true if the extra precision was not used. In other words, a datafeed would work fine on the date_nanos type providing every timestamp was an exact number of milliseconds since the epoch, i.e. nanoseconds since the epoch was a multiple of 1000000.

#51180 fixes the case of a fractional number of milliseconds since the epoch. (And note that the smallest possible ML bucket_span remains 1 second, and anomaly detection will still operate on summary statistics at the bucket level regardless of the underlying timestamp accuracy.)

droberts195 added a commit that referenced this issue Jan 21, 2020
Allows ML datafeeds to work with time fields that have
the "date_nanos" type _and make use of the extra precision_.
(Previously datafeeds only worked with time fields that were
exact multiples of milliseconds.  So datafeeds would work
with "date_nanos" only if the extra precision over "date" was
not used.)

Relates #49889
droberts195 added a commit that referenced this issue Jan 21, 2020
Allows ML datafeeds to work with time fields that have
the "date_nanos" type _and make use of the extra precision_.
(Previously datafeeds only worked with time fields that were
exact multiples of milliseconds.  So datafeeds would work
with "date_nanos" only if the extra precision over "date" was
not used.)

Relates #49889
@droberts195
Copy link
Contributor

I believe this is fixed in 7.7 and above by elastic/kibana#59017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :ml Machine learning
Projects
None yet
Development

No branches or pull requests

5 participants