Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] File Data Visualiser - Allow custom timestamp format #31055

Open
markwalkom opened this issue Feb 14, 2019 · 6 comments
Open

[ML] File Data Visualiser - Allow custom timestamp format #31055

markwalkom opened this issue Feb 14, 2019 · 6 comments
Labels

Comments

@markwalkom
Copy link
Contributor

Describe the feature: It'd be great to be able to specify custom timestamp formats in the Override Settings section.

Describe a specific use case for the feature:
I had a CSV file with a timestamp like 1/31/2019 1:40pm, which isn't listed in the dropdown of possible date formats.

Having the ability to create your own custom notation would be useful for occasions like this.

@markwalkom markwalkom added the :ml label Feb 14, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui

@peteharverson peteharverson added the Feature:Anomaly Detection ML anomaly detection label Feb 14, 2019
@peteharverson peteharverson changed the title Data Visualiser - Allow custom timestamp format [ML] Data Visualiser - Allow custom timestamp format Feb 14, 2019
@jgowdyelastic jgowdyelastic changed the title [ML] Data Visualiser - Allow custom timestamp format [ML] File Data Visualiser - Allow custom timestamp format Feb 14, 2019
@droberts195
Copy link
Contributor

This is the second checkbox in elastic/elasticsearch#38086. After that's implemented on the backend it should be easy to add to the UI.

@droberts195
Copy link
Contributor

The change to the backend (currently targeting version 7.3) will not allow completely arbitrary timestamp formats, but will allow formats build from a reasonable variety of components in any order - see the docs change in https://github.com/elastic/elasticsearch/pull/41948/files#diff-cf0bc7031119acc3f6c3de16748677e6 for what is proposed to be supported.

1/31/2019 1:40pm would then be M/d/yyyy h:mma, and all those letter groups will be supported.

@droberts195
Copy link
Contributor

Argh, there's a complication. It seems that Java makes it extremely hard to parse lower case am/pm (see https://stackoverflow.com/a/46480738).

So M/d/yyyy h:mma will be make it possible to parse 1/31/2019 1:40PM, but sadly not 1/31/2019 1:40pm.

However, this is not really the fault of the file structure finder, as downstream functionality like ingest pipeline and mappings also suffers from this.

@droberts195
Copy link
Contributor

#36880 will add support for a certain amount of custom timestamp formats, including 1/31/2019 1:40PM. But, as stated in the previous comment, 1/31/2019 1:40pm is more problematic as it is not supported by Java.

@droberts195
Copy link
Contributor

#38868 covers what was part 4 of #36880, as that is not being implemented in #38727.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants