Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] File Data Visualizer - error on certain custom timestamp formats #44110

Closed
pheyos opened this issue Jul 9, 2019 · 2 comments · Fixed by #44127
Closed

[ML] File Data Visualizer - error on certain custom timestamp formats #44110

pheyos opened this issue Jul 9, 2019 · 2 comments · Fixed by #44127
Assignees
@droberts195
Labels
>bug :ml Machine learning

Comments

@pheyos
Copy link
Member

pheyos commented Jul 9, 2019

Found in version

  • 7.3.0-bc1

Steps to reproduce

  • View the following delimited files in the file data viz and try to use the custom timestamp pattern
  • A) pattern: 'some_prefix 'dd.MM.yyyy HH:mm:ss.SSSSSS, data:
timestamp|loglevel|message|id1|id2
some_prefix 06.01.2018 16:56:14.295748|INFO|abc|1|2
some_prefix 06.01.2018 17:19:44.465252|INFO|def|3|4
some_prefix 06.01.2018 17:21:25.764368|INFO|ghi|5|6
  • B) pattern: 'dd.MM. yyyy HH:mm:ss.SSSSSS, data:
timestamp|loglevel|message|id1|id2
06.01. 2018 16:56:14.295748|INFO|abc|1|2
06.01. 2018 17:19:44.465252|INFO|def|3|4
06.01. 2018 17:21:25.764368|INFO|ghi|5|6

Expected result

  • The timestamp pattern matches the timestamp field, the operation returns with a success in both cases

Actual result

  • In both cases a [syntax_exception] unmatched close parenthesis error is shown, the ES log has the following error:
[2019-07-09T13:36:38,576][WARN ][r.suppressed             ] [elastic-ro] path: /_ml/find_file_structure, params: {timestamp_format=dd.MM. yyyy HH:mm:ss.SSSSSS, charset=, has_header_row=, quote=, should_trim_fields=, grok_pattern=, delimiter=, format=, column_names=, timestamp_field=timestamp, lines_to_sample=1000}
org.joni.exception.SyntaxException: unmatched close parenthesis
	at org.joni.ScannerSupport.newSyntaxException(ScannerSupport.java:166) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Parser.parseExp(Parser.java:731) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Parser.parseBranch(Parser.java:1078) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Parser.parseSubExp(Parser.java:1095) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Parser.parseExp(Parser.java:724) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Parser.parseBranch(Parser.java:1069) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Parser.parseSubExp(Parser.java:1095) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Parser.parseRegexp(Parser.java:1128) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Parser.parse(Parser.java:65) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Analyser.compile(Analyser.java:80) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Regex.<init>(Regex.java:159) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Regex.<init>(Regex.java:136) ~[joni-2.1.6.jar:1.1.7]
	at org.joni.Regex.<init>(Regex.java:122) ~[joni-2.1.6.jar:1.1.7]
	at org.elasticsearch.grok.Grok.<init>(Grok.java:106) ~[elasticsearch-grok-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.grok.Grok.<init>(Grok.java:86) ~[elasticsearch-grok-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.filestructurefinder.TimestampFormatFinder$CandidateTimestampFormat.<init>(TimestampFormatFinder.java:1391) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.filestructurefinder.TimestampFormatFinder$CandidateTimestampFormat.<init>(TimestampFormatFinder.java:1373) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.filestructurefinder.TimestampFormatFinder.makeCandidateFromOverrideFormat(TimestampFormatFinder.java:373) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.filestructurefinder.TimestampFormatFinder.<init>(TimestampFormatFinder.java:228) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.filestructurefinder.FileStructureUtils.findCandidates(FileStructureUtils.java:153) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.filestructurefinder.FileStructureUtils.guessTimestampField(FileStructureUtils.java:75) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.filestructurefinder.DelimitedFileStructureFinder.makeDelimitedFileStructureFinder(DelimitedFileStructureFinder.java:111) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.filestructurefinder.DelimitedFileStructureFinderFactory.createFromSample(DelimitedFileStructureFinderFactory.java:67) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.filestructurefinder.FileStructureFinderManager.makeBestStructureFinder(FileStructureFinderManager.java:305) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.filestructurefinder.FileStructureFinderManager.findFileStructure(FileStructureFinderManager.java:156) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.filestructurefinder.FileStructureFinderManager.findFileStructure(FileStructureFinderManager.java:125) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.action.TransportFindFileStructureAction.buildFileStructureResponse(TransportFindFileStructureAction.java:54) ~[x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.action.TransportFindFileStructureAction.lambda$doExecute$0(TransportFindFileStructureAction.java:42) [x-pack-ml-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.3.0-SNAPSHOT.jar:7.3.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]

Additional information

  • The following (only slightly different) example works fine:
  • C) pattern: 'dd.MM.yyyy HH:mm:ss.SSSSSS, data:
timestamp|loglevel|message|id1|id2
06.01.2018 16:56:14.295748|INFO|abc|1|2
06.01.2018 17:19:44.465252|INFO|def|3|4
06.01.2018 17:21:25.764368|INFO|ghi|5|6
@pheyos pheyos added >bug :ml Machine learning labels Jul 9, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core

@droberts195 droberts195 self-assigned this Jul 9, 2019
droberts195 added a commit to droberts195/elasticsearch that referenced this issue Jul 9, 2019
@droberts195
[ML] Fix custom timestamp override with dot-separated fractional seconds
581afdd 
Custom timestamp overrides provided to the find_file_structure
endpoint produced an invalid Grok pattern if the fractional
seconds separator was a dot rather than a comma or colon.
This commit fixes that problem and adds tests for this sort
of timestamp override.

Fixes elastic#44110
@droberts195 droberts195 mentioned this issue Jul 9, 2019
Merged
@droberts195
Copy link
Contributor

This bug is triggered when there's a dot before the fractional seconds rather than a comma or colon. I've opened #44127 to fix it.

droberts195 added a commit that referenced this issue Jul 10, 2019
@droberts195
[ML] Fix custom timestamp override with dot-separated fractional seco…
bca4edc 
…nds (#44127)

Custom timestamp overrides provided to the find_file_structure
endpoint produced an invalid Grok pattern if the fractional
seconds separator was a dot rather than a comma or colon.
This commit fixes that problem and adds tests for this sort
of timestamp override.

Fixes #44110
droberts195 added a commit that referenced this issue Jul 10, 2019
@droberts195
[ML] Fix custom timestamp override with dot-separated fractional seco…
853ddb5 
…nds (#44127)

Custom timestamp overrides provided to the find_file_structure
endpoint produced an invalid Grok pattern if the fractional
seconds separator was a dot rather than a comma or colon.
This commit fixes that problem and adds tests for this sort
of timestamp override.

Fixes #44110
droberts195 added a commit that referenced this issue Jul 10, 2019
@droberts195
[ML] Fix custom timestamp override with dot-separated fractional seco…
9a2290c 
…nds (#44127)

Custom timestamp overrides provided to the find_file_structure
endpoint produced an invalid Grok pattern if the fractional
seconds separator was a dot rather than a comma or colon.
This commit fixes that problem and adds tests for this sort
of timestamp override.

Fixes #44110
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@droberts195 droberts195

Labels
>bug :ml Machine learning
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML] Fix custom timestamp override with dot-separated fractional seconds droberts195/elasticsearch
3 participants
@pheyos @droberts195 @elasticmachine