[ML] find_file_structure endpoint override can change field type as an unexpected side effect

[jgowdyelastic]

Using the example file ts.log if you override the name of an unrelated field, the type of the field extra_timestamp will change from a date to a keyword

Steps to reproduce:

1. open syslog-sample.log in the File Data Visualizer
2. note the extra_timestamp field is a date type
3. Open the overrides flyout and in the Grok pattern text field change ipaddress to ipaddress1
4. Press apply
5. The results are reloaded and now extra_timestamp is a keyword type.

[elasticmachine]

Pinging @elastic/ml-core

[droberts195]

This happens because the majority of messages in the input file have an extra_timestamp of the form Mar 30 12:31:20 CDT. But there is one message that has Mar 30 13:31:19.038 EDT for its extra_timestamp.

The Grok pattern SYSLOGTIMESTAMP will consume the .038 in the odd one out, but the date format for syslog timestamps doesn't accept fractional seconds.

Because one of the timestamps is in a different format, it's arguably more correct not to detect the field as a timestamp, because the timestamp parsing will fail for that message.

This problem comes under the theme of "Grok patterns are more lenient than date parsing patterns". Because of the need to maintain backwards compatibility there is no trivial solution to this problem. I don't think we should try to rush a fix into 6.5, especially as the feature is experimental. One possible fix would be to have both a strict regex as well as a lenient Grok pattern for each candidate pattern in the GrokPatternCreator, and only detect a field if the strict regular expression matches. But more thought and testing is required.

[droberts195]

I tested this with the changes of #41948 and the problem is solved. The mappings after changing ipaddress to ipaddress1 are:

    "mappings" : {
      "_meta" : {
        "created_by" : "ml-file-data-visualizer"
      },
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "extra_timestamp" : {
          "type" : "date",
          "format" : "MMM dd HH:mm:ss||MMM  d HH:mm:ss||MMM dd HH:mm:ss.SSS||MMM  d HH:mm:ss.SSS"
        },
        "field" : {
          "type" : "long"
        },
        "field2" : {
          "type" : "long"
        },
        "ipaddress1" : {
          "type" : "ip"
        },
        "message" : {
          "type" : "text"
        }
      }
    }

extra_timestamp survives as a date.