[ML] Fixes for multi-line start patterns in text structure endpoint

[droberts195]

This PR contains 3 fixes for the way multi-line start patterns are
created in the text structure endpoint:

1. For delimited files the multi-line start pattern used to be based
   on the first field that was a timestamp, boolean or number. This
   PR adds the option of a low cardinality keyword field as an
   alternative (i.e. an enum field effectively). It means there is
   more chance of a field early in each record being chosen as the
   mechanism for determining whether a line is the first line of a
   record.
2. The multi-line start pattern for delimited files now only permits
   the delimiter character between fields, not within quoted fields.
   Previously it was possible for the multi-line start pattern to
   match continuation lines. Unfortunately this may mean we can no
   longer determine a multi-line start pattern for files whose only
   suitable field is to the right of fields that sometimes contain
   commas, and the only solution in this case will be to reorder the
   columns before importing the data. Hopefully this problem will be
   very rare.
3. For semi-structured text log files there is now a cap on the
   complexity of the multi-line start pattern. It has been observed
   that the patterns generated for slightly malformed CSV files could
   run for days against the malformed lines of those files - the
   classic problem of a regex that doesn't match but nearly does doing
   lots of backtracking. We now throw an error in this situation and
   suggest overriding the format to delimited.

Relates #79708
Fixes elastic/kibana#121966

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Hi @droberts195, I've created a changelog YAML for you.

[przemekwitek]

I think it would be a bit easier to follow the control flow if there was only one loop through the sequence:

    static int longestRun(List<?> sequence) {
        if (sequence.size() <= 1) {
            return sequence.size();
        }
        int maxSoFar = 0;
        int thisCount = 1;
        for (int index = 1; index < sequence.size(); ++index) {
            if (sequence.get(index).equals(sequence.get(index - 1))) {
                ++thisCount;
            } else {
                maxSoFar = Math.max(maxSoFar, thisCount);
                thisCount = 1;
            }
        }
        maxSoFar = Math.max(maxSoFar, thisCount);
        return maxSoFar;
    }

[przemekwitek]

It is kind of covered by the random, but you could add explicit test cases for:

• the longest sequence is the prefix
• the longest sequence is the suffix

[droberts195]

I made it more likely that these cases get tested by making each true 50% of the time. I tested all the scenarios locally and I think one test with higher probability of testing the edge cases should be sufficient to prevent regressions. There shouldn't be much need to change this code again.

[przemekwitek]

LGTM

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	7.17	Commit could not be cherrypicked due to conflicts
✅	8.1

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 85066