[ML] properly nesting objects in document source

[benwtrent]

While working through use-cases, I found it impossible to push data to a range type field mapping. This was because we were not properly nesting documents in the JSON we are pushing to the index.

This change creates objects in the document source itself. This has two major benefits:

• Allows _preview to show the mapped objects as they will be actually stored
• Allows for more interesting and complex use cases with user provided mappings

Example use case that is enabled with this change:

PUT data-logs-by-client
{
  "mappings": {
    "properties": {
      "time_frame": {
          "type": "date_range"
        }
    }
  }
}

PUT _data_frame/transforms/data_log
{
  "source": {
    "index": "kibana_sample_data_logs"
  },
  "dest": {
    "index": "data-logs-by-client"
  },
  "pivot": {
    "group_by": {
      "machine.os": {"terms": {"field": "machine.os.keyword"}},
      "machine.ip": {"terms": {"field": "clientip"}}
    },
    "aggregations": {
      "time_frame.lte": {
        "max": {
          "field": "timestamp"
        }
      },
      "time_frame.gte": {
        "min": {
          "field": "timestamp"
        }
      }
    }
  }
}

This will result in an index where range queries are possible to determine which clients accessed the website over a given range.

Implementation details:

• I did not choose to create objects in the group_by fields as I could not think of a use case for it. The mapping created still treats the fields as an object (machine in the above use case), the document source just does not show it as plainly. I could be convinced otherwise :)
• I am not throwing an error when parsing and discovering duplicate fields, or objects that conflict. These validations should occur earlier in the process (see: [ML] verify that there are no duplicate leaf fields in aggs #41895) and any errors here should be logged then allowed, this is consistent with how we treat unsupported aggregations - validate ahead of time and log if something weird occurred.

[elasticmachine]

Pinging @elastic/ml-core

[przemekwitek]

What is the visible effect from the user perspective when this assertion fires?

[benwtrent]

There is none unless they are running with the -ea JVM flag.

[hendrikmuhs]

added some comments, my only concern are the error logs where we should IMHO throw (I understood both cases shouldn't be possible when validation is in)

[hendrikmuhs]

in 99% of cases fieldName is flat, so I wonder if we should have a shortcut optimization like:

if (fieldName.contains(".")) {
    document.put(fieldName, value);
    return;
}

and then briefly explain as a code comment what the special handling does.

[benwtrent]

That sounds good to me. It may provide a logic flow optimization, but I doubt there will be a performance one as updateDocument will just add the field and return.

[benwtrent]

Looking at how String#split is written, it does a handful of constant time calculations, and since we are splitting on a single char ., String#split when the char does not exist in the string is essentially the same as doing String.contains once.

I will add a clause that checks if (fieldTokens.length == 1) as an early shortcut, but I don't think it will cause any performance increase.

[benwtrent]

run elasticsearch-ci/docbldesx

[benwtrent]

run elasticsearch-ci/1

[hendrikmuhs]

LGTM!

Thanks for taking the extra mile fixing the flawed error logging.