[Transform] enhance the output of preview to return full destination index details

[hendrikmuhs]

changes the output format of preview regarding deduced mappings and enhances it to return all the details about auto-index creation. This allows the user to customize the index creation. Using HLRC you can create a index request from the output of the response. Long term this allows us to add UI support, see elastic/kibana#57059 and/or improve setting defaults.

The output of the preview itself is not affected. The UI does not use/display mappings at the moment (CC @walterra )

Breaking:

• supported:
  • mixed clusters internal communication, all versions
  • HLRC >=7.7 <-> <7.7
• not supported:
  • HRLC <7.7 <-> >=7.7 (mappings are parsed as empty)

New format:

{ 
    "preview": [
       ...
    ],
    "generated_dest_index": {
        "mappings" : {
        ...
        },
        "settings": {
            "index" : {
                "number_of_shards" : 1,
                "auto_expand_replicas" : "0-1",
                ...
            }
        },
        "aliases": {}
    }
}

Please raise concerns about the naming: generated_dest_index.

Old format:

{ 
    "preview": [
       ...
    ],
    "mappings" : {
       ...
    }
}

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[benwtrent]

love this change.

One comment about a logging level, but looks good.

Tricky thing with HLRC, all versions need to be forward-compatible with minor versions in the same major.

This change should be OK as the HLRC continues to work as a whole.

[walterra]

Just caught up with emails and missed the CC unfortunately: We use mappings in the UI to infer field types for the preview tables. I'll come up with a fix this afternoon!

[walterra]

Fixed in Kibana for master/7.x: elastic/kibana#60609