[ML] make inference model definitions writeable

[benwtrent]

Model inference definitions are currently not serializable between nodes.

However, it is required for future LTR work that inference optimized models be serializable from the coordinator -> data nodes.

This adds wire serialization code for our ensemble and tree inference models. From a user perspective, this change does nothing.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[benwtrent]

Pinging @elastic/es-search (Team:Search)

[droberts195]

LGTM

There's nothing wrong with this change in itself. However, I think there could be complications with running boosted tree models extensively during searches to support LTR. Autoscaling doesn't take into account the memory required for inference models inside the JVM. Currently it's possible to work around this by disabling autoscaling and selecting bigger nodes manually. In serverless that won't be possible. This isn't a new problem, but introducing LTR could change it from an obscure edge case that hardly affects anyone into a major problem.

[benwtrent]

Autoscaling doesn't take into account the memory required for inference models inside the JVM. Currently it's possible to work around this by disabling autoscaling and selecting bigger nodes manually. In serverless that won't be possible.

Good point. I will think a bit more about how we expose this. It would be bad for 1000s of search requests to all instantiate the same model needlessly.

[benwtrent]

I am going to close this for now and think a bit more.

Regardless of the autoscaling issue, I am not sure we want to serialize models for the sake of inferencing at all without caching them in their location (like the ModelLoader).