MLJAR models in production

[RafaD5]

Hi!
I'm using MLJAR to train models that will be used in production, so prediction time is an important metric.
For example, MLJAR found that the model with the best performance is an ensemble with a hot prediction time of 1.2 seconds for one sample. I have a limit of 0.6 seconds. So what I did was organize the models that make up the ensemble by weight (repeat) and calculate the predictions time of each one.

Then I trimmed from bottom to top until the prediction time of the ensemble was lower than 0.6 seconds and, for my surprise, the performance of the ensemble decreased very little.

I think it would be nice to have a parameter max_single_prediction_time so that this is taken into account during training. Furthermore, single models with a prediction time greater than max_single_prediction_time could be ignore and not further optimize in the remaining steps of the training.

[pplonski]

@RafaD5 great idea! I need to think about this. For sure there is a need to select only models that are fast during prediction. Maybe this should be the default set-up for Perform mode?

How do you compute the 0.6 sec limit for a single sample prediction?

How are you going to deploy models as REST API?

[RafaD5]

Yes, I think I'd be great!

We have a total time of 1.5 sec for the whole process: Get the request from the user, calculating the prediction and printing the result. The 0.6 seconds for the model isn't a calculation, it's a design decision.

We're deploying the model as a RESTful API with Flask

[pplonski]

@RafaD5 I will do as you suggested. There will be the max_single_prediction_time argument passed to the AutoML constructor. By default the max_single_prediction_time=None, which will mean that we don't need to worry about prediction time. If the max_single_prediction_time will be set then AutoML will search for the best model with prediction time under the limit. For constructing Ensemble will be considered only models with smaller prediction time. The Ensemble will be optimized in a way to restrict the prediction time limit (of course if parameter set). If there will be no model with a prediction time smaller than the limit, then Ensemble will not be trained, and the fastest ML model will be returned.

I will try to implement this in the 0.9.0 release. Should be this week (I hope!).

@RafaD5 I have also a plan to implement drift detection, so if there will be new data it will be easy to detect if you need to retrain the AutoML. Details: #179 Do you have such a problem in your production setting? Would you be interested in testing this feature?

[RafaD5]

The max_single_prediction_time implementation sounds great. I'm eager to try it. Thanks a lot.

I think the drift detection it's a terrific idea! We collect data on a daily basis and retrain our models every one or two weeks, but we don't really know if retraining is necessary, so this new feature would help us with this. We will gladly test it.

@pplonski I tried adding you on LinkedIn so we can talk easily but I couldn't send you the invitation.

[pplonski]

@RafaD5 I added the argument max_single_prediction_time in AutoML() constructor. The code is ready for try in the dev branch.

[ta-oliver]

Having max_single_prediction_time as part of the model selection would be great. We previously tried creating estimations of time required for input length etc, so that we would only make calls when they would return within a specified time, but this ended up with a second model (for estimating run time) needed on top of the model itself.

[pplonski]

Hi @ta-oliver, the max_single_prediction_time is already added. You can specify the time limit for single prediction. The time is set in seconds. The limit is considered also in building an Ensemble. By default the Perform mode has max_single_prediction_time set to 0.5 second.