PUBDEV-7831: Add new learning curve plotting function/method

[tomasfryda]

https://h2oai.atlassian.net/browse/PUBDEV-7831

Need to be tested for multiple different model configurations.
For now, this should work at least on the models that we get from AutoML (it is tested mainly on those).

Questionable choices:

• When provided with StackedEnsemble, draw learning curve for the metalearner and make sure to have the metalearner model id in the plot title to make this choice explicitly visible.

• If it seems to be hard to make nice ribbon, don't make a ribbon and instead draw individual CV lines. This is because of "DeepLearning epochs doesn't seem to be saved in the same point so the sd estimation for ribbon fails".

Related fixes

• CoxPH scoring history is missing last entry: Fix CoxPH scoring history #5186 (merged)
• GLM with lambda search reports number of iterations of the last model, not the winning submodel (merged)

Example

[wendycwong]

@tomasfryda

The scoring history only has xval deviance with it and nothing else. However, if you want to checkout other values apart from deviance, I suggest that you add them to scoring history instead of having cv_scoring_history. I think it will be easier.

Also, the CV in GLM only calculated the deviance values. If you need other metrics calculated, you can add it there (in method cv_computeAndSetOptimalParameters(ModelBuilder[] cvModelBuilders)) as well. The CV is used in GLM to figure out the best alpha and lambda values to use and nothing else. Also, the number of iterations may not be the same as in the main run.

I will check what happens when you limit the max runtime.

Wendy

[wendycwong]

Also, at the end of cross-validation, we will calculate the deviance on the hold-out set and pick the best deviance for the alpha/lambda value. However, after that, we will not bother to save the deviance values for the later iterations. This is where it happens (in method cv_computeAndSetOptimalParameters(ModelBuilder[] cvModelBuilders)):

  _parms._lambda = Arrays.copyOf(_parms._lambda,lmin_max+1);
  _xval_deviances = Arrays.copyOf(_xval_deviances, lmin_max+1);
  _xval_sd = Arrays.copyOf(_xval_sd, lmin_max+1);

where lmin_max is the submodel with the lowest test deviance.

[wendycwong]

Okay, when you set a max_runtime_secs and nfold > 1, each cv model and main model will be allocated equal time to build their model like this (in maxRuntimeSecsPerModel(int cvModelsCount, int parallelization) of ModelBuilder.java):

_parms._max_runtime_secs / Math.ceil((double)cvModelsCount / parallelization + 1).

So, in theory, they should not run out of time. Also, for GLM, it will always run for one iteration before it quits so as not to return an empty model. This will implies that we may use more time than is allowed by max_runtime_secs. Hope this helps. Wendy

[tomasfryda]

_parms._max_runtime_secs / Math.ceil((double)cvModelsCount / parallelization + 1).

So, in theory, they should not run out of time. Also, for GLM, it will always run for one iteration before it quits so as not to return an empty model. This will implies that we may use more time than is allowed by max_runtime_secs. Hope this helps. Wendy

Thank you @wendycwong. Sorry, I most likely misled you by concentrating on the time constraint.

I added a lot of logging and fortunately I was able to reproduce it after couple runs. It looks like the problem is that GLM selects one of the first alpha values tested but generateSummary is called after the model is finished and it uses the _state._iter as number_of_iterations, however, the _state._iter contains the number of iteration from the last tried alpha value (which in this case is not the alpha_best). My fix would be something like adding a line to hex.glm.GLMModel#generateSummary:

if (_parms._lambda_search) { //h2o-algos/src/main/java/hex/glm/GLMModel.java#L1534
    lambdaSearch = 1;
    iter = _output._submodels[_output._selected_submodel_idx].iteration;  // THIS IS THE ADDED LINE
    _output._model_summary.set(0, 3, "nlambda = " + _parms._nlambdas + ", lambda.max = " + MathUtils.roundToNDigits(_lambda_max, 4) + ", lambda.min = " + MathUtils.roundToNDigits(_output.lambda_best(), 4) + ", lambda.1se = " + MathUtils.roundToNDigits(_output.lambda_1se(), 4));
}

Note that I still don't understand well how GLM works inside, e.g., I don't know if the submodels are created during CV or just final model and if the iteration in a submodel corresponds to the iteration of the final model that is used for prediction so this fix might be incorrect. What do you think?

---

The scoring history only has xval deviance with it and nothing else. However, if you want to checkout other values apart from deviance, I suggest that you add them to scoring history instead of having cv_scoring_history. I think it will be easier.

I created the cv_scoring_history[] because it seemed to me easier than adding it to the scoring_history.

I don't have a strong opinion about this but when I look at glm@model$scoring_history I get the table below, which seems to me like a very wide table with a lot of NAs already.

And since I want to get the actual scoring history of the CV models I would probably have to either

• add a new column that would indicate whether it is a CV model or final model and rbind the cv scoring_histories, or

• add multiple columns such as cv_training_rmse so there is a clear distinction between the final model and CV models and since some models are scored based on some time interval (I think deep learning) this would add a lot of new rows that would have NAs for the cv model or the final one.

But I might be missing some better way to do it - if you still think your idea is better could you please describe it more concretely?

            timestamp   duration iterations negative_log_likelihood objective training_rmse training_logloss
1 2020-12-11 12:25:13  0.000 sec          0               689.51201   0.66236            NA               NA
2 2020-12-11 12:25:13  0.001 sec          1               252.03686   0.36284            NA               NA
3 2020-12-11 12:25:13  0.002 sec          2               222.59586   0.35667            NA               NA
4 2020-12-11 12:25:13  0.002 sec          3               220.00584   0.35661            NA               NA
5 2020-12-11 12:25:13  0.003 sec          4               219.97507   0.35661       0.21102          0.21131
  training_r2 training_auc training_pr_auc training_lift training_classification_error validation_rmse
1          NA           NA              NA            NA                            NA              NA
2          NA           NA              NA            NA                            NA              NA
3          NA           NA              NA            NA                            NA              NA
4          NA           NA              NA            NA                            NA              NA
5     0.81033           NA              NA       2.65561                       0.02498         0.20668
  validation_logloss validation_r2 validation_auc validation_pr_auc validation_lift
1                 NA            NA             NA                NA              NA
2                 NA            NA             NA                NA              NA
3                 NA            NA             NA                NA              NA
4                 NA            NA             NA                NA              NA
5            0.20681       0.82245        0.99494           0.99149         2.48148
  validation_classification_error
1                              NA
2                              NA
3                              NA
4                              NA
5                         0.02239

Thank you again for helping me with the GLM.

[tomasfryda]

@sebhrusen @ledell I went through the algos and created a mapping between stopping criteria (specified in documentation) and columns present in scoring_history. I couldn't find some stopping criteria, namely mean_per_class_error, MSE, RMSLE. I added some "metadata" after / to note when the criterium is present in the scoring history.

[{
    "mean_per_class_error": {},
    "MSE": {},
    "RMSLE": {},
    "anomaly_score": {
      "IsolationForest": ["mean_anomaly_score"]
    },
    "custom": {
      "GBM":["training_custom", "validation_custom"],
      "DRF":["training_custom", "validation_custom"]
    },
    "custom_increasing": {
      "GBM":["training_custom", "validation_custom"],
      "DRF":["training_custom", "validation_custom"]
    },
    "deviance": {
      "GLM/lambda_search": ["deviance_train", "deviance_test", "deviance_xval", "deviance_se"],
      "DRF/regression": ["training_deviance", "validation_deviance"],
      "GBM/regression": ["training_deviance", "validation_deviance"],
      "DeepLearning/regression": ["training_deviance", "validation_deviance"],
      "XGBoost/regression": ["training_deviance", "validation_deviance"],
    },
    "logloss/binomial,multinomial": {
      "DeepLearning": ["training_logloss", "validation_logloss"],
      "DRF": ["training_logloss", "validation_logloss"],
      "GBM": ["training_logloss", "validation_logloss"],
      "XGBoost": ["training_logloss", "validation_logloss"]
    },
    "RMSE/binomial,multinomial,regression": {
      "DeepLearning": ["training_rmse", "validation_rmse"],
      "DRF": ["training_rmse", "validation_rmse"],
      "GBM": ["training_rmse", "validation_rmse"],
      "XGBoost": ["training_rmse", "validation_rmse"]
    },
    "MAE/regression": {
      "DRF": ["training_mae", "validation_mae"],
      "GBM": ["training_mae", "validation_mae"],
      "DeepLearning": ["training_mae", "validation_mae"],
      "XGBoost": ["training_mae", "validation_mae"]
    },
    "AUC/binomial, + opt-in in multinomial": {
      "DeepLearning": ["training_auc", "validation_auc"],
      "DRF": ["training_auc", "validation_auc"],
      "GBM": ["training_auc", "validation_auc"],
      "XGBoost": ["training_auc", "validation_auc"]
    },
    "AUCPR/binomial, + opt-in in multinomial": {
      "DeepLearning": ["training_pr_auc", "validation_pr_auc"],
      "DRF": ["training_pr_auc", "validation_pr_auc"],
      "GBM": ["training_pr_auc", "validation_pr_auc"],
      "XGBoost": ["training_pr_auc", "validation_pr_auc"]
    },
    "lift_top_group/binomial": {
      "DeepLearning": ["training_lift", "validation_lift"],
      "DRF": ["training_lift", "validation_lift"],
      "GBM": ["training_lift", "validation_lift"],
      "XGBoost": ["training_lift", "validation_lift"]
    },
    "misclassification/binomial,multinomial": {
      "DeepLearning": ["training_classification_error", "validation_classification_error"],
      "DRF": ["training_classification_error", "validation_classification_error"],
      "GBM": ["training_classification_error", "validation_classification_error"],
      "XGBoost": ["training_classification_error", "validation_classification_error"]
    }
  },
  "Not a stopping criterium(based on docs) but present in scoring history": {
  "DeepLearning" : ["training_r2", "validation_r2"],
  "CoxPH": ["loglik"],
  "IsolationForest": ["mean_tree_path_length"],
  "GLM": ["objective", "convergence", "negative_log_likelihood", "sum(etai-eta0)^2"],
  "PCA": ["objective"]
  }]

Should I use only the names from stopping criteria (misclassification) or should I allow user specify both (misclassification or classification_error)?

[tomasfryda]

FYI I created a separate PR for the GLM fix: #5191 (merged now)

[ledell]

We discussed this on a call, but I am summarizing the requested changes here:

1. New plot label names:

• Training
• Training (CV Models)
• Validation
• Cross-validation

2. Support lower and upper versions of names (e.g. tolower() in R), with the official version being lower case. We should consider whether to make all our metric names default to lower case... something to think about for another day.

> h2o.learning_curve_plot(gbm, metric = "AUC")
Error in match.arg(metric) : 
  'arg' should be one of “AUTO”, “auc”, “aucpr”, “mae”, “rmse”, “anomaly_score”, “convergence”, “custom”, “custom_increasing”, “deviance”, “lift_top_group”, “logloss”, “misclassification”, “negative_log_likelihood”, “objective”, “sumetaieta02”

3. Remove the suffix of the AutoML model IDs (for plotting purposes) so that it's easier to read.

4. Change cutoff line color to red or green.

5. Add newline after "Selected" plot label for more space (let's see if this looks better or worse).

6. Notes about GLM differences:

• deviance (lambda search), iteration
• objective (no lambda search), iterations

The GLM scoring history missing some metrics in GLM, and also in the case of no lambda search, we don't have the cross-validation metrics either. Can we add all of this? Let's discuss in #dev-h2o-3 to see if we can fill in the missing pieces in GLM so we can have a more unified plotting experience (especially since we are using GLM metalearner as the curve we plot for Stacked Ensembles).

[tomasfryda]

I made the modification (1-5) and here are the results:

Python

R

[tomasfryda]

New doc string here. Python version is basically the same except for NULL -> None

[tomasfryda]

scoring_iteration_interval might change depending on benchmark results.

When _valid is specified, we use lambda search which provides plenty of information for the learning curve. Otherwise, lambda search is off and the information in scoring history is often just one or two points entries. Even metalearner trained on Airlines dataset subset with 250k rows has less than 10 iterations so even this score_iteration_iterval might be too big.

[tomasfryda]

With benchmark results from @wendycwong we decided to keep the value to 5 as it has less than 2% performance impact but it improves the learning curve in some situations significantly. This affects only the AUTO metalearner in SE.

[michalkurka]

does cloning the object not work for this use case? it seems like you are just copying the table

[michalkurka]

would be nice to generalize it, instead of having this enumerated add "an interface" that the algo would implement

this would let us get rid of the big if that determines allowed_metrics and allowed_timesteps