AutoML: define simplified modeling plan when max_models is enabled

[exalate-issue-sync]

We should see {{max_models}} parameter as used mainly when user wants AutoML to be reproducible, it’s actually a requirement for this.

Another obvious requirement is that it’s not because user specifies {{max_models}} that we’re allowed to “waste” computation time.

Expected behaviour when user sets {{max_models = N}}:

• train exactly N base models.
• all base models are trained until convergence (except if {{max_runtime_per_model}} is also set explicitely).
• the {{N}} models are distributed between
  ** default models first (same order as default modeling plan)
  ** various grids
• no intermediate SE model trained
• 2 default SE are trained after all base models:
  ** Best of Each Algo, using GLM metalearner with lambda search.
  ** All Models, using GLM metalearner with lambda search.
• 1 monotonic SE trained if conditions are satisfied.
• if user ALSO specifies {{max_runtime_secs}}, then AutoML should stay MAINLY reproducible, which means that the time budget is used ONLY to interrupt the last step, not as a time constraint during individual model training. This also means that in this case, the last model interrupted by this time cap should be completely thrown away.

[h2o-ops]

JIRA Issue Details

Jira Issue: PUBDEV-8613
Assignee: Sebastien Poirier
Reporter: Sebastien Poirier
State: Resolved
Fix Version: 3.36.1.1
Attachments: N/A
Development PRs: Available

[h2o-ops]

Linked PRs from JIRA

#6116