Actual ntrees value in tree based models is incorrect when using train/valid for early stopping

[exalate-issue-sync]

There's a bug where if you set nfolds = 0, and turn on early stopping in any tree based model (GBM, DRF, XGBoost), the reported actual value for the ntrees parameters are incorrect. This is supposed to be updated with the final/found/actual value, but for some reason it’s just replacing it with the “input” value.

When you have CV turned on (and use CV for early stopping), it will correctly report the "actual" (found) value for ntrees (which can be confirmed by looking at model summary with actual # of trees). However, if you turn off CV, then it incorrectly reports the input value as the “actual” value. Bug can be reproduced from the attached notebook.

This is only currently a bug in Python right now because there's a [https://h2oai.atlassian.net/browse/PUBDEV-8051|https://h2oai.atlassian.net/browse/PUBDEV-8051|smart-link] where we are not storing the actual value at all, so that's currently unaffected (but we need to fix that ASAP as well).

[h2o-ops-ro]

JIRA Issue Details

Jira Issue: PUBDEV-8531
Assignee: Adam Valenta
Reporter: Erin LeDell
State: Resolved
Fix Version: 3.40.0.1
Attachments: Available (Count: 5)
Development PRs: Available

[h2o-ops-ro]

Attachments From Jira

Attachment Name: H2O-3_AutoML_Starter.html
Attached By: Erin LeDell
File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-8531/H2O-3_AutoML_Starter.html

Attachment Name: H2O-3_AutoML_Starter.ipynb
Attached By: Erin LeDell
File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-8531/H2O-3_AutoML_Starter.ipynb

Attachment Name: Screen Shot 2022-01-17 at 6.53.37 PM.png
Attached By: Erin LeDell
File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-8531/Screen Shot 2022-01-17 at 6.53.37 PM.png

Attachment Name: Screen Shot 2022-01-17 at 6.53.48 PM.png
Attached By: Erin LeDell
File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-8531/Screen Shot 2022-01-17 at 6.53.48 PM.png

Attachment Name: Screen Shot 2022-01-17 at 7.08.41 PM.png
Attached By: Erin LeDell
File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-8531/Screen Shot 2022-01-17 at 7.08.41 PM.png

[h2o-ops-ro]

Linked PRs from JIRA

#6476
#6493
#6660