Allow Grid Search to Continue After Cluster Restart

[exalate-issue-sync]

Allow grid search to continue in a new cluster, taking into account models previously built in another cluster (one that may have crashed).

For example, the grid search object could save checkpoint details to an HDFS directory, and then continue the grid search where it left off after a cluster restart.

While a cluster is running, H2O-3 currently has the ability to restart and skip building models that were already build, but not once the cluster is shutdown.

[exalate-issue-sync]

trushant.kalyanpur commented: #94981 (https://support.h2o.ai/a/tickets/94981) - Feature Request - saving the hyperparameters and metrics

[exalate-issue-sync]

Pavel Pscheidl commented: Is there a Jira for it [~accountid:5d1185d2b020930c9705ec5c] ? Because this Jira is definitely not about hyperparam saving per se.

[exalate-issue-sync]

trushant.kalyanpur commented: Please ignore. I was testing Jira + Freshdesk integration that was broken and it is fixed again now. I had unlinked the issue but it looks like it shows up as a comment.

[h2o-ops]

JIRA Issue Migration Info

Jira Issue: PUBDEV-6749
Assignee: Pavel Pscheidl
Reporter: Lauren DiPerna
State: Resolved
Fix Version: 3.28.0.1
Attachments: N/A
Development PRs: Available

Linked PRs from JIRA

#3919
#3962