[PUBDEV-7914] grid with failing CV models would hang indefinitely

[honzasterba]

https://h2oai.atlassian.net/browse/PUBDEV-7914

[michalkurka]

Thank you. Moving the callback to the driver level resolves the issue with the callback being triggered for CV models as well as the user reported issue.

This fits the use case of parallel grid search. Do you see a need for the original more powerful API in the future - callback also for CV models (eg. releasing GPUs to make available to the next model builder)?

[honzasterba]

This fits the use case of parallel grid search. Do you see a need for the original more powerful API in the future - callback also for CV models (eg. releasing GPUs to make available to the next model builder)?

cv model build uses a different approach, but this API would be usable for that if we wanted to improve the performance of CV model builidng