PUBDEV-5975: Proposal for a consistent behaviour of AutoML reruns

[sebhrusen]

https://0xdata.atlassian.net/browse/PUBDEV-5975

Due to the inherent issues with the current Python API for AutoML, user can obtain surprising results when looking at the leaderboard when doing reruns in AutoML, especially in the following cases:

• when running multiple aml.train on a single aml (AutoML) instance.
• when creating multiple AutoML instances without any project_name.

On backend side, it seems that it was also designed with a feature in mind that was never completed: the notion of project distinct from the automl_id. But it's very difficult to guess what designers had in mind as there is not true support for the concept of a project that would include multiple AutoML instances for example.

The contract for this proposal is detailed in the pyunit_automl_reruns.py test suite, so I encourage to look at it first.
To sum it up, the idea is:

• to always create a new unique project (and therefore leaderboard), each time the user creates and AutoML instance without specifying the project_name.
• to allow cumulative reruns (keep adding new models to the leaderboard), if user is calling train multiple times on the same project name, with compatible data (same training_frame, same response column).
• to allow reruns but with a new leaderboard, if user is changing the response column or the training_frame (previous leaderboard is still accessible by id).

More issues

https://0xdata.atlassian.net/browse/PUBDEV-6708

This issue is due to reruns as well.
Current behaviour allows changing the leaderboard_frame and the new models will still be appended to the existing leaderboard (ignoring the new frame), this is just plain wrong!

The leaderboard should be identified uniquely by project_name + leaderboard_frame, making this rerun logic still more complicated.

Rerun behaviour contract

see https://github.com/h2oai/h2o-3/pull/3907/files#diff-1281d5db9141adc08c3047885255f970

[sebhrusen]

@ledell will probably loosen the constraint on training_frame now that I fixed PUBDEV-6494 (#3910), and will create a new leaderboard iff response_column has changed, allowing the user to add new models to the leaderboard even when passing a different training_frame to the train method.

[ledell]

Great improvement. LGTM.