Add ability to specify algo parameters (e.g. monotonicity constraints) in H2O AutoML #4005

sebhrusen · 2019-10-22T20:21:05Z

https://0xdata.atlassian.net/browse/PUBDEV-6737

Current implementation restricts the parameters that can be assigned by the user to:

monotone_constraints as required by initial customer request.

Added a System property to allow overriding of any parameter: the plan is to use this for development/benchmarking.
Tuning algorithms, and running benchmarks against versions with different parameters is currently a hassle (recompile, rename built jar, upload it, keep reference...).
Having those parameters configurable from client will allow us to try more combinations faster (may also support hyperparameters override in the future for benchmarking as well).

h2o-automl/src/main/java/ai/h2o/automl/modeling/GBMStepsProvider.java

h2o-core/src/main/java/water/util/JSONValue.java

ledell · 2019-10-23T23:09:12Z

h2o-automl/src/main/java/ai/h2o/automl/AutoMLBuildSpec.java

+      stopping_criteria = new AutoMLStoppingCriteria();
+
+      // reasonable defaults:
+      stopping_criteria.set_max_models(0);


Is -1 or 0 more common as a replacement for None/NULL for an integer-valued parameter on the backend? I have seen both... (which I guess means we can choose whichever one is "better"?).

@ledell I didn't change that code, just moved it down: usually in Java, attributes are declared before constructor, I was just applying this "norm" here.

Regarding the -1/0 debate... there's no rule written in stone. -1 is commonly used for "unset" (although Java assigns 0 by default to int variables), and when there's no ambiguity like here, I tend to use 0 for "unlimited". Maybe it deserves a tiny comment next to it.

h2o-r/h2o-package/R/automl.R

ledell · 2019-10-23T23:19:58Z

@seb-h2o Can you update the description to remove ntrees and to add algo_parameters instead?

My concern with algo_parameters is that if someone only wants to limit ntrees in GBMs, for example, there is no way to limit that to just GBMs. Since DRF and XGBoost also has an ntrees parameter, it will also limit it there too (from my understanding of your current implementation). I guess the only way to give further precision is to have it be a nested dict/list:

algo_parameters = {"GBM": {"ntrees": 1000, "learn_rate": 0.01}, "DRF": {"ntrees": 200}}

In general, GBMs require more (usually more shallow) trees than Random Forest, so I think it makes sense to allow the user to be precise in their configurations.

deil87

Thanks Seb, great feature!

h2o-automl/src/main/java/ai/h2o/automl/AutoMLBuildSpec.java

deil87 · 2019-10-24T11:33:54Z

h2o-automl/src/main/java/ai/h2o/automl/AutoMLBuildSpec.java

+        if (!addParameter(param.algo, param.name, param.value))
+          throw new H2OIllegalValueException(param.name, ROOT_PARAM, param.value);
+      }
+      return this;


@seb-h2o usually build pattern returns different type so that we can't do chain calls anymore( SmthBuilder.add().add().build() -> Smth). With current approach we maybe don't need to return this at all.

Yeah, I hesitated into turning this into a proper "builder" (that's also why I renamed the build method).
Mainly trying to keep it simple: not sure builders are justified most of the time.
When I see

AutoMLCustomParameters algo_parameters;

my reflex is:

algo_parameters = new AutoMLCustomParameters();

not: "damn, constructor is private, any factory method? AutoMLCustomParameters.newInstance(...)? damn, what then? Oh found it new AutoMLCustomParametersBuilder()....build()"

So trying to combine the best of both:

algo_parameters = new AutoMLCustomParameters().add(...) .add(...) .end();

It's not without flaws though, conscious about this. Will reconsider and try to find sth more satisfying.

ended up with AutoMLCustomParameters.create().add(...).build()
create returns an internal Builder instance`

deil87 · 2019-10-24T11:43:34Z

h2o-automl/src/main/java/ai/h2o/automl/AutoMLBuildSpec.java

+      return algo_parameter_names.get(algo.name());
+    }
+
+    public Model.Parameters getCustomParameters(Algo algo) {


as we actually get both default and custom parameters here.... naming is slightly confusing. Maybe getCustomisedDefaults?

Not sure this should be public anymore actually, which would make the naming less important.
getCustomDefaults has something oxymoronic in it that makes me like it somehow :)

renamed, and lowered visibility (used in tests)

h2o-automl/src/main/java/ai/h2o/automl/AutoMLBuildSpec.java

h2o-automl/src/main/java/water/automl/api/schemas3/AutoMLBuildSpecV99.java

h2o-core/src/main/java/water/api/SchemaMetadata.java

h2o-r/tests/testdir_algos/automl/runit_automl_model_selection.R

sebhrusen · 2019-10-24T13:07:39Z

Can you update the description to remove ntrees and to add algo_parameters instead?

@ledell you mean in the JIRA ticket?

regarding the implementation, it is actually possible to set params for a specific algo using the following syntax (works both in R and Py):

algo_parameters = dict(
    ntrees=200,    # applies to all algos supporting ntrees
    GBM__ntrees=1000 .   # applies only to GBMs
)

Anyway, as soon as we're not sure about exposing ntrees, waiting for discussion with customer, I'll keep this feature hidden.
Remains the question of monotone_constraints: do we want to have the param at top level as Michal suggests?

ledell · 2019-10-24T15:52:26Z

@ledell you mean in the JIRA ticket?

@seb-h2o No I mean in this PR description (it still says ntrees).

Thanks for clarifying about how to apply the parameter to a particular algorithm. I remember seeing this algoname__paramname format before (was that here or do we use this format in another part of H2O?).

Can you list the strings that you accept for each algorithm? I want to make sure we are using consistent naming of the algorithms... I filed this JIRA recently because I noticed that H2O has several sets of algorithm names. It would be great to unify this across H2O moving forward.

Regarding algo_parameters in general -- I know that we had planned to allow users to fully customize the AutoML "plan" at some point (we had this on a roadmap, but it's not a JIRA yet). Would the algo_parameters parameter become redundant or conflicting at that point, or do you think that it will still provide value? I wonder how often people will want to fix a particular hyperparameter value across all the models, since the goal in AutoML is usually to try many values for a single hyperparameter...instead of just one. Just thinking out-loud.

sebhrusen · 2019-10-24T19:03:22Z

@ledell

Thanks for clarifying about how to apply the parameter to a particular algorithm. I remember seeing this algoname__paramname format before (was that here or do we use this format in another part of H2O?).

It is the "standard" flat format used by scikit learn to specify nested parameters: I must say that I like it. When we have the possibility to avoid an additional level of depth without possibility of confusion, it brings only clarity I think.

Can you list the strings that you accept for each algorithm? I want to make sure we are using consistent naming of the algorithms...

AutoML is using the algo names listed in https://github.com/h2oai/h2o-3/blob/master/h2o-automl/src/main/java/ai/h2o/automl/Algo.java everywhere.
I also noticed that the parsing logic we use to extract params from the JSON body is case-sensitive with enums, meaning we usually can't use gbm but we have to use GBM.
I think we should be more lenient here and accept both: it's a very easy fix.

Regarding algo_parameters in general - I know that we had planned to allow users to fully customize the AutoML "plan" at some point

For me algo_parameters is useful when user wants to set 1 or 2 specific constraints on algos, not to define complete models, at least not at end API level (tbh, when exposing this feature, I've been wondering if we could not create a model designer on top of it, only client side: it would be easy to have a "custom steps" in Java that got only fed by some of those params, internally all "custom___xxx" params for example...), also thinking out loud.

But overall, I think the idea of full customization should be thought like DAI recipes, with a mini-language describing additional models, grids, feature engineering...
This needs to be designed top down.

This being said, after disabling ntrees, there will be only monotone_constraints allowed by default (as said in the description, I keep the possibility to fully enable the feature for dev/benchmark through a java / h2o.init parameter).
So the question remains if we want to keep monotone_constraints here only, or if expose it as a 1st level param in the API.

Co-Authored-By: Erin LeDell <erin@h2o.ai>

sebhrusen · 2019-10-29T12:53:56Z

@ledell As all concerns on this PR have been resolved, I'm merging it.
See additional PR #4020 for the decision to expose monotone_constraints as a top level param in AutoML.

…) in H2O AutoML (h2oai#4005) * Java API for custom params * specs for custom params * move some logic from schema to BuildSpec * Py support for algo_parameters * removed useless schema from AutoMLBuildSpec * improved parsing logic for REST API polymorphic params * R client support * disabled ntrees, improved tests and addressed PR comments

sebhrusen commented Oct 22, 2019

View reviewed changes

h2o-automl/src/main/java/ai/h2o/automl/modeling/GBMStepsProvider.java Show resolved Hide resolved

sebhrusen requested review from deil87 and ledell October 22, 2019 20:24

sebhrusen commented Oct 22, 2019

View reviewed changes

h2o-core/src/main/java/water/util/JSONValue.java Show resolved Hide resolved

sebhrusen added WIP please review and removed WIP labels Oct 22, 2019

ledell reviewed Oct 23, 2019

View reviewed changes

h2o-r/h2o-package/R/automl.R Show resolved Hide resolved

ledell reviewed Oct 23, 2019

View reviewed changes

h2o-r/h2o-package/R/automl.R Outdated Show resolved Hide resolved

deil87 approved these changes Oct 24, 2019

View reviewed changes

Sebastien Poirier and others added 15 commits October 25, 2019 19:21

Java API for custom params

0ac8a07

specs for custom params

d821a8c

move some logic from schema to BuildSpec

849506d

Py support for algo_parameters

c13aa1c

removed useless schema from AutoMLBuildSpec

d911a41

improved parsing logic for REST API polymorphic params

f4d0dfc

more tests

12ed92e

minor reformatting

d484283

removed unused imports

6db3d0b

removed unused imports

ab23505

restrict import scope in AutoMLBuildSpec

f8d142f

R client support

d8ef104

added short description of JSONVAlue class

0f18903

Update h2o-r/h2o-package/R/automl.R

08af4f8

Co-Authored-By: Erin LeDell <erin@h2o.ai>

disabled ntrees, improved tests and addressed PR comments

7281e40

sebhrusen force-pushed the seb_pubdev-6737 branch from 3cae0d3 to 7281e40 Compare October 25, 2019 17:21

removing reference to ntrees from client doc

0eca2f5

sebhrusen mentioned this pull request Oct 25, 2019

PUBDEV-6737: expose monotone_constraints as a top level parameter in AutoML #4020

Merged

sebhrusen merged commit 97d757c into master Oct 29, 2019

sebhrusen deleted the seb_pubdev-6737 branch March 16, 2020 17:34

h2o-ops mentioned this pull request May 14, 2023

Add ability to specify algo parameters (e.g. monotonicity constraints) in H2O AutoML #8896

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to specify algo parameters (e.g. monotonicity constraints) in H2O AutoML #4005

Add ability to specify algo parameters (e.g. monotonicity constraints) in H2O AutoML #4005

sebhrusen commented Oct 22, 2019 •

edited

Loading

ledell Oct 23, 2019

sebhrusen Oct 24, 2019 •

edited

Loading

ledell commented Oct 23, 2019 •

edited

Loading

deil87 left a comment

deil87 Oct 24, 2019

sebhrusen Oct 24, 2019

sebhrusen Oct 25, 2019

deil87 Oct 24, 2019

sebhrusen Oct 24, 2019

sebhrusen Oct 25, 2019

sebhrusen commented Oct 24, 2019 •

edited

Loading

ledell commented Oct 24, 2019 •

edited

Loading

sebhrusen commented Oct 24, 2019 •

edited

Loading

sebhrusen commented Oct 29, 2019

Add ability to specify algo parameters (e.g. monotonicity constraints) in H2O AutoML #4005

Add ability to specify algo parameters (e.g. monotonicity constraints) in H2O AutoML #4005

Conversation

sebhrusen commented Oct 22, 2019 • edited Loading

ledell Oct 23, 2019

Choose a reason for hiding this comment

sebhrusen Oct 24, 2019 • edited Loading

Choose a reason for hiding this comment

ledell commented Oct 23, 2019 • edited Loading

deil87 left a comment

Choose a reason for hiding this comment

deil87 Oct 24, 2019

Choose a reason for hiding this comment

sebhrusen Oct 24, 2019

Choose a reason for hiding this comment

sebhrusen Oct 25, 2019

Choose a reason for hiding this comment

deil87 Oct 24, 2019

Choose a reason for hiding this comment

sebhrusen Oct 24, 2019

Choose a reason for hiding this comment

sebhrusen Oct 25, 2019

Choose a reason for hiding this comment

sebhrusen commented Oct 24, 2019 • edited Loading

ledell commented Oct 24, 2019 • edited Loading

sebhrusen commented Oct 24, 2019 • edited Loading

sebhrusen commented Oct 29, 2019

sebhrusen commented Oct 22, 2019 •

edited

Loading

sebhrusen Oct 24, 2019 •

edited

Loading

ledell commented Oct 23, 2019 •

edited

Loading

sebhrusen commented Oct 24, 2019 •

edited

Loading

ledell commented Oct 24, 2019 •

edited

Loading

sebhrusen commented Oct 24, 2019 •

edited

Loading