[Tune] Add `HEBOSearch` Searcher #13863

Yard1 · 2021-02-02T19:47:28Z

Why are these changes needed?

This PR adds a Tune Searcher using HEBO (Heteroscedastic Evolutionary Bayesian Optimization). HEBO is a cutting edge black-box optimization algorithm created by Huawei's Noah Ark's lab. It was the winning submission to NeurIPS 2020 Black-Box Optimisation Challenge which included real-life objective functions, such as tuning performance of standard machine learning models on real data sets.

As of now, this PR is a draft due to some minor issues in HEBO implementation that prevent it from offering the same functionality as the rest of the algorithms included already in Tune, as well as due to lack of a PyPi package (installing from github causes an error due to fixable dependency conflicts). I am trying to get in contact with authors of HEBO in order to solve those issues. The code in this PR has full functionality.

Docs, examples and tests have been included, though CI may fail due to issues described above.

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Yard1 · 2021-02-02T19:48:00Z

CC @krfricke @richardliaw

krfricke

Generally this looks great - thanks! Left a couple of minor comments.

Also, the tests are currently failing:

E       ModuleNotFoundError: No module named 'bo'

And

E       AssertionError: HEBO must be installed!. You can install HEBO with the command: `pip install git+https://github.com/huawei-noah/noah-research.git#subdirectory=HEBO`.

Maybe @amogkam can help with how to include this into the tune dependencies?

python/ray/tune/examples/hebo_example.py

python/ray/tune/suggest/hebo.py

krfricke · 2021-02-10T13:13:40Z

python/ray/tune/suggest/hebo.py

+def _validate_warmstart(parameter_names: List[str],
+                        points_to_evaluate: List[Union[List, Dict]],
+                        evaluated_rewards: List):
+    if points_to_evaluate:
+        if not isinstance(points_to_evaluate, list):
+            raise TypeError(
+                "points_to_evaluate expected to be a list, got {}.".format(
+                    type(points_to_evaluate)))
+        for point in points_to_evaluate:
+            if not isinstance(point, (dict, list)):
+                raise TypeError(
+                    f"points_to_evaluate expected to include list or dict, "
+                    f"got {point}.")
+
+            if not len(point) == len(parameter_names):
+                raise ValueError("Dim of point {}".format(point) +
+                                 " and parameter_names {}".format(
+                                     parameter_names) + " do not match.")
+
+    if points_to_evaluate and evaluated_rewards:
+        if not isinstance(evaluated_rewards, list):
+            raise TypeError(
+                "evaluated_rewards expected to be a list, got {}.".format(
+                    type(evaluated_rewards)))
+        if not len(evaluated_rewards) == len(points_to_evaluate):
+            raise ValueError(
+                "Dim of evaluated_rewards {}".format(evaluated_rewards) +
+                " and points_to_evaluate {}".format(points_to_evaluate) +
+                " do not match.")


This looks similar (identical?) to the one we use in skopt. Should we just move this either to utils or to the base searcher class and use the same function then? We might want to use this for other searchers in the futures as well.

It is indeed identical. We could make it a part of the Searcher class, I think.

@krfricke Should I make this change in this PR?

Yep, that would be great!

python/ray/tune/suggest/hebo.py

krfricke · 2021-02-10T13:23:43Z

python/ray/tune/suggest/hebo.py

+            self._metric_op = -1.
+        elif self._mode == "min":
+            self._metric_op = 1.
+


We should check here that self._space is not empty (apparently this check seems to be missing from some other searchers?)

I copied it from one of the existing ones, so it would appear so. I'll add it here, though the rest should probably be fixed in another PR.

@krfricke Should it throw an exception if self._space is empty?

Yes - it's enough to do it for this Searcher here. We'll tackle the other ones somewhere else

python/ray/tune/suggest/hebo.py

Co-authored-by: Kai Fricke <krfricke@users.noreply.github.com>

Yard1 · 2021-02-10T13:59:35Z

As for the whole dependency thing - I am still waiting for any response from the principal author (made a PR on github and sent an email). If that doesn't come soon and considering that HEBO is released under MIT license, I think the best way would be for me to create my own fork, sort out those small issues, and upload it to PyPi (with full notice of authorship), thus removing the need to install from git. I'd be happy to transfer ownership of the package to the principal author should they prefer that too.

krfricke · 2021-02-11T13:01:56Z

Sounds good to me!

Yard1 · 2021-02-12T22:44:56Z

I have uploaded the package to PyPi and updated requirements_tune.in but I am not sure what's the right way to generate txt requirements for different Python versions. @krfricke @amogkam what would be the process here for adding a dependency?

amogkam · 2021-02-12T23:41:06Z

Hey @Yard1 we actually changed the dependency stuff very recently. If you merge master, you should then just have to add the new dependency to requirements_tune.txt in alphabetical order.

Yard1 · 2021-02-13T17:47:56Z

Most of the test failures seem to be unrelated to this PR.

krfricke

Almost ready, the changes should be easy to do

krfricke · 2021-02-15T15:11:28Z

python/ray/tune/examples/hebo_example.py

+    # The config will be automatically converted to SkOpt's search space
+
+    # Optional: Pass the parameter space yourself
+    # space = {
+    #     "width": (0, 20),
+    #     "height": (-100, 100),
+    #     "activation": ["relu", "tanh"]
+    # }


nit: this comment refers to SkOpt.

Related: Is this commented-out space definition actually compatible with HEBO?

No, it's not compatible. I'll put in the actual design space example there.

krfricke · 2021-02-15T15:11:53Z

python/ray/tune/suggest/hebo.py

+def _validate_warmstart(parameter_names: List[str],
+                        points_to_evaluate: List[Union[List, Dict]],
+                        evaluated_rewards: List):
+    if points_to_evaluate:
+        if not isinstance(points_to_evaluate, list):
+            raise TypeError(
+                "points_to_evaluate expected to be a list, got {}.".format(
+                    type(points_to_evaluate)))
+        for point in points_to_evaluate:
+            if not isinstance(point, (dict, list)):
+                raise TypeError(
+                    f"points_to_evaluate expected to include list or dict, "
+                    f"got {point}.")
+
+            if not len(point) == len(parameter_names):
+                raise ValueError("Dim of point {}".format(point) +
+                                 " and parameter_names {}".format(
+                                     parameter_names) + " do not match.")
+
+    if points_to_evaluate and evaluated_rewards:
+        if not isinstance(evaluated_rewards, list):
+            raise TypeError(
+                "evaluated_rewards expected to be a list, got {}.".format(
+                    type(evaluated_rewards)))
+        if not len(evaluated_rewards) == len(points_to_evaluate):
+            raise ValueError(
+                "Dim of evaluated_rewards {}".format(evaluated_rewards) +
+                " and points_to_evaluate {}".format(points_to_evaluate) +
+                " do not match.")


Yep, that would be great!

krfricke · 2021-02-15T15:12:46Z

python/ray/tune/suggest/hebo.py

+            self._metric_op = -1.
+        elif self._mode == "min":
+            self._metric_op = 1.
+


Yes - it's enough to do it for this Searcher here. We'll tackle the other ones somewhere else

krfricke

Just some suggestions to create better error messages. Let me know if you're happy with these suggestions. (please note that this might break the linter, so if you choose to commit the suggestions, pull locally, run the format script, and push again a final time)

In the latest check the HEBO search was flaky, but it seems that this is due to a general pytorch error? Can you take a quick look to confirm?

Otherwise this is great. Thank you very much for the contribution!

python/ray/tune/BUILD

python/ray/tune/suggest/hebo.py

Co-authored-by: Kai Fricke <krfricke@users.noreply.github.com>

Yard1 · 2021-02-17T17:24:57Z

@krfricke Thanks, I applied the suggestions and reformatted it. Not sure what caused the test to flake before but I had no errors when running it locally, so it could have just been a fluke.

krfricke · 2021-02-17T21:53:25Z

Awesome! Thanks again!

Yard1 added 4 commits February 1, 2021 23:36

HEBO first pass

f02ee1c

Fix bad quotes

bd01336

Fixes

00f9a8b

Reproductibility

f4cd9e8

Yard1 marked this pull request as draft February 2, 2021 19:47

krfricke self-assigned this Feb 2, 2021

krfricke reviewed Feb 10, 2021

View reviewed changes

Update python/ray/tune/suggest/hebo.py

7e9b2ca

Co-authored-by: Kai Fricke <krfricke@users.noreply.github.com>

Yard1 added 3 commits February 12, 2021 19:28

Add hebo_example.py to BUILD

c317b8c

Nit

69d88b3

Update to pypi package

da0b636

Yard1 added 3 commits February 13, 2021 12:16

Merge branch 'master' into tune-hebo-suggester

057c594

Alphabetical HEBO requirement

da3d4dc

Fix syntax error

a59ee27

krfricke reviewed Feb 15, 2021

View reviewed changes

Yard1 added 4 commits February 15, 2021 16:38

Fix wrong space in hebo example

6b85047

Move validate_warmstart to utils

315cf3a

Space assertion in HEBO

8752125

Comment

c680ee5

Yard1 marked this pull request as ready for review February 15, 2021 15:46

krfricke approved these changes Feb 17, 2021

View reviewed changes

python/ray/tune/BUILD Outdated Show resolved Hide resolved

python/ray/tune/suggest/hebo.py Outdated Show resolved Hide resolved

python/ray/tune/suggest/hebo.py Outdated Show resolved Hide resolved

Yard1 and others added 2 commits February 17, 2021 18:22

Apply suggestions from code review

e7f76e8

Co-authored-by: Kai Fricke <krfricke@users.noreply.github.com>

Formatting

09f68ff

krfricke merged commit 58d7398 into ray-project:master Feb 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tune] Add `HEBOSearch` Searcher #13863

[Tune] Add `HEBOSearch` Searcher #13863

Yard1 commented Feb 2, 2021 •

edited

Yard1 commented Feb 2, 2021

krfricke left a comment

krfricke Feb 10, 2021

Yard1 Feb 10, 2021

Yard1 Feb 12, 2021

krfricke Feb 15, 2021

krfricke Feb 10, 2021

Yard1 Feb 10, 2021

Yard1 Feb 12, 2021

krfricke Feb 15, 2021

Yard1 commented Feb 10, 2021 •

edited

krfricke commented Feb 11, 2021

Yard1 commented Feb 12, 2021

amogkam commented Feb 12, 2021

Yard1 commented Feb 13, 2021

krfricke left a comment

krfricke Feb 15, 2021

Yard1 Feb 15, 2021

krfricke Feb 15, 2021

krfricke Feb 15, 2021

krfricke left a comment

Yard1 commented Feb 17, 2021

krfricke commented Feb 17, 2021

[Tune] Add HEBOSearch Searcher #13863

[Tune] Add HEBOSearch Searcher #13863

Conversation

Yard1 commented Feb 2, 2021 • edited

Why are these changes needed?

Checks

Yard1 commented Feb 2, 2021

krfricke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yard1 commented Feb 10, 2021 • edited

krfricke commented Feb 11, 2021

Yard1 commented Feb 12, 2021

amogkam commented Feb 12, 2021

Yard1 commented Feb 13, 2021

krfricke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krfricke left a comment

Choose a reason for hiding this comment

Yard1 commented Feb 17, 2021

krfricke commented Feb 17, 2021

[Tune] Add `HEBOSearch` Searcher #13863

[Tune] Add `HEBOSearch` Searcher #13863

Yard1 commented Feb 2, 2021 •

edited

Yard1 commented Feb 10, 2021 •

edited