Add Ax plugin for Hydra #321

shagunsodhani · 2019-12-10T02:29:00Z

Motivation

Hydra Plugin to interface with Adaptive Experimentation Platform

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

To be added

Related Issues and PRs

#291

omry · 2019-12-10T12:46:16Z

plugins/hydra_ax/README.md

+pip install hydra-ax
+```
+
+We include an example of how to use this plugin. The file `plugins/hydra_ax/example/banana.py` implements the [Rosenbrock function (aka Banana function)](https://en.wikipedia.org/wiki/Rosenbrock_function). The return value of the function should be the value that we want to optimize.


Remember to link this to the actual file in GitHub once this lands.

plugins/hydra_ax/README.md

plugins/hydra_ax/example/banana.py

omry · 2019-12-10T12:53:13Z

plugins/hydra_ax/hydra_plugins/ax_sweeper/ax_sweeper.py

+Trial = NewType("Trial", List[str])
+
+# TODO: output directory is overwriting, job.num should be adjusted (depends on issue #284)
+# TODO: Support running multiple random seeds, aggregate mean and SEM


While chatting with the Ax team they suggested that maybe Ax would do a better job at this automatically. not sure I am convinced but worth discussing with them.

plugins/hydra_ax/hydra_plugins/ax_sweeper/ax_sweeper.py

plugins/hydra_ax/hydra_plugins/ax_sweeper/conf/hydra/sweeper/ax.yaml

plugins/hydra_ax/hydra_plugins/ax_sweeper/ax_sweeper.py

plugins/hydra_ax/setup.py

plugins/hydra_ax/tests/test_example_sweeper_plugin.py

omry · 2019-12-30T02:24:50Z

The next major version of Hydra will break compatibility (dropping Python <= 3.5, switching to OmegaConf 2.0 etc).
I may be a months or two before this version is ready.
The 0.11_branch will be the release branch for the current 0.11 release line.
in the mean time I suggest that you continue your development on top of the 0.11_branch to ensure it's working for people using the stable version.

omry · 2019-12-30T17:14:54Z

You can also just ignore this and continue working against master, testing at the end that the plugin actually works against 0.11 as well (I suspect it will).

shagunsodhani · 2019-12-30T22:30:32Z

@omry yeah I think I have most of the pieces in place now :)

omry · 2019-12-30T22:32:16Z

FYI:
I just landed the OmegaConf 2.0 upgrade, and I am going to start cleaning up Python 2.7 left overs now, as well as annotate the code and enable mypy.

omry · 2020-01-06T18:37:34Z

Master is now on OmegaConf 2.0 and Python >= 3.6.
New requirements are that all code passes mypy --strict . and isort for import sorting.
When you merge in, expect to spend some time type annotating the new code.

plugins/hydra_ax_sweeper/example/banana.py

The name of the module was different from the name of the plugin. This caused some tests to fail.

omry · 2020-01-12T22:37:08Z

no need to force push normally. you can just keep piling up the diffs. when it's all ready I can squash it at commit or you can have one last force push to clean things up.

Force pushing makes it hard for others to interact with your diff and hard for me to see what changed since the last time.

shagunsodhani · 2020-01-12T22:55:43Z

no need to force push normally. you can just keep piling up the diffs. when it's all ready I can squash it at commit or you can have one last force push to clean things up.

Force pushing makes it hard for others to interact with your diff and hard for me to see what changed since the last time.

Will take care :)

omry · 2020-01-15T01:00:40Z

Heads up:
There is some API cleanup in progress that will effect plugins.
See the examples plugins for updated examples. if you run into issues rebasing let me know.

* Use latest version of Ax * Fix incorrect package names * Add type hints * Remove all iterators

omry · 2020-01-25T18:06:51Z

Let's try to finish this?
What's left before we can land this to master?

shagunsodhani · 2020-01-25T18:08:00Z

Fixing the isort check. I fixed the other issues/comments.

omry · 2020-01-26T03:17:42Z

So we have two issues:
the lint issue on all systems, and the dependency problem on Windows.

omry

Getting there.

plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/ax_sweeper.py

plugins/hydra_ax_sweeper/README.md

plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/ax_sweeper.py

plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/conf/hydra/sweeper/ax.yaml

plugins/hydra_ax_sweeper/setup.py

plugins/hydra_ax_sweeper/tests/test_example_sweeper_plugin.py

lgtm-com · 2020-02-15T22:02:35Z

This pull request introduces 1 alert when merging 144e453 into af78b06 - view on LGTM.com

new alerts:

1 for Module imports itself

lgtm-com · 2020-02-16T20:40:28Z

This pull request introduces 1 alert when merging c08838d into df0b6a1 - view on LGTM.com

new alerts:

1 for Module imports itself

plugins/hydra_ax_sweeper/example/conf/config.yaml

plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/ax_sweeper.py

plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/conf/hydra/sweeper/ax.yaml

plugins/hydra_ax_sweeper/tests/test_example_sweeper_plugin.py

hydra/test_utils/ax_sweeper_plugin.py

hydra/test_utils/configs/ax_sweeper_plugin/banana.yaml

lgtm-com · 2020-02-17T01:34:32Z

This pull request introduces 2 alerts when merging 77e037d into df0b6a1 - view on LGTM.com

new alerts:

1 for Module imports itself
1 for Variable defined multiple times

lgtm-com · 2020-02-17T03:59:23Z

This pull request introduces 2 alerts when merging ece1aa5 into df0b6a1 - view on LGTM.com

new alerts:

1 for Module imports itself
1 for Variable defined multiple times

lgtm-com · 2020-02-17T04:18:02Z

This pull request introduces 2 alerts when merging dbecffc into df0b6a1 - view on LGTM.com

new alerts:

1 for Module imports itself
1 for Variable defined multiple times

lgtm-com · 2020-02-17T05:12:12Z

This pull request introduces 2 alerts when merging b961968 into df0b6a1 - view on LGTM.com

new alerts:

1 for Module imports itself
1 for Variable defined multiple times

hydra/test_utils/launcher_common_tests.py

hydra/test_utils/test_utils.py

plugins/examples/example_sweeper_plugin/tests/test_example_sweeper_plugin.py

plugins/hydra_ax_sweeper/example/conf/config.yaml

plugins/hydra_ax_sweeper/tests/banana.py

plugins/hydra_ax_sweeper/tests/config/banana.yaml

plugins/hydra_ax_sweeper/tests/test_ax_sweeper_plugin.py

lgtm-com · 2020-02-17T15:08:15Z

This pull request introduces 2 alerts when merging b56e4dd into df0b6a1 - view on LGTM.com

new alerts:

1 for Module imports itself
1 for Variable defined multiple times

plugins/examples/example_sweeper_plugin/tests/test_example_sweeper_plugin.py

plugins/hydra_ax_sweeper/tests/test_ax_sweeper_plugin.py

lgtm-com · 2020-02-20T00:04:43Z

This pull request introduces 2 alerts when merging 646aac9 into 45ba5e3 - view on LGTM.com

new alerts:

1 for Module imports itself
1 for Variable defined multiple times

plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/ax_sweeper.py

plugins/hydra_ax_sweeper/tests/test_ax_sweeper_plugin.py

lgtm-com · 2020-02-20T01:08:20Z

This pull request introduces 2 alerts when merging 89d31cc into 5f8d689 - view on LGTM.com

new alerts:

1 for Module imports itself
1 for Variable defined multiple times

omry · 2020-02-20T01:21:08Z

plugins/hydra_ax_sweeper/tests/test_ax_sweeper_plugin.py

+    return z
+
+
+def test_launched_jobs(sweep_runner: TSweepRunner,) -> None:  # noqa: F811


Add additional test covering:

command line only usage (no config).

mixed command line and config usage.

lgtm-com · 2020-02-20T14:42:31Z

This pull request introduces 1 alert when merging b9148d1 into 4106845 - view on LGTM.com

new alerts:

1 for Variable defined multiple times

lgtm-com · 2020-02-20T18:47:05Z

This pull request introduces 1 alert when merging a0546d2 into 4106845 - view on LGTM.com

new alerts:

1 for Variable defined multiple times

omry · 2020-02-22T16:05:06Z

let's close this one?

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 10, 2019

omry requested changes Dec 10, 2019

View reviewed changes

omry mentioned this pull request Dec 29, 2019

initial hydra_ax plugin #291

Closed

omry reviewed Jan 10, 2020

View reviewed changes

plugins/hydra_ax_sweeper/example/banana.py Outdated Show resolved Hide resolved

shagunsodhani added 2 commits January 12, 2020 14:24

Add Ax plugin for Hydra

6be35ec

Rename a module

de30805

The name of the module was different from the name of the plugin. This caused some tests to fail.

shagunsodhani added 4 commits January 18, 2020 11:39

Merge branch 'master' into shagun_ax_plugin

18d0d9a

Fix build errors

b56af5e

* Use latest version of Ax * Fix incorrect package names * Add type hints * Remove all iterators

Fix the build errors after merging the code

1b4b9c6

Resolve some previous comments

70d7c8d

shagunsodhani added 2 commits January 26, 2020 09:35

Merge branch 'master' into shagun_ax_plugin

9cd9a78

Create batches of trial without using generators

07435ac

omry requested changes Jan 27, 2020

View reviewed changes

omry mentioned this pull request Jan 31, 2020

[Feature Request] Range support in BasicSweeper #393

Closed

shagunsodhani and others added 5 commits February 1, 2020 14:42

Fix linting error

c89b393

moved lint to be first nox test

53687ba

pin master to OmegaConf 2.0.0rc4 until Structured configs branch lands

3a42c0b

Fix config files in hydra

e996cc5

Resolve some leftover comments

295d523

Add news item for the Ax plugin

144e453

Add test for Ax sweeper plugin

c08838d

omry requested changes Feb 16, 2020

View reviewed changes

Resolve some issues

77e037d

Fix broken tests

ece1aa5

Fix breaking tests

dbecffc

Fix breaking tests

b961968

omry reviewed Feb 17, 2020

View reviewed changes

Resolve comments

b56e4dd

omry reviewed Feb 17, 2020

View reviewed changes

plugins/examples/example_sweeper_plugin/tests/test_example_sweeper_plugin.py Show resolved Hide resolved

plugins/hydra_ax_sweeper/tests/test_ax_sweeper_plugin.py Outdated Show resolved Hide resolved

Fix breaking errors

646aac9

omry reviewed Feb 20, 2020

View reviewed changes

plugins/hydra_ax_sweeper/hydra_plugins/hydra_ax_sweeper/ax_sweeper.py Outdated Show resolved Hide resolved

plugins/hydra_ax_sweeper/tests/test_ax_sweeper_plugin.py Outdated Show resolved Hide resolved

Use Quadratic function instead of Banana function

89d31cc

omry reviewed Feb 20, 2020

View reviewed changes

Add additional tests

b9148d1

Remove OSX from setup.py

a0546d2

Replace an if condition by elif condition

ad58701

shagunsodhani closed this Feb 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Ax plugin for Hydra #321

Add Ax plugin for Hydra #321

shagunsodhani commented Dec 10, 2019

omry Dec 10, 2019

omry Dec 10, 2019

omry commented Dec 30, 2019

omry commented Dec 30, 2019

shagunsodhani commented Dec 30, 2019

omry commented Dec 30, 2019

omry commented Jan 6, 2020

omry commented Jan 12, 2020

shagunsodhani commented Jan 12, 2020

omry commented Jan 15, 2020

omry commented Jan 25, 2020

shagunsodhani commented Jan 25, 2020

omry commented Jan 26, 2020

omry left a comment

lgtm-com bot commented Feb 15, 2020

lgtm-com bot commented Feb 16, 2020

lgtm-com bot commented Feb 17, 2020

lgtm-com bot commented Feb 17, 2020

lgtm-com bot commented Feb 17, 2020

lgtm-com bot commented Feb 17, 2020

lgtm-com bot commented Feb 17, 2020

lgtm-com bot commented Feb 20, 2020

lgtm-com bot commented Feb 20, 2020

omry Feb 20, 2020

lgtm-com bot commented Feb 20, 2020

lgtm-com bot commented Feb 20, 2020

omry commented Feb 22, 2020

		return z


		def test_launched_jobs(sweep_runner: TSweepRunner,) -> None: # noqa: F811

Add Ax plugin for Hydra #321

Add Ax plugin for Hydra #321

Conversation

shagunsodhani commented Dec 10, 2019

Motivation

Have you read the Contributing Guidelines on pull requests?

Test Plan

Related Issues and PRs

omry Dec 10, 2019

Choose a reason for hiding this comment

omry Dec 10, 2019

Choose a reason for hiding this comment

omry commented Dec 30, 2019

omry commented Dec 30, 2019

shagunsodhani commented Dec 30, 2019

omry commented Dec 30, 2019

omry commented Jan 6, 2020

omry commented Jan 12, 2020

shagunsodhani commented Jan 12, 2020

omry commented Jan 15, 2020

omry commented Jan 25, 2020

shagunsodhani commented Jan 25, 2020

omry commented Jan 26, 2020

omry left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Feb 15, 2020

lgtm-com bot commented Feb 16, 2020

lgtm-com bot commented Feb 17, 2020

lgtm-com bot commented Feb 17, 2020

lgtm-com bot commented Feb 17, 2020

lgtm-com bot commented Feb 17, 2020

lgtm-com bot commented Feb 17, 2020

lgtm-com bot commented Feb 20, 2020

lgtm-com bot commented Feb 20, 2020

omry Feb 20, 2020

Choose a reason for hiding this comment

lgtm-com bot commented Feb 20, 2020

lgtm-com bot commented Feb 20, 2020

omry commented Feb 22, 2020