Add Joblib Launcher for parallel execution (multi-process and multi-threaded) #392

jan-matthis · 2020-01-31T10:55:26Z

Motivation

Launcher for local parallel sweeps using Joblib.Parallel, as proposed in #357, and discussed with @omry.

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

Added tests following the example_launcher_plugin.

Related Issues and PRs

Implements #357

omry · 2020-01-31T19:18:48Z

Awesome!

Let's call this jobid_launcher.
Tell me when you are ready for code review.

jan-matthis · 2020-02-02T14:04:43Z

PR is ready for code review

As far as I can tell the CI failures should be unrelated to the plugin (?)

omry

Awesome! first round of review.

rebase on master, it should address the omegaconf related errors.
Address lint errors (nox -s lint is the easiest for now).
Add a news fragment
See my comments.

If you haven't already go through the contributing page in the website.

plugins/joblib_launcher/README.md

plugins/joblib_launcher/example/conf/hydra/launcher/example.yaml

plugins/joblib_launcher/example/my_app.py

plugins/joblib_launcher/hydra_plugins/joblib_launcher/conf/hydra/launcher/example.yaml

plugins/joblib_launcher/hydra_plugins/joblib_launcher/joblib_launcher.py

omry · 2020-02-03T21:26:27Z

About the sort lint failure, I saw the same with the ax plugin.
try to add jobid to the third party projects listed in isort.cfg.

plugins/joblib_launcher/README.md

plugins/joblib_launcher/hydra_plugins/joblib_launcher/joblib_launcher.py

plugins/joblib_launcher/tests/test_joblib_launcher.py

plugins/joblib_launcher/hydra_plugins/joblib_launcher/joblib_launcher.py

plugins/joblib_launcher/hydra_plugins/joblib_launcher/conf/hydra/launcher/joblib.yaml

plugins/joblib_launcher/hydra_plugins/joblib_launcher/joblib_launcher.py

plugins/joblib_launcher/setup.py

plugins/joblib_launcher/tests/test_joblib_launcher.py

lgtm-com · 2020-02-05T13:23:31Z

This pull request introduces 1 alert when merging 81a2616 into e19b96b - view on LGTM.com

new alerts:

1 for Module imports itself

lgtm-com · 2020-02-05T14:32:30Z

This pull request introduces 1 alert when merging 61a1f9a into e19b96b - view on LGTM.com

new alerts:

1 for Module imports itself

omry · 2020-02-12T01:49:31Z

plugins/hydra_joblib_launcher/README.md

-[__main__][INFO] - Process ID 14335 executing task 4 ...
-[__main__][INFO] - Process ID 14337 executing task 5 ...
-```
+See [website](https://hydra.cc/docs/plugins/joblib_launcher) for more information


Until 1.0.0 is released, the url will be:

https://hydra.cc/docs/next/plugins/joblib_launcher

website/versioned_sidebars/version-0.11-sidebars.json

lgtm-com · 2020-02-12T01:57:38Z

This pull request introduces 1 alert when merging 74eb284 into 0bdf355 - view on LGTM.com

new alerts:

1 for Module imports itself

omry · 2020-02-12T02:02:49Z

Run isort . at the root (and add the pre-commit hooks, see development).
Please rebase on top of master to address the previous failures (from code coverage going down).

lgtm-com · 2020-02-12T02:32:43Z

This pull request introduces 1 alert when merging 96744ec into 9f2cd9f - view on LGTM.com

new alerts:

1 for Module imports itself

lgtm-com · 2020-02-12T08:34:49Z

This pull request introduces 1 alert when merging d86c242 into 9f2cd9f - view on LGTM.com

new alerts:

1 for Module imports itself

lgtm-com · 2020-02-12T09:56:15Z

This pull request introduces 1 alert when merging 3b9cc8a into 9f2cd9f - view on LGTM.com

new alerts:

1 for Module imports itself

lgtm-com · 2020-02-12T10:18:46Z

This pull request introduces 1 alert when merging 1d10c9b into 9f2cd9f - view on LGTM.com

new alerts:

1 for Module imports itself

lgtm-com · 2020-02-12T10:53:50Z

This pull request introduces 1 alert when merging febcf68 into 9f2cd9f - view on LGTM.com

new alerts:

1 for Module imports itself

lgtm-com · 2020-02-12T13:09:52Z

This pull request introduces 1 alert when merging 4a421f7 into 9f2cd9f - view on LGTM.com

new alerts:

1 for Module imports itself

Supresses error: Untyped decorator makes function "test_discovery" untyped

lgtm-com · 2020-02-12T13:31:10Z

This pull request introduces 1 alert when merging 9f92a1c into 9f2cd9f - view on LGTM.com

new alerts:

1 for Module imports itself

Using a different circleci python 3.6 Image for Linux to work around new issue

lgtm-com · 2020-02-12T13:56:41Z

This pull request introduces 1 alert when merging 7f2e75a into 9f2cd9f - view on LGTM.com

new alerts:

1 for Module imports itself

omry · 2020-02-12T15:19:07Z

Because you are cherry-picking from master it's hard to say what this diff is adding and what is new.
Can you rebase instead of cherry picking?

.circleci/config.yml

omry · 2020-02-12T15:17:04Z

website/docs/configure_hydra/logging.md

- 
-This example demonstrates how to to customize the logging behavior of your Hydra app, by making the following changes
+
+This example demonstrates how to customize the logging behavior of your Hydra app, by making the following changes


nice catch.

omry · 2020-02-12T15:20:43Z

And btw: awesome that tests are now passing! :)

omry · 2020-02-12T16:57:18Z

This is awesome work, thanks for putting so much time into it!

Add Parallel Launcher

3e83d0e

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 31, 2020

Renamed to JoblibLauncher

f5d2986

jan-matthis changed the title ~~Add Parallel Launcher - WIP~~ Add Joblib Launcher - WIP Feb 2, 2020

omry requested changes Feb 2, 2020

View reviewed changes

omry added 2 commits February 3, 2020 21:29

moved lint to be first nox test

6b5061c

pin master to OmegaConf 2.0.0rc4 until Structured configs branch lands

619a23f

jan-matthis added 6 commits February 3, 2020 22:28

Implementing changes from first review

0ca221e

Using default config in test

8e94d38

Two test cases in task_launcher_cfg

263882a

Add joblib to third_party

17ff451

Fix SearchPath

d2847a3

Modify testcase 2 to have no defaults

dc0295c

omry reviewed Feb 3, 2020

View reviewed changes

Changes after review round 2

6ffd77e

jan-matthis commented Feb 4, 2020

View reviewed changes

plugins/joblib_launcher/hydra_plugins/joblib_launcher/joblib_launcher.py Outdated Show resolved Hide resolved

omry reviewed Feb 5, 2020

View reviewed changes

jan-matthis added 6 commits February 5, 2020 14:09

Rename plugin with hydra prefix

d16cbb6

Add news entry

b284b68

Change handling of Singleton state

af6d7ab

None to null, add comments

5949bf9

Update README with comments on options

4a9921a

Remove unused import

81a2616

Change folder name

61a1f9a

Additional renaming for consistency with hydra_colorlog

2ebdc3e

jan-matthis added 3 commits February 12, 2020 02:46

Move website page

af70c54

README.md as a pointer to website

d226762

Reduce n_jobs to 2 (trying to debug CircleCI failures)

74eb284

omry reviewed Feb 12, 2020

View reviewed changes

Sort imports

96744ec

Sort imports

d86c242

Restore IntegrationTest, limit TestSuite

3b9cc8a

jan-matthis added 2 commits February 12, 2020 10:58

Remove n_jobs limit again

d341d1d

Revert to original tests; exclude windows

1d10c9b

Formatting

febcf68

Mypy fix

4a421f7

Add type ignore to decorator

9f92a1c

Supresses error: Untyped decorator makes function "test_discovery" untyped

omry and others added 3 commits February 12, 2020 14:46

updated description in intro

b810010

Fix typos

65b3517

Work around CircleCI test coverage issue (#413)

7f2e75a

Using a different circleci python 3.6 Image for Linux to work around new issue

omry requested changes Feb 12, 2020

View reviewed changes

omry merged commit 4a81bce into facebookresearch:master Feb 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Joblib Launcher for parallel execution (multi-process and multi-threaded) #392

Add Joblib Launcher for parallel execution (multi-process and multi-threaded) #392

jan-matthis commented Jan 31, 2020 •

edited

omry commented Jan 31, 2020

jan-matthis commented Feb 2, 2020

omry left a comment •

edited

omry commented Feb 3, 2020

lgtm-com bot commented Feb 5, 2020

lgtm-com bot commented Feb 5, 2020

omry Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

omry commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

omry commented Feb 12, 2020

omry Feb 12, 2020

omry commented Feb 12, 2020

omry commented Feb 12, 2020


		This example demonstrates how to to customize the logging behavior of your Hydra app, by making the following changes

		This example demonstrates how to customize the logging behavior of your Hydra app, by making the following changes

Add Joblib Launcher for parallel execution (multi-process and multi-threaded) #392

Add Joblib Launcher for parallel execution (multi-process and multi-threaded) #392

Conversation

jan-matthis commented Jan 31, 2020 • edited

Motivation

Have you read the Contributing Guidelines on pull requests?

Test Plan

Related Issues and PRs

omry commented Jan 31, 2020

jan-matthis commented Feb 2, 2020

omry left a comment • edited

Choose a reason for hiding this comment

omry commented Feb 3, 2020

lgtm-com bot commented Feb 5, 2020

lgtm-com bot commented Feb 5, 2020

omry Feb 12, 2020

Choose a reason for hiding this comment

lgtm-com bot commented Feb 12, 2020

omry commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

lgtm-com bot commented Feb 12, 2020

omry commented Feb 12, 2020

omry Feb 12, 2020

Choose a reason for hiding this comment

omry commented Feb 12, 2020

omry commented Feb 12, 2020

jan-matthis commented Jan 31, 2020 •

edited

omry left a comment •

edited