New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Joblib Launcher for parallel execution (multi-process and multi-threaded) #392
Merged
Merged
Changes from 42 commits
Commits
Show all changes
53 commits
Select commit
Hold shift + click to select a range
3e83d0e
Add Parallel Launcher
jan-matthis f5d2986
Renamed to JoblibLauncher
jan-matthis 6b5061c
moved lint to be first nox test
omry 619a23f
pin master to OmegaConf 2.0.0rc4 until Structured configs branch lands
omry 0ca221e
Implementing changes from first review
jan-matthis 8e94d38
Using default config in test
jan-matthis 263882a
Two test cases in task_launcher_cfg
jan-matthis 17ff451
Add joblib to third_party
jan-matthis d2847a3
Fix SearchPath
jan-matthis dc0295c
Modify testcase 2 to have no defaults
jan-matthis 6ffd77e
Changes after review round 2
jan-matthis d16cbb6
Rename plugin with hydra prefix
jan-matthis b284b68
Add news entry
jan-matthis af6d7ab
Change handling of Singleton state
jan-matthis 5949bf9
None to null, add comments
jan-matthis 4a9921a
Update README with comments on options
jan-matthis 81a2616
Remove unused import
jan-matthis 61a1f9a
Change folder name
jan-matthis 2ebdc3e
Additional renaming for consistency with hydra_colorlog
jan-matthis 0deaeb3
Newline
jan-matthis 3b0d50c
Remove timestamps
jan-matthis 6b01866
Add link to example
jan-matthis f77ac3f
Moving config.yaml to same folder as my_app.py for consistency with h…
jan-matthis 4ada0ac
Add assertions for runs, fix job.id
jan-matthis 69ebde8
Comments before variables
jan-matthis 65ec255
Add no_output_timeout to linux and win tests
jan-matthis b4f0d41
Use default config for thread-based test
jan-matthis 832ece5
Fix flake8
jan-matthis 152d76c
Set no_output_timeout to 30m for mac and win
jan-matthis 2e6c930
Add comments to tests
jan-matthis e2f29d8
Remove extra line
jan-matthis ac4e41f
Small docstring fix
jan-matthis 2483955
Move configure_log up
jan-matthis 8622db0
Remove configure_log, it is called inside of run_job
jan-matthis 23fb20c
Rename HydraJoblibLauncher to JoblibLauncher
jan-matthis f38b342
Imports
jan-matthis 7170eef
Add website
jan-matthis 766f864
Disable thread-based test
jan-matthis 3d43f6e
Add note about adding new pages
jan-matthis af70c54
Move website page
jan-matthis d226762
README.md as a pointer to website
jan-matthis 74eb284
Reduce n_jobs to 2 (trying to debug CircleCI failures)
jan-matthis 96744ec
Sort imports
jan-matthis d86c242
Sort imports
jan-matthis 3b9cc8a
Restore IntegrationTest, limit TestSuite
jan-matthis d341d1d
Remove n_jobs limit again
jan-matthis 1d10c9b
Revert to original tests; exclude windows
jan-matthis febcf68
Formatting
jan-matthis 4a421f7
Mypy fix
jan-matthis 9f92a1c
Add type ignore to decorator
jan-matthis b810010
updated description in intro
omry 65b3517
Fix typos
nzw0301 7f2e75a
Work around CircleCI test coverage issue (#413)
omry File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
Add Joblib Launcher plugin. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
global-exclude *.pyc | ||
global-exclude __pycache__ | ||
recursive-include hydra_plugins/* *.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
# Hydra Joblib Launcher | ||
See [website](https://hydra.cc/docs/plugins/joblib_launcher) for more information | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
defaults: | ||
- hydra/launcher: joblib | ||
|
||
task: 1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved | ||
import logging | ||
import os | ||
import time | ||
|
||
import hydra | ||
|
||
log = logging.getLogger(__name__) | ||
|
||
|
||
@hydra.main(config_path="config.yaml") | ||
def my_app(cfg): | ||
log.info(f"Process ID {os.getpid()} executing task {cfg.task} ...") | ||
|
||
time.sleep(1) | ||
|
||
|
||
if __name__ == "__main__": | ||
my_app() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved | ||
# type: ignore | ||
__path__ = __import__("pkgutil").extend_path(__path__, __name__) |
4 changes: 4 additions & 0 deletions
4
plugins/hydra_joblib_launcher/hydra_plugins/hydra_joblib_launcher/__init__.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved | ||
from .joblib_launcher import JoblibLauncher | ||
|
||
__all__ = ["JoblibLauncher"] |
1 change: 1 addition & 0 deletions
1
plugins/hydra_joblib_launcher/hydra_plugins/hydra_joblib_launcher/conf/__init__.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved |
39 changes: 39 additions & 0 deletions
39
...hydra_joblib_launcher/hydra_plugins/hydra_joblib_launcher/conf/hydra/launcher/joblib.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
hydra: | ||
launcher: | ||
class: hydra_plugins.hydra_joblib_launcher.JoblibLauncher | ||
params: | ||
joblib: ${hydra.joblib} | ||
|
||
joblib: | ||
# maximum number of concurrently running jobs. if -1, all CPUs are used | ||
n_jobs: -1 | ||
|
||
# allows to hard-code backend, otherwise inferred based on prefer and require | ||
backend: null | ||
|
||
# processes or threads, soft hint to choose backend | ||
prefer: processes | ||
|
||
# null or sharedmem, sharedmem will select thread-based backend | ||
require: null | ||
|
||
# if greater than zero, prints progress messages | ||
verbose: 0 | ||
|
||
# timeout limit for each task | ||
timeout: null | ||
|
||
# number of batches to be pre-dispatched | ||
pre_dispatch: 2*n_jobs | ||
|
||
# number of atomic tasks to dispatch at once to each worker | ||
batch_size: auto | ||
|
||
# folder used for memmapping large arrays for sharing memory with workers | ||
temp_folder: null | ||
|
||
# thresholds size of arrays that triggers automated memmapping | ||
max_nbytes: 1M | ||
|
||
# memmapping mode for numpy arrays passed to workers | ||
mmap_mode: r |
138 changes: 138 additions & 0 deletions
138
plugins/hydra_joblib_launcher/hydra_plugins/hydra_joblib_launcher/joblib_launcher.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,138 @@ | ||
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved | ||
import logging | ||
from pathlib import Path | ||
from typing import Any, Dict, List, Optional, Sequence | ||
|
||
from joblib import Parallel, delayed # type: ignore | ||
from omegaconf import DictConfig, open_dict | ||
|
||
from hydra.core.config_loader import ConfigLoader | ||
from hydra.core.config_search_path import ConfigSearchPath | ||
from hydra.core.hydra_config import HydraConfig | ||
from hydra.core.singleton import Singleton | ||
from hydra.core.utils import ( | ||
JobReturn, | ||
configure_log, | ||
filter_overrides, | ||
run_job, | ||
setup_globals, | ||
) | ||
from hydra.plugins.launcher import Launcher | ||
from hydra.plugins.search_path_plugin import SearchPathPlugin | ||
from hydra.types import TaskFunction | ||
|
||
log = logging.getLogger(__name__) | ||
|
||
|
||
class JoblibLauncherSearchPathPlugin(SearchPathPlugin): | ||
""" | ||
This plugin is allowing configuration files provided by the JoblibLauncher plugin to be | ||
discovered and used once the plugin is installed | ||
""" | ||
|
||
def manipulate_search_path(self, search_path: ConfigSearchPath) -> None: | ||
# Appends the search path for this plugin to the end of the search path | ||
search_path.append( | ||
"hydra-joblib-launcher", "pkg://hydra_plugins.hydra_joblib_launcher.conf", | ||
) | ||
|
||
|
||
class JoblibLauncher(Launcher): | ||
def __init__(self, joblib: Dict[str, Any] = {},) -> None: | ||
"""Joblib Launcher | ||
|
||
Launches parallel jobs using Joblib.Parallel. For details, refer to: | ||
https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html | ||
|
||
This plugin is based on the idea and inital implementation of @emilemathieutmp: | ||
https://github.com/facebookresearch/hydra/issues/357 | ||
""" | ||
self.config: Optional[DictConfig] = None | ||
self.config_loader: Optional[ConfigLoader] = None | ||
self.task_function: Optional[TaskFunction] = None | ||
|
||
self.joblib = joblib | ||
|
||
def setup( | ||
self, | ||
config: DictConfig, | ||
config_loader: ConfigLoader, | ||
task_function: TaskFunction, | ||
) -> None: | ||
self.config = config | ||
self.config_loader = config_loader | ||
self.task_function = task_function | ||
|
||
def launch(self, job_overrides: Sequence[Sequence[str]]) -> Sequence[JobReturn]: | ||
""" | ||
:param job_overrides: a List of List<String>, where each inner list is the arguments for one job run. | ||
:return: an array of return values from run_job with indexes corresponding to the input list indexes. | ||
""" | ||
setup_globals() | ||
assert self.config is not None | ||
assert self.config_loader is not None | ||
assert self.task_function is not None | ||
|
||
configure_log(self.config.hydra.hydra_logging, self.config.hydra.verbose) | ||
sweep_dir = Path(str(self.config.hydra.sweep.dir)) | ||
sweep_dir.mkdir(parents=True, exist_ok=True) | ||
log.info( | ||
"Joblib.Parallel({}) is launching {} jobs".format( | ||
",".join([f"{k}={v}" for k, v in self.joblib.items()]), | ||
len(job_overrides), | ||
) | ||
) | ||
log.info("Sweep output dir : {}".format(sweep_dir)) | ||
|
||
singleton_state = Singleton.get_state() | ||
|
||
runs = Parallel(**self.joblib)( | ||
delayed(dispatch_job)( | ||
idx, | ||
overrides, | ||
self.config_loader, | ||
self.config, | ||
self.task_function, | ||
singleton_state, | ||
) | ||
for idx, overrides in enumerate(job_overrides) | ||
) | ||
|
||
assert isinstance(runs, List) | ||
for run in runs: | ||
assert isinstance(run, JobReturn) | ||
|
||
return runs | ||
|
||
|
||
def dispatch_job( | ||
idx: int, | ||
overrides: Sequence[str], | ||
config_loader: ConfigLoader, | ||
config: DictConfig, | ||
task_function: TaskFunction, | ||
singleton_state: Dict[Any, Any], | ||
) -> JobReturn: | ||
"""Calls `run_job` in parallel | ||
|
||
Note that Joblib's default backend runs isolated Python processes, see | ||
https://joblib.readthedocs.io/en/latest/parallel.html#shared-memory-semantics | ||
""" | ||
setup_globals() | ||
Singleton.set_state(singleton_state) | ||
|
||
log.info("\t#{} : {}".format(idx, " ".join(filter_overrides(overrides)))) | ||
sweep_config = config_loader.load_sweep_config(config, list(overrides)) | ||
with open_dict(sweep_config): | ||
sweep_config.hydra.job.id = "{}_{}".format(sweep_config.hydra.job.name, idx) | ||
sweep_config.hydra.job.num = idx | ||
HydraConfig.instance().set_config(sweep_config) | ||
|
||
ret = run_job( | ||
config=sweep_config, | ||
task_function=task_function, | ||
job_dir_key="hydra.sweep.dir", | ||
job_subdir_key="hydra.sweep.subdir", | ||
) | ||
|
||
return ret |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Until 1.0.0 is released, the url will be:
https://hydra.cc/docs/next/plugins/joblib_launcher