Executor Keyword Arguments #351
Conversation
for more information, see https://pre-commit.ci
WalkthroughThe recent changes primarily focus on refactoring the Changes
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (7)
- pympipool/scheduler/init.py (3 hunks)
- pympipool/scheduler/flux.py (5 hunks)
- pympipool/scheduler/local.py (6 hunks)
- pympipool/scheduler/slurm.py (6 hunks)
- tests/test_flux_executor.py (4 hunks)
- tests/test_local_executor.py (18 hunks)
- tests/test_local_executor_future.py (4 hunks)
Additional comments not posted (8)
tests/test_flux_executor.py (4)
52-55: The refactoring of the test case to useexecutor_kwargsaligns well with the changes in the executor classes.
65-66: The refactoring of the test case to useexecutor_kwargsfor threading configuration is consistent with the changes in the executor classes.
77-78: The refactoring of the test case to useexecutor_kwargsfor parallel execution settings is consistent with the changes in the executor classes.
86-87: The refactoring of the test case to useexecutor_kwargsfor single task execution is consistent with the changes in the executor classes.tests/test_local_executor_future.py (3)
21-22: The refactoring of the test case to useexecutor_kwargsaligns well with the changes in the executor classes.
36-37: The refactoring of the test case to useexecutor_kwargsfor multi-core configuration is consistent with the changes in the executor classes.
67-69: The refactoring of the test case to useexecutor_kwargsfor ensuring independence from the executor is consistent with the changes in the executor classes.Also applies to: 106-108
tests/test_local_executor.py (1)
102-104: Ensure consistency in the use ofexecutor_kwargsacross all test cases.The changes to the test cases correctly reflect the updated constructors of
PyLocalExecutorandPyLocalStepExecutorto useexecutor_kwargs. This aligns with the PR's objective to simplify parameter passing and enhance flexibility.Also applies to: 114-116, 129-133, 138-142, 152-153, 162-163, 181-182, 195-196, 205-206, 224-225, 236-240, 277-278, 291-292, 302-305, 318-319, 327-328, 335-336, 355-356, 376-377, 390-391, 407-408
| cwd: Optional[str] = None, | ||
| hostname_localhost: bool = False, | ||
| command_line_argument_lst: list[str] = [], | ||
| executor_kwargs: dict = {}, |
There was a problem hiding this comment.
Ensure that executor_kwargs is safely initialized to avoid issues with mutable default arguments.
- def __init__(self, max_cores: int = 1, executor_kwargs: dict = {}):
+ def __init__(self, max_cores: int = 1, executor_kwargs: dict = None):
+ if executor_kwargs is None:
+ executor_kwargs = {}Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| executor_kwargs: dict = {}, | |
| def __init__(self, max_cores: int = 1, executor_kwargs: dict = None): | |
| if executor_kwargs is None: | |
| executor_kwargs = {} |
| hostname_localhost: bool = False, | ||
| command_line_argument_lst: list[str] = [], | ||
| executor_kwargs: dict = {}, | ||
| ): |
There was a problem hiding this comment.
Consider using a more robust default for executor_kwargs to avoid potential issues with mutable default arguments.
- def __init__(self, max_workers: int = 1, executor_kwargs: dict = {}):
+ def __init__(self, max_workers: int = 1, executor_kwargs: dict = None):
+ if executor_kwargs is None:
+ executor_kwargs = {}Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| executor_kwargs: dict = {}, | |
| def __init__(self, max_workers: int = 1, executor_kwargs: dict = None): | |
| if executor_kwargs is None: | |
| executor_kwargs = {} |
| oversubscribe: bool = False, | ||
| cwd: Optional[str] = None, | ||
| hostname_localhost: bool = False, | ||
| executor_kwargs: dict = {}, |
There was a problem hiding this comment.
Ensure that executor_kwargs is safely initialized to avoid issues with mutable default arguments.
- def __init__(self, max_cores: int = 1, executor_kwargs: dict = {}):
+ def __init__(self, max_cores: int = 1, executor_kwargs: dict = None):
+ if executor_kwargs is None:
+ executor_kwargs = {}Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| executor_kwargs: dict = {}, | |
| executor_kwargs: dict = None): | |
| if executor_kwargs is None: | |
| executor_kwargs = {} |
| hostname_localhost: bool = False, | ||
| ): | ||
| def __init__(self, max_workers: int = 1, executor_kwargs: dict = {}): | ||
| super().__init__() |
There was a problem hiding this comment.
Consider using a more robust default for executor_kwargs to avoid potential issues with mutable default arguments.
- def __init__(self, max_workers: int = 1, executor_kwargs: dict = {}):
+ def __init__(self, max_workers: int = 1, executor_kwargs: dict = None):
+ if executor_kwargs is None:
+ executor_kwargs = {}Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| def __init__(self, max_workers: int = 1, executor_kwargs: dict = {}): | |
| def __init__(self, max_workers: int = 1, executor_kwargs: dict = None): | |
| if executor_kwargs is None: | |
| executor_kwargs = {} |
| executor: Optional[flux.job.FluxExecutor] = None, | ||
| pmi: Optional[str] = None, | ||
| hostname_localhost: Optional[bool] = False, | ||
| executor_kwargs: dict = {}, |
There was a problem hiding this comment.
Consider using a more robust default for executor_kwargs to avoid potential issues with mutable default arguments.
- def __init__(self, max_workers: int = 1, executor_kwargs: dict = {}):
+ def __init__(self, max_workers: int = 1, executor_kwargs: dict = None):
+ if executor_kwargs is None:
+ executor_kwargs = {}Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| executor_kwargs: dict = {}, | |
| executor_kwargs: dict = None, | |
| if executor_kwargs is None: | |
| executor_kwargs = {} |
| executor: Optional[flux.job.FluxExecutor] = None, | ||
| pmi: Optional[str] = None, | ||
| hostname_localhost: Optional[bool] = False, | ||
| executor_kwargs: dict = {}, |
There was a problem hiding this comment.
Ensure that executor_kwargs is safely initialized to avoid issues with mutable default arguments.
- def __init__(self, max_cores: int = 1, executor_kwargs: dict = {}):
+ def __init__(self, max_cores: int = 1, executor_kwargs: dict = None):
+ if executor_kwargs is None:
+ executor_kwargs = {}Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| executor_kwargs: dict = {}, | |
| def __init__(self, max_cores: int = 1, executor_kwargs: dict = None): | |
| if executor_kwargs is None: | |
| executor_kwargs = {} |
| backend=backend, flux_installed=flux_installed, slurm_installed=slurm_installed | ||
| ) | ||
| check_pmi(backend=backend, pmi=pmi) | ||
| executor_kwargs = { |
There was a problem hiding this comment.
Consider using a more robust default for executor_kwargs to avoid potential issues with mutable default arguments.
- executor_kwargs = {
+ executor_kwargs = executor_kwargs or {}Committable suggestion was skipped due low confidence.
for more information, see https://pre-commit.ci
…to executor_kwargs # Conflicts: # tests/test_local_executor.py
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
Actionable comments posted: 2
Outside diff range and nitpick comments (1)
pympipool/shared/executorbase.py (1)
17-17: Ensure consistent import order and grouping.Consider grouping all imports from
pympipool.shared.interfacetogether for better readability and maintainability.
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- pympipool/shared/executorbase.py (5 hunks)
- tests/test_local_executor.py (19 hunks)
Files skipped from review as they are similar to previous changes (1)
- tests/test_local_executor.py
| future_queue: queue.Queue, | ||
| cores: int, | ||
| interface_class: BaseInterface, | ||
| cores: int = 1, |
There was a problem hiding this comment.
Standardize default parameter values across similar functions.
It's good practice to ensure consistency in default parameter values across similar functions to avoid confusion and potential bugs. For instance, cores and interface_class have default values in execute_parallel_tasks and execute_separate_tasks, but not in _submit_function_to_separate_process. Consider adding default values for these parameters in _submit_function_to_separate_process as well.
Also applies to: 259-259, 312-312, 313-313, 553-553
| if "cores" not in kwargs.keys(): | ||
| kwargs["cores"] = 1 |
There was a problem hiding this comment.
Handle default parameter values more robustly.
Instead of checking if cores is not in kwargs and then setting it to 1, you could use kwargs.get('cores', 1) which is a more Pythonic and concise way to handle default values.
- if "cores" not in kwargs.keys():
- kwargs["cores"] = 1
+ kwargs.setdefault('cores', 1)Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| if "cores" not in kwargs.keys(): | |
| kwargs["cores"] = 1 | |
| kwargs.setdefault('cores', 1) |
Summary by CodeRabbit
New Features
executor_kwargsparameter to simplify and enhance the flexibility of executor initialization across various backends (Flux, Local, Slurm).Refactor
executor_kwargsdictionary for executor classes to streamline configurations.executor_kwargsfor passing parameters.Bug Fixes
coresandmax_coresparameters in task execution functions to improve reliability.Chores