New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add support for running shell commands asynchronously with run_shell_cmd
#4444
Conversation
…sing run_shell_cmd
…ock.skip_extensions_parallel
35c8a41
to
a19776a
Compare
Quick test:
And that's only the skipping part being done in parallel... |
run_shell_cmd
(WIP)run_shell_cmd
(WIP)
I've tested this with
The scaling is not spectacular, but I think that's mainly determine by the order in which extensions are being considered, not so much the implementation of running the extension installations in parallel in the background. So, I think this is ready for review/merge. Do note that these changes require a corresponding change in easyblocks, see: I prefer keeping the support for There's a small additional change needed to make parallel installation of extensions work for easyconfigs like |
run_shell_cmd
(WIP)run_shell_cmd
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
TODO:
run_shell_cmd
for usingasynchronous=True
run_shell_cmd
withasynchronous=True
inEasyBlock.skip_extensions_parallel
run_shell_cmd
withasynchronous=True
inExtension.async_cmd_start
R
easyconfig and--parallel-extensions-install
(and--skip
)