Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for SLURM submission locally #55

Open
Andrew-S-Rosen opened this issue May 6, 2023 · 2 comments
Open

Allow for SLURM submission locally #55

Andrew-S-Rosen opened this issue May 6, 2023 · 2 comments
Labels
feature New feature or functionality help-wanted

Comments

@Andrew-S-Rosen
Copy link
Contributor

Andrew-S-Rosen commented May 6, 2023

What should we add?

The SLURM executor relies on SSHing into the machine and submitting the SLURM jobs. This is excellent, but it would be ideal if the user also had the option to use the SLURM executor locally if they are dispatching from the HPC machine itself. One could imagine this might be particularly relevant, for instance, if there are intense security measures that restrict SSH access in a way that the plugin can't work with (e.g. requiring a physical Yubikey).

To be clear, this would be a large feature addition, so I am not suggesting the maintainers or myself work on it ASAP, but I wanted to log this prior discussion here anyway in case someone feels ambitious. In the meantime, it should be possible to use the Dask executor to write SLURM jobs locally (perhaps worth a tutorial if it's ultimately too ambitious to include here).

Describe alternatives you've considered.

No response

@Andrew-S-Rosen Andrew-S-Rosen added the feature New feature or functionality label May 6, 2023
@santoshkumarradha
Copy link
Member

hey @arosen93 I think this is a neat feature.

IMO this does not seem like a lot of work, I think it should be as simple as abstracting https://github.com/AgnostiqHQ/covalent-slurm-plugin/blob/develop/covalent_slurm_plugin/slurm.py#L261C2-L266 asyncssh.connect , which returns a SSHClientConnection object and making it such that we have an local abstraction for this with a toggle switch inside the plugin kwargs.

@Andrew-S-Rosen
Copy link
Contributor Author

Andrew-S-Rosen commented Jun 28, 2023

@santoshkumarradha: Do you know if there is a way to use asyncssh such that the connection object returned simply performs actions on the local machine? I had trouble finding something like that in the docs. If there's an easy way to return a connection object that will run locally, I'll happily test things out and submit a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or functionality help-wanted
Projects
None yet
Development

No branches or pull requests

2 participants