Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: support for external executor plugins (#2305)
Hi @johanneskoester! 👋 As we chat about in a thread somewhere, I think it would be really powerful to allow for installing (and discovering) external plugins to Snakemake. Specifically for the Flux Operator, I have easily three designs I'm testing, and it's not really appropriate to add them proper to snakemake - but I believe the developer user should be empowered to flexibly add/remove and test them out. This pull request is first try demo of how snakemake could allow external executor plugins. I say "first try" because it's the first time I've experimented with plugins, and I tried to choose a design that optimizes simplicity and flexibility without requiring external packages, or specific features of setuptools or similar (that are likely to change). The basic design here uses pkgutil to discover snakemake_executor_* plugins, and then provides them to the client (to add arguments) and scheduler to select using one with `--executor`. I've written up an entire tutorial and the basic design in this early prototype, which is basically the current Flux integration as a plugin! https://github.com/snakemake/snakemake-executor-flux. The user would basically do: ```bash # Assuming this was released on pypi (it's not yet) $ pip install snakemake-executor-flux # Run the workflow using the flux custom executor $ snakemake --jobs 1 --executor flux ``` I've designed it so that plugins are validated only when chosen, and each plugin can add or otherwise customize the parser, and then (after parsing) further tweak the args if chosen. Then in scheduler.py, we simply look if the user selected a plugin, and call the main executor (and local_executor) classes if this is the case. The one hard piece is having a flexible way to pass forward all those custom arguments. The current snakemake design has basically a custom boolean for every executor hard coded (e.g., `--flux` or `--slurm`) and while we don't want to blow that up, I'm worried moving forward passing all these custom namespaced arguments through the init, workflow, scheduler/dag, is going to get very messy. So the approach here is a suggested way to handle the expanding space of additional executors by way of passing forward full args, and then allowing the plugins to customize the parser before or after. If we were to, for example, turn current executors into plugins (something I expect we might want to do for the Google Life Sciences API that is going to be deprecated in favor of batch) we could write out a more hardened spec - some configuration class that is passed from the argument parser through the executor and through execution (instead of all the one off arguments). Anyway - this is just a first shot and I'm hoping to start some discussion! This is a totally separate thing from TBA work with Google Batch - this is something that I've wanted to try for a while as I've wanted to add more executors and have seen the executor space exploding. :laughing: I haven't written tests or updated any docs yet pending our discussion! ### QC * [ ] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [ ] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake). --------- Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de> Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
- Loading branch information
4 people
committed
Aug 7, 2023
1 parent
1c5d154
commit c9eaa4e