fix: enable usage of job grouping in GLS #1054
Conversation
|
Kudos, SonarCloud Quality Gate passed!
|
docs/snakefiles/rules.rst
Outdated
| @@ -1331,6 +1331,8 @@ However, if we would add ``group: "mygroup"`` to rule ``c``, all jobs would end | |||
| Alternatively, groups can be defined via the command line interface. | |||
| This enables to almost arbitrarily partition the DAG, e.g. in order to safe network traffic, see :ref:`here <job_grouping>`. | |||
|
|
|||
| For execution on the cloud using Google Life Science API and preemptible instances, note than if any of the rules in a group is included in the subset of rules to be executed using preembtible instances (with command line option ``--preemptible-rules``), the group will be also scheduled to be executed on a preemptible instance. | |||
There was a problem hiding this comment.
Mhm, I wonder if that is really the desired behavior? Shouldn't it rather be: all jobs must be preemptible for the group to be preemptible as well?
There was a problem hiding this comment.
I don't have a strong opinion, my only thought went into cost optimization. If one were to mark a rule as "preemptible" because being somehow expensive, I would like to make sure is executed in a preemptible instance.
That said, I wouldn't mind to replace the any by the all condition. Let me know if it is still your preference
There was a problem hiding this comment.
THinking about it, we could also fail with a WorkflowError in case not all jobs are preemptible, and request people to set this homogeneously for all jobs in the group, pointing to the corresponding section in the docs.
There was a problem hiding this comment.
I think this is best, because peoples credits depend on that behavior, so we'd rather not imply something that will cost more money than expected.
There was a problem hiding this comment.
It sounds reasonable to me! I implemented the few edits.
sposadac
changed the title
|
Kudos, SonarCloud Quality Gate passed!
|
|
Hi @johanneskoester, Thanks for approving the changes! Since I am not authorized to merge the PR, I am wondering what would be the next step. |
|
Merging! Sorry, I was a bit busy in the last weeks. |
* Allow using grouped rules on Google cloud using Google Life Science API * Explain new behaviour in the docs * Modify behaviour to request all rules in a group to be set as preemptible instances * Fix typo and add comment about documentation Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
Description
Allow using Google Life Science preemptible rules and job grouping, by checking if any of the rules in a given group has been flagged for execution using preemptible instances.
QC
docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).