Advanced DAG partitioning

[johanneskoester]

• Allow to define groups via the command line (--group rulename1=groupname rulename2=groupname)
• Allow groups to span multiple connected components (only 1 by default) (--group-components groupname=3 groupname2=5)

This solves all remaining use cases for DAG partitioning.

cc @cmeesters

[github-actions]

Please format your code with black: black snakemake tests/*.py.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities (and 0 Security Hotspots to review)
0 Code Smells

No Coverage information
0.0% Duplication

[mdshw5]

Hey @johanneskoester I really like this idea. I have some pretty large DAGs that I currently manage by using --batch. One thing I've noticed is that the snakemake process has trouble tracking the outputs from a large (>10k) number of targets, and so "finishing" jobs becomes rate limiting. Would this PR address that? Otherwise I was wondering about adding an "automatic batching" flag, where the user can decide how many times to split the DAG, then have snakemake automatically run each batch sequentially.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities (and 0 Security Hotspots to review)
1 Code Smell

No Coverage information
0.0% Duplication

[johanneskoester]

Hey @johanneskoester I really like this idea. I have some pretty large DAGs that I currently manage by using --batch. One thing I've noticed is that the snakemake process has trouble tracking the outputs from a large (>10k) number of targets, and so "finishing" jobs becomes rate limiting. Would this PR address that? Otherwise I was wondering about adding an "automatic batching" flag, where the user can decide how many times to split the DAG, then have snakemake automatically run each batch sequentially.

Hi @mdshw5. This is rather meant for submitting jobs together to one physical node (which already works vertically, while this PR enables to do this for not connected parts of the DAG). However, an automatic batching option is surely a good idea. In principle, you can get that now already by having a for loop over snakemake with --batch. Nevertheless, it doesn't hurt to add this internally.

[lpla]

Hi. Would this feature allow to submit jobs as a "job array" instead of a job group in a single node? #301

Job arrays would be able to run many grouped small jobs in a larger scale (e.g. all available nodes, instead of only one) with only one submit to the scheduler.