feat: introduce --workflow-profile for additional workflow specific profiles that overwrite global profiles; add ability to define key-value CLI flags like --set-threads or --set-resources as multi-level dictionaries in profile config yaml files

[johanneskoester]

Description

QC

• The PR contains a test case for the changes or the changes are already covered by an existing test case.
• The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

[github-actions]

Please format your code with black: black snakemake tests/*.py.

[sonarcloud]

Please retry analysis of this Pull-Request directly on SonarCloud.

[cmeesters]

Hoi @johanneskoester

You are WAY too fast for my taste ;-)

Anyway, I like this very much indeed. I will test it by the end of the week (just that some more meetings are in the way to do this any sooner). Allow me a minor question: Why the original --no-workflow-profile flag and currently the explicitly required None to the --workflow-profile flag? If there is no such flag, it could just be ignored. Or am I missing something here?

[johanneskoester]

Hoi @johanneskoester

You are WAY too fast for my taste ;-)

Anyway, I like this very much indeed. I will test it by the end of the week (just that some more meetings are in the way to do this any sooner). Allow me a minor question: Why the original --no-workflow-profile flag and currently the explicitly required None to the --workflow-profile flag? If there is no such flag, it could just be ignored. Or am I missing something here?

This is because of the behavior that if --workflow-profile is not specified the profile stored at workflow/profiles/default (if Snakefile is at workflow/Snakefile is automatically used. This default behavior is important to allow workflow designers to ensure certain flags are set. E.g. some default constraints on resources like for example:

resources:
  biomart_api_calls: 1

To ensure that some rule that queries biomart is not executed many times in parallel (which would lead to being blocked).
If there is no such profile next to the Snakefile, it will of course not be used and --workflow-profile none is equivalent to not specifying it at all.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
0.0% Duplication