Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to set a max limit of threads to all rules #1015

Closed
ShaiberAlon opened this issue Oct 17, 2018 · 3 comments

Comments

Projects
None yet
3 participants
@ShaiberAlon
Copy link
Member

commented Oct 17, 2018

This is an issue only because early on I made a design decision not to use the cluster-config option of snakemake, and instead to define the threads in the config file.

The problem is that now if we want the workflow to use up to 5 threads, we need to manually change each rule that uses more than 5 threads by default (for example, assembly uses 11 by default).

In order to make it easier I plan to add a general param to the config file: max_threads. By default it will be None and then there will be no limit. If the user sets max_threads, then the threads used for each rule, would be the minimum between what is set by default (or specified in the config file) and max_threads. So if for example, the config file looks like this:

{
    "megahit": {
        "run": true,
        "threads": 10
    },
    "max_threads": 5
}

Then megahit will only use 5 threads.

@ShaiberAlon ShaiberAlon added this to the v5.3 milestone Oct 17, 2018

@ShaiberAlon ShaiberAlon self-assigned this Oct 17, 2018

@ShaiberAlon

This comment has been minimized.

Copy link
Member Author

commented Oct 17, 2018

Ok, I now think about it, and maybe there is a way to do this using the already existing resources.nodes that we have.

@FlorianTrigodet

This comment has been minimized.

Copy link

commented Oct 18, 2018

Hi Alon,
So far, I used the flag --additional-params --jobs 5 with the anvi-run-workflow command to scale down the maximum threads used. What would be different with the max_threads param ?
I guess it is easier if every parameters are kept in the config file.

@ShaiberAlon

This comment has been minimized.

Copy link
Member Author

commented Oct 18, 2018

Hi @FlorianTrigodet,
This question I posted on stackoverflow might clarify why I think I need this:
https://stackoverflow.com/questions/52879145/limit-total-thread-use-number-of-jobs-and-maximum-threads-per-rule-when-using

Please let me know if it doesn't clarify it.

@meren meren removed this from the v5.3 milestone Mar 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.