all rules should have resources

[rekado]

On a system with limited resources snakemake will sometimes schedule too many concurrent tasks, which leads to unexpected memory exhaustion. This is hard to reproduce, but it is a result of not having a resources section for all rules.

We could avoid this by adding a resources section to all rules, not just the known memory intensive rules.

[smoe]

You observed that behaviour for local executions, right? With a regular 64GByte memory machine the default settings will start as many STAR instances as there are number of jobs allowed, which was 6 for as. This happily violated the constraint that 6*16GB (memory for star_map) >64GB (memory of machine) contraint.
Something is wrong somewhere IMHO. What helped me to just go somewhere is to limit the number of jobs to 2. We were then a bit suprised to find four bamCoverage jobs, but these don't nee much memory anyway, but should not be allowed since these are four jobs now running in parallel. Is there something I shall test to help locating the issue?

[rekado]

We've been using hisat2 for our tests on memory-restricted machines. (STAR has somewhat adventurous memory requirements, which makes it hard to use it in our current round of tests.)

Something is wrong somewhere IMHO

Yeah, I agree :)

I would like to avoid having to globally limit the number of jobs, because the scheduler should be able to respect our constraints. That's why I'd like us to mark up all rules first, so that we can rule out the bug in this obvious place.

[smoe]

I have updated snakemake to version 6.9.1 on Debian and do not see this problem any more. May I ask about what platform you made your observations with?

[rekado]

We had these problems on AWS. The problem is not easily reproduced because it depends on when snakemake schedules tasks on the actual machine.

I'll upgrade snakemake as well now.