fix: improved error handling for cluster status scripts and smarter job selector choice in case of cluster submission (use greedy for single jobs).

[johanneskoester]

Description

See title. Could help with issue #759.

QC

• The PR contains a test case for the changes or the changes are already covered by an existing test case.
• The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

[github-actions]

Please format your code with black: black snakemake tests/*.py.

[johanneskoester]

@vsoch do you have any idea why the GLS test suddenly fails here? I did not expect my changes to have any side effects on GLS.

[vsoch]

The last step in the GLS test it looks like is now using this greedy selector, so that's the change:

DEBUG snakemake.logging:logging.py:553 Using greedy selector because only single job has to be scheduled.

This one here:

And IIRC this step was added to make sure that we can handle the directory creation:

snakemake/tests/test_google_lifesciences/Snakefile

Line 39 in 7f54c39

rule directory:

I think it probably has to do with the fact that in this new function we try to calculate a size for the input directory, which won't have one / possibly hasn't been created yet.

snakemake/snakemake/scheduler.py

Line 897 in 3530f32

else:

Which ultimately gets us here:

snakemake/snakemake/jobs.py

Line 394 in 3530f32

self._inputsize = sum(f.size for f in self.input)

and then calls this size_local function on it:

snakemake/snakemake/remote/GS.py

Line 220 in 3530f32

return self._iofile.size_local

So ultimately it's this call that is wrong and isn't aware of directories perhaps?

snakemake/snakemake/io.py

Line 574 in cd5c58d

self.check_broken_symlink()

If I remember, we had to add a bunch of new logic to handle directories, and I think it would need to be added in here somewhere so we essentially don't stat something that is an input but doesn't exist yet.

That's my best guest without interacting with it directory - I would add directory handling, meaning some check using

snakemake/snakemake/io.py

Line 372 in 3530f32

@property

. It looks like there is other logic in that file for that case in other places!

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[johanneskoester]

Thanks a lot @vsoch for figuring out the issue!

[mbhall88]

I'm not sure if it's caused by these changes or not, but I've noticed recently that snakemake's status checking for cluster jobs is very delayed.

For example, in the LSF profile, if we can't get the job status using bjobs -o stat <jobid> - because it completed a while ago - then we go looking in the log file. I've noticed with a recent pipeline that has ~29k jobs that it spends a lot of time submitting jobs and then after a while, it tries to go back and check the status of the earlier jobs and they've now dropped out of the bjobs history.

I hope this makes sense? Basically I'm asking the there is some kind of new priority between submitting and status-checking?

[SichongP]

Dos bhist work? https://www.ibm.com/support/pages/finding-historical-job-information-old-completed-jobs

[mbhall88]

Sort of. It looks at the last 100 lsb.events files. Which means potentially opening 100 files. This is much slower than our current solution of looking in the job's cluster log file. Anway, that's not really the point of my question. I was just wanting to know if the submission/status priority has changed.