fixing issues with workflow on cluster with slurm (closes #369, #368)

[djarecka]

Acknowledgment

• I acknowledge that this contribution will be available under the Apache 2 license.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Summary

this started from #369

• adding tmpdir to tests run with slurm, so pytest could be run with --basetemp on a cluster
• fixing issue with propagating cache_dir for workflows that are used as nodes in another workflow
• addressing issue that sometimes a task might be finished, but no results are written yet, so the submitter can't get new available tasks. Adding a while loop that will try to get those tasks for another 60s, and raise an error if not possible. I've removed the old exception ("Nothing queued or todo - something went wrong") that sometimes was raised, I believe this was raised when due to this issue, so should be avoided or a new exception will be raised

Checklist

• All tests passing
• I have added tests to cover my changes
• I have updated documentation (if necessary)
• My code follows the code style of this project
  (we are using black: you can pip install pre-commit,
  run pre-commit install in the pydra directory
  and black will be run automatically with each commit)

[codecov]

Codecov Report

Merging #376 (7669b81) into master (581545c) will decrease coverage by 0.11%.
The diff coverage is 41.66%.

@@            Coverage Diff             @@
##           master     #376      +/-   ##
==========================================
- Coverage   82.41%   82.29%   -0.12%     
==========================================
  Files          19       19              
  Lines        3741     3751      +10     
  Branches     1015     1019       +4     
==========================================
+ Hits         3083     3087       +4     
- Misses        471      477       +6     
  Partials      187      187              

Flag	Coverage Δ
unittests	82.24% <41.66%> (-0.12%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pydra/engine/submitter.py	84.35% <41.66%> (-3.24%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 581545c...7669b81. Read the comment docs.