[feat] Retry job submission with Slurm on error conditions specified by the user

[ekouts]

Second, much simpler implementation of #1672. In this PR the submission blocks until slurm stops giving the QOSMaxSubmitJobPerUserLimit error. This could cause significant slowdown of the jobs.
Closes #1595.

[pep8speaks]

Hello @ekouts, Thank you for updating!

Cheers! There are no PEP8 issues in this Pull Request!Do see the ReFrame Coding Style Guide

Comment last updated at 2021-02-01 20:22:12 UTC

[vkarak]

We will also need documentation.

[codecov-io]

Codecov Report

Merging #1704 (920593b) into master (391ef9e) will decrease coverage by 0.01%.
The diff coverage is 10.00%.

@@            Coverage Diff             @@
##           master    #1704      +/-   ##
==========================================
- Coverage   87.30%   87.29%   -0.02%     
==========================================
  Files          46       46              
  Lines        7649     7704      +55     
==========================================
+ Hits         6678     6725      +47     
- Misses        971      979       +8     

Impacted Files	Coverage Δ
reframe/core/schedulers/slurm.py	52.73% <10.00%> (-1.05%)	⬇️
reframe/frontend/statistics.py	95.31% <0.00%> (-1.41%)	⬇️
reframe/core/schedulers/local.py	95.79% <0.00%> (-0.07%)	⬇️
reframe/frontend/executors/__init__.py	98.19% <0.00%> (+0.09%)	⬆️
reframe/core/exceptions.py	94.20% <0.00%> (+0.12%)	⬆️
reframe/frontend/executors/policies.py	99.67% <0.00%> (+0.34%)	⬆️
reframe/core/shell.py	95.45% <0.00%> (+0.45%)	⬆️
reframe/frontend/cli.py	76.79% <0.00%> (+0.77%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 391ef9e...920593b. Read the comment docs.

[vkarak]

I will also do some minor adaptations to the documentation.

[vkarak]

Works fine.