Parallel gru jobs

[kraih]

This is the final step for converting Gru into a normal Minion worker. All tasks have been adjusted to declare their own limits for parallel jobs. And the one job at a time limit has been lifted from the Gru worker (default is 4 parallel jobs, which can be adjusted with command line flags).

Task adjustements (based on feedback from IRC):

• limit_assets: no change if one job is still running new jobs of the type will be discarded
• download_asset: no change checks if the file already exists and discards new jobs if it does
• limit_results_and_logs: if one job is still running new jobs of the type will be discarded
• migrate_images/relink_testresults/rm_compat_symlinks: tasks have been removed since they were only used for very old migrations (more old migration code will be removed in the future)
• scan_needles: if one job is still running new jobs of the type will be discarded
• scan_images/scan_images_links: shared lock, if one job is still running new jobs will be retried with a 30 second delay

Progress: https://progress.opensuse.org/issues/45290.

[Martchus]

Looks good. I'll try to base my PR for needle commits on it.

[codecov]

Codecov Report

Merging #1984 into master will increase coverage by 0.46%.
The diff coverage is 37.5%.

@@            Coverage Diff             @@
##           master    #1984      +/-   ##
==========================================
+ Coverage   88.99%   89.45%   +0.46%     
==========================================
  Files         154      153       -1     
  Lines       10404    10349      -55     
==========================================
- Hits         9259     9258       -1     
+ Misses       1145     1091      -54

Impacted Files	Coverage Δ
lib/OpenQA/WebAPI/Command/gru/run.pm	92.3% <ø> (-0.55%)	⬇️
lib/OpenQA/Task/Job/Limit.pm	45.45% <0%> (-4.55%)	⬇️
lib/OpenQA/Task/Screenshot/Scan.pm	12.85% <0%> (-0.38%)	⬇️
lib/OpenQA/Task/Needle/Scan.pm	100% <100%> (ø)	⬆️
lib/OpenQA/WebAPI/Plugin/Gru.pm	100% <100%> (ø)	⬆️
lib/OpenQA/Utils.pm	92.38% <0%> (+0.17%)	⬆️
lib/OpenQA/Worker/Common.pm	84.15% <0%> (+2.11%)	⬆️
lib/OpenQA/Worker/Commands.pm	83.52% <0%> (+2.35%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 47070d7...ffcddb2. Read the comment docs.

[codecov]

Codecov Report

Merging #1984 into master will increase coverage by 0.46%.
The diff coverage is 37.5%.

@@            Coverage Diff             @@
##           master    #1984      +/-   ##
==========================================
+ Coverage   88.99%   89.45%   +0.46%     
==========================================
  Files         154      153       -1     
  Lines       10404    10349      -55     
==========================================
- Hits         9259     9258       -1     
+ Misses       1145     1091      -54

Impacted Files	Coverage Δ
lib/OpenQA/WebAPI/Command/gru/run.pm	92.3% <ø> (-0.55%)	⬇️
lib/OpenQA/Task/Screenshot/Scan.pm	12.85% <0%> (-0.38%)	⬇️
lib/OpenQA/Task/Job/Limit.pm	45.45% <0%> (-4.55%)	⬇️
lib/OpenQA/WebAPI/Plugin/Gru.pm	100% <100%> (ø)	⬆️
lib/OpenQA/Task/Needle/Scan.pm	100% <100%> (ø)	⬆️
lib/OpenQA/Utils.pm	92.38% <0%> (+0.17%)	⬆️
lib/OpenQA/Worker/Common.pm	84.15% <0%> (+2.11%)	⬆️
lib/OpenQA/Worker/Commands.pm	83.52% <0%> (+2.35%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 47070d7...334f53a. Read the comment docs.

[Martchus]

Likely @coolo should approve it, too.

In the meantime I can still base my other PR for Git on this and test it further.

[coolo]

please remove all of the old migrations, not just single scripts. Last time was 40bdcd9 I removed < 2016, I think it's time we drop < 2018

[kraih]

@coolo Think i've removed all up to 2018, schema version 63 seems to be the first of the year.

[coolo]

< 63 should do, thanks