Cancel runs if no progress is made in the manifest

[ayazhafiz]

Presently, active runs can reach a state where all associated workers die, and no progress is made on the test suite, but the test suite sticks around in the queue memory. Since such runs may not be returned to at all, we'd like to diminish the amount of pressure they might place on a running queue.

This series of patches addresses the problem by running a job every hour that checks whether test runs have had any progress in their manifest. If either

• there is no manifest associated with the run after an hour, or
• no more items have been popped off the manifest since the last time progress was checked

then the run will be cancelled. If progress has been made, or the run was already done, the run is left untouched. If progress has been made and the run is not yet done, a job to check the progress again later is re-enqueued.

In the future, we'll likely want to adjust the behavior to not outright cancel a job, but to admit some way to re-launch the job from the last failure state.

[github-actions]

✅ Bigtest for 9537e32 (run)

Benchmarks:

• RSpec: 11.24% overhead
  • RSpec time: 17.79 seconds
  • ABQ time: 19.79 seconds
• RSpec parallel, 10 runs: max 15.29% overhead
  • min 6.69% overhead
  • standard deviation: 2.87%
• Jest: 5.58% overhead
  • Jest time: 21.16 seconds
  • ABQ time: 22.341 seconds

Fuzz result sizes:

• PASSED

[github-actions]

✅ Bigtest for 66dac76 (run)

Benchmarks:

• RSpec: 15.57% overhead
  • RSpec time: 17.73 seconds
  • ABQ time: 20.49 seconds
• RSpec parallel, 10 runs: max 12.63% overhead
  • min 6.66% overhead
  • standard deviation: 1.51%
• Jest: 6.11% overhead
  • Jest time: 21.362 seconds
  • ABQ time: 22.667 seconds

Fuzz result sizes:

• PASSED