fix(router): retry backoff causing out-of-order job processing

[atzoum]

Description

Since job retry time is now persisted in the database and survives server restarts we need to adapt the backoff mechanism so that:

1. backoff logic is only performed if previous job status is failed and attempt number is greater than 0, and
2. if backoff is indeed performed for a job, then avoid picking subsequent jobs for the same ordering key in the same loop

Note: During server startup, jobsdb recovery updates jobs marked as executing to failed.

Notion Ticket

Link

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[fracasula]

How did you test this? I see no changes in any test file.

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.01 🎉

Comparison is base (ea3bbd5) 53.79% compared to head (0280132) 53.80%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3098      +/-   ##
==========================================
+ Coverage   53.79%   53.80%   +0.01%     
==========================================
  Files         348      348              
  Lines       53846    53847       +1     
==========================================
+ Hits        28965    28972       +7     
+ Misses      23240    23234       -6     
  Partials     1641     1641              

Impacted Files	Coverage Δ
router/router.go	77.68% <100.00%> (+0.01%)	⬆️

... and 8 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.