[9.x] Make it so non-existent jobs run down the failed path instead of crashing

[hdimitrov1]

Hi,

the issue that this is solving is that if you have a queued job (ie in SQS), that you have deleted (or renamed), the runner that picks up the job will fail with an Error.

That means that instead of going down the failed job path and dequeuing the job, it will crash, and keep retrying the job over and over again (in our case 361k times before we ran out of error log allowance).

This makes it instead fail with an Exception and go down the failed job path, so the job dequeues and doesn't keep retrying.

For us this happened to a job that was queued before a deployment in which it was renamed.

[antonkomarev]

I think this is not a best path.
By this way you may lose job and data unexpectedly because it's not so obvious :(

Async jobs renaming may be safely done in multiple steps:

1. Add job with new name and start dispatching it; remove old job dispatching
2. Wait until all old jobs in queue will be processed
3. Delete old job class

@taylorotwell

[hdimitrov1]

This should still give you an exception and a failed job job so you can rerun it, it just shouldn't infinitely loop while trying to do it - same behaviour as if your job failed for another reason.

[taylorotwell]

@antonkomarev this PR doesn't prevent that - right? This just prevents infinite retry loops.