task: Fix checkCatalyst loop when the queues are long

[victorges]

A current bug that is happening now:

• catalyst finishes processing a task
• sends a final callback to task runner
• task runner enqueues that success so it finalizes the processing with the appropriate load balancing
• if the queues are long, this message might end up staying in the queue for a couple minutes
• during that time, there will be no updated to the updatedAt of the task, as callbacks have stopped
• when the checkCatalyst loop finally gets to it, it will see a task that has had no updates for minutes-long, treat it as lost and mark it as errored on the API
• error only gets worse cause the API will then retry the task which has the same error again

This fixes it by improving the exit condition of the checkCatalyst step. Instead of only finishing it once
the task has stoped running, we will now update the task step on the API and make sure to stop the
checkCatalyst loop as soon as the task gets to the finalize step. We make sure to update the step
from the catalyst callback handler already, so it happens early and stops the checkCatalyst loop, not
only when the finalization event gets processed.

Fixes STU-29

[linear]

STU-29 Fix task-runner checkCatalyst logic when queue is long

[thomshutt]

LGTM, is there a similar situation where Catalyst API reports a terminal error?

[victorges]

@thomshutt in case of errors we don't need the finalization step. So we just publish a task result which goes to a different queue and is processed by the API instead.

While debugging this i did notice one interesting scenario tho: currently the delay we enforce between API retries is literally just a sleep in the API code. It holds a message for a while without a king it until it schedules the retry. In case of too many tasks retrying, it might be the case that all API consumers are busy sleeping until a retry. This can cause a delay in processing task results and retries. But it shouldn't be that bad anyway as we have 12 API instances per region, maybe 2 consumers per instance. Either way something to keep in mind and potentially refactor our retry logic or increase number of consumers etc cc @gioelecerati