Support "steps" for processors so that long operations can be broken down in atomic ones

[manast]

In some jobs, several asynchronous operations need to be performed, for example storing a value in a database and updating some remote api. If the job fails and we want to retry it, the job needs to be idempotent to avoid side effects. However, sometimes is not possible to design an idempotent job, instead we could support dividing the job in steps, and if any step fails, the retry mechanism should only retry from the failed step.

[wenq1]

This should better be left in application code than in the job queue itself. You cannot satisfy all needs. Think about the the weird stacktrace appending implementation (#151) in the current codebase which at the end of the day people can do nothing about but workaround.

Your "steps" proposal sounds much likely to be fulfilled by a simple state machine. If the job is retried or attempted the first time (idempotent), depending on its state it should resume from different points. A state recording mechanism will be good enough and more generic than your "steps" proposal. Think about the case where there's a job with 3 steps. If step 3 fails one wants to retry from step 1 rather than step 3 itself. Such case is not uncommon in today's programming, and will render your "steps" proposal rather useless.

For state recording, by the way, job.updateProgress () is already doing good enough. Currently its only limitation is its numeric argument requirement. A rather interesting proposal is to have a job.updateData () function to allow for more generic data updating ().

[roggervalf]

This issue fits the current step jobs pattern https://docs.bullmq.io/patterns/process-step-jobs