Skip to content

Task Queue Improvement

TeusTan edited this page Jan 8, 2020 · 1 revision

Currently, when you retrieve a task using the next (/workflows/<workflow-id>/task/next) endpoint the backend just retrieves any task that is not completed and sends it back. This is not ideal because of two reasons which are related:

  1. We want a set number of workers to processes a task to be able to have a verification model. This is essential to our product and thus we need to have a system where we distribute the same task to 3 different workers before we move on to the next task.
  2. Given the previous point, we want to keep a count on how many tasks are completed and also have an expiration for each task processing time. For instance, if we want to have each task completed by 3 workers we want to distribute each task to 3 workers before moving on to the next task, this does, however, not guarantee that all three workers will complete the task and thus we need to set an expiration time for each worker to complete a task. If the task is not completed by the expiration then the next endpoint should send the expired task again and this should repeat until each task has been completed by 3 different workers.

This problem is similar to distributed systems or event-driven systems. On that basis, we could use an existing solution to this problem.

Clone this wiki locally