User facing task queues

[ankush]

The Problem

Frappe provides a simple way to enqueue things in the background... and forget about it. It even lets you control a few important parameters like queue and timeout.

def slow_function():
    ...

frappe.enqueue(slow_function)

But sometimes you don't just want to fire and forget the job but also...

• remember that you did it
• notify user when it's completed.
• Keep user posted about progress of the task for tasks that can publish progress information.
• Make power users aware of the pending tasks queue, so they can avoid panicking when things aren't happening immediately.
• let cient side code subscribe to progress and do something when the job is completed.
• if task fails and it can be retried, let user retry it.

All this might seem overkill but everything I mentioned has multiple ad-hoc implementations in our codebase, here are just a few examples:

• Prepared Report: We enqueue, create a document, tell the user it will be read, fire a realtime event when it's ready and then act on that action in client-side code. EVERYTHING that a task queue interface should do?
• Bulk actions (delete/update): Gets enqueued and We show a blocking progressbar, this can be moved to task's UI when tasks have first class UI (similar to how notifications)
• Rename doc: User triggers enqueue, gets processed in background and client side code refreshes when rename is successful using socketio event. This is just "alert" which you can easily miss.
• Background submit: Creates a document called Submission Queue for tracking insertion, completion and UI.
• Newsletter: Sending newsletter is done from background and client side code is notified of the progress
• Data Import: Same as bulk action, we have blocking progress but it's not really required. Once started users should be allowed to move away from that page and do something else while it's going on.
• Repost Item Valuation: When required a document is created, scheduled job in background processes it and publishes updates. Here user doesn't need to know much about it, but code duplication for progress publishing can be avoided somehow using task.publish_progress or equivalent abstraction.
• Automatic payment reco
• Bulk transactions (ERPNext)

Possible implementation

• Introduce a new APIs for enqueueing, subscribing and acting on the tasks.
• Possibly a virtual doctype backed by redis or regular doctype (?)
• introduce task_id, for all practical purposes this can just be a UUID / RQ's job ID. This can be standardized in response too, so we can respond with 201 accepted and task ID for checking/fetching results later.
• Wrap RQ completely for this. Relying on RQ internals might not be a good idea in case we want to switch to something more robust and feature-rich in future (celery or even something like rabbitmq/kafka)