Awaitable task result

[bblanchon]

Checklist

• I have checked the issues list
  for similar or identical feature requests.
• I have checked the pull requests list
  for existing proposed implementations of this feature.
• I have checked the commit log
  to find out if the if the same feature was already implemented in the
  master branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

• How to use AsyncResult.then in async/await #4353

Possible Duplicates

• How to use AsyncResult.then in async/await #4353

Also related:

• How to combine Celery with asyncio? (StackOverflow question)

Brief Summary

Being able to await an AsyncResult

The goal is to offload CPU intensive tasks to dedicated workers and return a result to the client once the task is completed.

Design

Architectural Considerations

I don't know how to implement this feature, but it seems that this project does:
https://github.com/kai3341/celery-pool-asyncio
However, it's not compatible with Celery 5 (see kai3341/celery-pool-asyncio#29)

Proposed Behavior

Proposed UI/UX

In a Django 3.1 async view:

async def my_view(request):
    result = await my_cpu_intensive_task.delay()
    return HttpResponse(f"Result = {result}")

Diagrams

N/A

Alternatives

I've considered polling AsyncResult.state every second, but it's a very inelegant solution.

[auvipy]

we can try this in django-celery-results along with celery core as a POC first. but it is a part of a bigger design decision which we are planning for celery 6.0. additionally we can try to use an aio-libs in one of our result back end and see how things need to be done as experiments.

[philmday]

this would be a very useful feature.

[bblanchon]

As I wrote on stackoverflow, I currently use the following helper as a workaround:

import asyncio
from asgiref.sync import sync_to_async

# Converts a Celery tasks to an async function
def task_to_async(task):
    async def wrapper(*args, **kwargs):
        delay = 0.1
        async_result = await sync_to_async(task.delay)(*args, **kwargs)
        while not async_result.ready():
            await asyncio.sleep(delay)
            delay = min(delay * 1.5, 2)  # exponential backoff, max 2 seconds
        return async_result.get()
    return wrapper

Like sync_to_async, it can be used as a direct wrapper:

@shared_task
def get_answer():
    sleep(10) # simulate long computation
    return 42    

result = await task_to_async(get_answer)()

...and as a decorator:

@task_to_async
@shared_task
def get_answer():
    sleep(10) # simulate long computation
    return 42    

result = await get_answer()

Of course, this is not a perfect solution since it relies on polling, but it is a good workaround for calling tasks from Django async views until Celery officially provides a better solution.

[thedrow]

We're working on it.
Anyone is welcome to join the Jumpstarter task force.
https://github.com/celery/jumpstarter

[amacfie]

We're working on it.
Anyone is welcome to join the Jumpstarter task force.
https://github.com/celery/jumpstarter

@thedrow can you explain more about what Jumpstarter is?

[NoraGithub]

we can try this in django-celery-results along with celery core as a POC first. but it is a part of a bigger design decision which we are planning for celery 6.0. additionally we can try to use an aio-libs in one of our result back end and see how things need to be done as experiments.

https://github.com/celery/django-celery-results may be this one