Parallel computing using Celery - threador

Install library:

pip install threador

Step 1

Init docker container with RabbitMQ

docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management

Step 2

Create fite `threador_config.py` in your root place at the project.

import time
from celery import Celery
from threador.tasks import Task

app = Celery('threador_celery', backend='rpc://', broker='pyamqp://guest@localhost//')

// Your task for parallel execution
class SleepTask(Task):
    def fnc(self, *args, **kwargs):
        _r = kwargs.get('timeout', 0)
        time.sleep(_r)
        return _r

// Register all your task here
tasks = {
    'sleep': SleepTask(),
}

// Required function for call your task parallel
@app.task
def call_task(fnc_name: str, *args, **kwargs):
    fnc = tasks.get(fnc_name, None)
    if fnc is not None:
        result = fnc(*args, **kwargs)
        return result

Step 3

Run celery for correct working.

celery -A threador_config worker --loglevel=debug -n threador_celery@parallel

Step 4

Using parallel computing in code.

Result will be order by position in tasks.

from threador.contrib import Executor
parallel = Executor(tasks=(
    ['sleep', None, {'timeout': 3}],
    ['sleep', None, {'timeout': 2}],
    ['sleep', None, {'timeout': 4}],
))
result = parallel.run()
print(result)