An application for queueing tasks for later execution, uses Redis as message broker.
- Install
redis
, on Arch Linux do:
sudo pacman install redis
- If you are running docker related tasks then install that too.
- It uses
redis
anddocker
python client. Install them with:
sudo pip install -r requirements.txt
- Start a redis server and update configurations in
config.json
file. - Define any other required tasks in
tasks.py
. - Enqueue any task with:
from task_queue import TaskQueue
q = TaskQueue(config_file='config.json')
q.push('task_function_name', arguments)
Here arguments
is in dictionary like {'image': 'alpine', 'cmd': ['ls', '-a'], 'env': {'FOO': 'bar'}, 'resources': {'cpu': '2', 'mem': '2'}}
where value of cpu
is the percentange of CPU allowed and mem
is the memory (bytes) allowed for
this task.
- Start the worker with (you might want to run this as background process):
python worker.py <available_cpu> <available_memory>
Here available_cpu
is in percentange and available_memory
is in bytes. To run any docker related task, minimum memory limit is 4MB
- Almost all the functionality can be understood by reading examples in
tests.py
. Execute tests by:
python tests.py
For a successful execution, this should not throw any exceptions.
- If you need help setting up, please contact.
- It is assumed that if worker has started a task, it will be executed successfully.
- Currently only single channel/queue called
tasks
is implemented, should support multiples. - Since worker is executing multiple tasks in parallel, depending on the execution time, the order of execution of tasks might not remain same as they were enqueued.
- Available CPU given to worker as well as to the jobs while queueing them is in percentange i.e. multiple CPUs should be considered while queueing jobs itself.
- When any task starts executing, it is assumed it is taking all the resources that are allowed for it.