Celery-node-python-example

This article serves as a boilerplate/example template for an hybrid Celery architecture using Python and Javascript (can be extended to other programming language as well)

Philosophy :

You will see how you can layout your code to maintain 2 or more development flows (here 2, one in python and the other in node)

We have a folder for python, another one for node, each is structured as much as possible as an independent project with it's own Dockerfile, Makefile.

Each of Python's and Node's folder provide a worker that listens on queue ready to be used by celery in order to perform one of the tasks defined in it's tasks package (see folder structure)

At the root of the project we have one Makefile that glue everything together by including the Makefiles from python and node folders (to build worker docker images), starting the message broker, the result-backend, the workers through a docker-compose.yml file

Project layout

• configmaps/
  • pyenvs/
• node/
  • js_workers/
    • tasks/
    • client_example.js
    • config.js
    • worker.js
  • Dockerfile
  • Makefile
  • package-lock.json
  • package.json
  • yarn.lock
• python/
  • py_workers/
    • tasks/
    • utils/
    • init.py
    • client_example.py
    • config.py
    • worker.py
  • scripts/
    • entrypoints/
  • tests/
    • init.py
    • test_py_worker.py
  • Dockerfile
  • Makefile
  • poetry.lock
  • pyproject.toml
• .gitignore
• LICENSE
• Makefile
• README.md
• docker-compose.yml

Schema, Demo & Infos

Schema

You can check the schema bellow :

Demo

You can check a complete demo video of the project in YouTube

Infos

For more complete information you can check my medium article about this pattern