[QUESTION] How to start celery beat together with workers?

[duz-sg]

I'm trying to use the celery beat service to start some scheduled tasks, and publish these tasks into a different queue, so I can have a different type of worker to consume the tasks.

My current approach is to use the similar way of starting normal worker in dockerfile, however, seems I can only start the beat service, none of my workers started.

COPY ./app/worker-start.sh /worker-start.sh

RUN chmod +x /worker-start.sh

CMD ["bash", "/worker-start.sh"]

COPY ./app/checker-start.sh /checker-start.sh

RUN chmod +x /checker-start.sh

CMD ["bash", "/checker-start.sh"]

COPY ./app/beat-start.sh /beat-start.sh

RUN chmod +x /beat-start.sh

CMD ["bash", "/beat-start.sh"]

Is this the correct way of starting a new type of worker along with beat in docker? Any comment is appreciated. Thanks.

[euri10]

It's not correct imho, you'd need a beat and a worker container. You can look at a working example here: https://gitlab.com/euri10/euri10_fastapi_base/tree/master/docker-compose-dev.yml Le mer. 31 juil. 2019 à 9:21 PM, Du Zheng <notifications@github.com> a écrit :

…

I'm trying to use the celery beat service to start some scheduled tasks, and publish these tasks into a different queue, so I can have a different type of worker to consume the tasks. My current approach is to use the similar way of starting normal worker in dockerfile, however, seems I can only start the beat service, none of my workers started. COPY ./app/worker-start.sh /worker-start.sh RUN chmod +x /worker-start.sh CMD ["bash", "/worker-start.sh"] COPY ./app/checker-start.sh /checker-start.sh RUN chmod +x /checker-start.sh CMD ["bash", "/checker-start.sh"] COPY ./app/beat-start.sh /beat-start.sh RUN chmod +x /beat-start.sh CMD ["bash", "/beat-start.sh"] Is this the correct way of starting a new type of worker along with beat in docker? Any comment is appreciated. Thanks. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#46>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAINSPUDJQ5IUU27IK7ORWDQCHX6HANCNFSM4IIKHH2Q> .

[duz-sg]

It's not correct imho, you'd need a beat and a worker container. You can look at a working example here: https://gitlab.com/euri10/euri10_fastapi_base/tree/master/docker-compose-dev.yml Le mer. 31 juil. 2019 à 9:21 PM, Du Zheng notifications@github.com a écrit :
…
I'm trying to use the celery beat service to start some scheduled tasks, and publish these tasks into a different queue, so I can have a different type of worker to consume the tasks. My current approach is to use the similar way of starting normal worker in dockerfile, however, seems I can only start the beat service, none of my workers started. COPY ./app/worker-start.sh /worker-start.sh RUN chmod +x /worker-start.sh CMD ["bash", "/worker-start.sh"] COPY ./app/checker-start.sh /checker-start.sh RUN chmod +x /checker-start.sh CMD ["bash", "/checker-start.sh"] COPY ./app/beat-start.sh /beat-start.sh RUN chmod +x /beat-start.sh CMD ["bash", "/beat-start.sh"] Is this the correct way of starting a new type of worker along with beat in docker? Any comment is appreciated. Thanks. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#46?email_source=notifications&email_token=AAINSPU43XYTR73RDMJPN6DQCHX6HA5CNFSM4IIKHH22YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HCVDFZA>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAINSPUDJQ5IUU27IK7ORWDQCHX6HANCNFSM4IIKHH2Q .

Cool, will look into it. Thanks!

[tiangolo]

Thanks for the help @euri10 ! 🎉

And thanks @duz-sg for reporting back and closing the issue ✔️

[JennyCGT]

Hello @tiangolo Can you give me a piece of advice. I am trying to create a dynamic task scheduling system using FastAPI. On Django, before I have used 'django-celery-beats' in order to update dynamically task. There is any similar package for FastAPI. I really appreciate your help, I am new on FastAPI.