Dynamic update of the celery_crontab_schedule table?

[xtiansimon]

Hello,
I've installed this package. The echo task examples are all running as expected. I added my own toy task that writes a test file to a directory--All good. Great project. Thank you.

Reading your code and examples I wish for one feature not mentioned in documentation. I am searching for a solution to support my Flask project. I need to dynamically change (create, update, delete) the SQL tables in celery-sqlalchemy-scheduler without restarting Flask, Celery workers, or Celery Beat.

I don't need to add new functions to tasks.py. I just need to add new task instances and change schedules.
For example,

• Add a new task row (create new tasks.add with args [1,5]) in the celery_periodic_task table.
• Update row feature args from [1,2] to [10,2] in the celery_periodic_task table.
• Update row with features of the celery_crontab_schedule table, and change (0 4 * * *) to (* */4 * * *).

Can I update tasks in the SQL tables to achieve 'dynamic' updates with this project?

There is another older package celery_sqlalchemy_scheduler which promised to 'dynamically add tasks at runtime' and this package is based on Celery v3. I was happy to find your project using Celery v4, because I understand v4 has added additional decorators to update Celery Beat at runtime--if I understand this correctly, but I'm still a little new to the guts and gears of Celery. It's all very complex.

Thanks again. Great project.

[AngelLiang]

Yes! Thank you for raising the issue.

You can update tasks in the SQL tables to achieve 'dynamic' updates the schedule tasks.

There is an example in proejct, you can quickstart as follow:

Environment:

• Windows PowerShell
• Python 36+
• RabbitMQ ('amqp://guest:guest@127.0.0.1:5672//')

First, download the repo and enter pipenv shell environment

$ git clone git@github.com:AngelLiang/celery-sqlalchemy-scheduler.git
# $ git checkout db8e265
$ cd celery-sqlalchemy-scheduler

# ensure install pipenv at first
$ pipenv install
$ pipenv shell

Second, run the celery worker and celery beat for example:

# console 1 , in pipenv shell
$ cd examples/base
$ celery worker -A tasks:celery -l info

Create console 2 to run celery beat:

# console 2, also in pipenv shell
$ cd examples/base
$ celery beat -A tasks:celery -S tasks:DatabaseScheduler -l info

After run celery beat, console 2 wil print as follow:

[2020-05-08 12:01:14,238: INFO/MainProcess] Scheduler: Sending due task echo-every-3-seconds (tasks.echo)
[2020-05-08 12:01:17,213: INFO/MainProcess] Scheduler: Sending due task echo-every-3-seconds (tasks.echo)

Third, create Console 3 in pipenv shell:

# console 3, in pipenv shell
$ cd examples/base
$ python -m doctest tasks.py

After run python -m doctest tasks.py, it will disable the echo-every-3-seconds task and add add-every-10s task.

[2020-05-08 12:04:57,847: INFO/MainProcess] Scheduler: Sending due task echo-every-3-seconds (tasks.echo)
[2020-05-08 12:04:59,993: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2020-05-08 12:04:59,994: INFO/MainProcess] Writing entries...
[2020-05-08 12:05:10,149: INFO/MainProcess] Scheduler: Sending due task add-every-10s (tasks.add)

It don't need to restart celery worker or celery beat to update scheduler task.

Is this solved your problem? You can read example/base/task.py for more detail.

Thanks.

BTW, django-celery-beat maybe help for you though it is for django.

[xtiansimon]

Your new Python interpreter example worked exactly as I hoped. Thank you. (^_^)//

Celery is a complex project. Adapting Celery and your module to the Flask app context will be challenging.

Do you have any experiencing implementing your module in a Flask app?
For example, Should I add these tables to my existing Flask app's database (login users, etc), or should Celery-SQLAlchemy-Scheduler remain separate database instance? Are you aware of any problems with this idea (or any other weak points of failure)?

[AngelLiang]

Thank you for your compliment.

I think Flask is flexibly. So what should you do is not important. And now there are no best practices for Flask with celery-sqlalchemy-scheduler.

You can use same datebase with flask app, or use a separate database instance for celery-sqlalchemy-scheduler.

Maybe flask-celery-example is help for you.

[xtiansimon]

"...there are no best practices for Flask with celery-sqlalchemy-scheduler."
You stated my question in a better way. Ok. Good to know.