fix: fix clocked schedule #341

yywing · 2020-05-14T09:40:12Z

Clocked schedule is stateless.
It can run once because of PeriodicTask.one_off.

auvipy

can you explain more of your reason for this approach?

auvipy · 2020-05-14T10:09:05Z

django_celery_beat/models.py

@@ -186,12 +186,6 @@ class ClockedSchedule(models.Model):
        verbose_name=_('Clock Time'),
        help_text=_('Run the task at clocked time'),
    )
-    enabled = models.BooleanField(


this is a breaking change

can you explain more of your reason for this approach?

I maybe handle it before this weekends(timezone: Beijing). too busy now.

we can introduce swapalbe/ abstract base model as Django-activity-stream did

Or we can keep it but it always True.

what about default=False?

But it act as always True.

yywing · 2020-06-22T11:23:44Z

I will work on this PR.
And I maybe pull a new feature for mult time schedule.

yywing · 2020-06-23T02:54:44Z

yywing · 2020-06-23T03:26:25Z

is _due return None will make schedule stuck. Return None will never update heap first object.
Clocked schedule changed clocked schedule model, and it will make schedule reload. As picture I upload.
1. I create many clocked schedule which work on the same time.
2. first clocked schedule run ，and change model，and schedule reload.
3. reload wiled call is_due and all clocked schedule set to false, but it olny called one
4. schedule change and all schedule reload, all scheled set false at 2),so never called.

So only two task run.

yywing · 2020-06-23T03:36:58Z

I change clocked model, remove enabled field, use PeriodicTask.one_off.

superandrew · 2020-07-06T21:26:54Z

I am looking forward for this PR, @yywing it is great that you have been able to track, understand fix one of the most hideous and awkward bugs of this extension.

superandrew · 2020-07-06T21:36:22Z

just to add some hype: pip install git+https://github.com/yywing/django-celery-beat/@f2bdc5214b3c70ee3805654855ab08e55452df69
fixes the problem reported on #310

superandrew · 2020-07-13T09:47:50Z

hi, any news on when this PR is going to be released?

auvipy · 2020-07-14T12:24:18Z

there are some concerns regarding backward compat

superandrew · 2020-07-14T12:29:36Z

too bad. My experience with my use case and django_celery_beat was that everything worked for days before stop working randomly (then I understood it was not literally randomly but there was a reason), and this fix definitely fixed everything. So I really hope this could go in production as soon as possible. I am not confident with celery or this package, but if I can help with some real world testing I'd be more than happy to do so.

auvipy · 2020-07-14T12:32:26Z

please do some more real-world testing and report back

superandrew · 2020-07-17T06:34:46Z

my production environment started using this package about 10 days ago.

We were able to run hundreds of task, and the scheduled tasks never stopped once.

Before this fix, after a few days, some condition would make celery beat block on one task and stop process the tasks all together.

I hope this helps. Please let me know if there is anything specific I could test.

sfdye · 2020-08-06T08:08:22Z

Is it possible to release this fix please?

auvipy · 2020-08-06T21:20:56Z

Next week Hopefully

adl-asi · 2020-08-25T13:17:23Z

@auvipy Will this be released soon? Trying to use clocked schedule in a production setting and running into issues

adl-asi · 2020-09-08T13:14:08Z

Just checking in again. Is there an eta for when this fix will be released?

auvipy · 2020-09-12T13:48:25Z

hope full this week. ping me again

saethor · 2020-09-18T10:36:26Z

Any updates about this?

belongwqz · 2020-09-19T03:16:12Z

still waiting

Lotram · 2020-09-22T08:00:44Z

@auvipy any news?

revoteon · 2020-09-30T00:18:00Z

@auvipy Any updates?

ghazi-git · 2020-10-20T13:00:28Z

I really wish I'd seen this PR earlier. I started using clocked schedule last month. Like everyone else, things started working fine and then at some point, the clocked tasks stopped being processed. What I found out is that whenever there is one task where PeriodicTask.enabled is True but ClockedSchedule.enabled is False, all tasks after it are never executed. So, the question became how did I end up in a situation where the task PeriodicTask.enabled is True but ClockedSchedule.enabled is False.

Later I was able to reproduce it consistently by scheduling a task, then stopping celery beat until the scheduled time of the task has passed. When celery beat is brought back up after that, the scheduled task, that should have executed when celery beat was down, is picked up and gets executed. So, ClockedSchedule.enabled is set to False, but then PeriodicTask.enabled is never changed to False, which results in the next tasks not executed. In production, that might be translated to "a clocked task needs to be scheduled to execute when there is some downtime/maintenance for the issue to start happening"

Anyway, right before deciding to switch to one-off tasks using crontab schedule, I saw that there is a new version and when checking the commits , I found this PR. Now, that ClockedSchedule.enabled is removed, I installed the new version and retested the issue and it is no longer happening. So, thank you @yywing for fixing this and @auvipy for releasing the fix.

sfdye · 2020-10-21T02:49:09Z

Looks like 2.1.0 was released yesterday
https://pypi.org/project/django-celery-beat/#history

yywing force-pushed the fix_clocked branch from ceeefa1 to 601b3dc Compare May 14, 2020 09:44

yywing mentioned this pull request May 14, 2020

Clocked periodic tasks doesn't run if they are in same time #283

Closed

auvipy requested changes May 14, 2020

View reviewed changes

yywing force-pushed the fix_clocked branch from 601b3dc to d67cbb3 Compare May 18, 2020 10:22

bob-r mentioned this pull request Jun 2, 2020

troubleshooting a database scheduler which does nothing at all #310

Closed

vbabiy mentioned this pull request Jun 12, 2020

Running one-off task stops all periodical tasks #150

Closed

yywing force-pushed the fix_clocked branch from 7df22f8 to 71a2c47 Compare June 23, 2020 02:58

fix: fix clock schedule and one off task stuck

f2bdc52

yywing force-pushed the fix_clocked branch from cfe9ab8 to f2bdc52 Compare June 23, 2020 07:29

auvipy approved these changes Jul 24, 2020

View reviewed changes

auvipy merged commit 28ba977 into celery:master Jul 24, 2020

yujinyuz mentioned this pull request Nov 10, 2020

Revert back to ClockedSchedule bytedeck/bytedeck#703

Merged

auvipy linked an issue Feb 16, 2021 that may be closed by this pull request

troubleshooting a database scheduler which does nothing at all #310

Closed

dependabot bot mentioned this pull request Mar 17, 2021

Bump django-celery-beat from 2.0.0 to 2.2.0 pkuphysu/MeetPlan#12

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: fix clocked schedule #341

fix: fix clocked schedule #341

yywing commented May 14, 2020

auvipy left a comment

auvipy May 14, 2020

yywing May 15, 2020

auvipy May 15, 2020

yywing Jun 24, 2020

auvipy Jul 14, 2020

yywing Jul 24, 2020

yywing commented Jun 22, 2020

yywing commented Jun 23, 2020

yywing commented Jun 23, 2020 •

edited

yywing commented Jun 23, 2020

superandrew commented Jul 6, 2020

superandrew commented Jul 6, 2020

superandrew commented Jul 13, 2020

auvipy commented Jul 14, 2020

superandrew commented Jul 14, 2020

auvipy commented Jul 14, 2020

superandrew commented Jul 17, 2020

sfdye commented Aug 6, 2020

auvipy commented Aug 6, 2020

adl-asi commented Aug 25, 2020 •

edited

adl-asi commented Sep 8, 2020

auvipy commented Sep 12, 2020

saethor commented Sep 18, 2020

belongwqz commented Sep 19, 2020

Lotram commented Sep 22, 2020

revoteon commented Sep 30, 2020

ghazi-git commented Oct 20, 2020

sfdye commented Oct 21, 2020

fix: fix clocked schedule #341

fix: fix clocked schedule #341

Conversation

yywing commented May 14, 2020

auvipy left a comment

Choose a reason for hiding this comment

auvipy May 14, 2020

Choose a reason for hiding this comment

yywing May 15, 2020

Choose a reason for hiding this comment

auvipy May 15, 2020

Choose a reason for hiding this comment

yywing Jun 24, 2020

Choose a reason for hiding this comment

auvipy Jul 14, 2020

Choose a reason for hiding this comment

yywing Jul 24, 2020

Choose a reason for hiding this comment

yywing commented Jun 22, 2020

yywing commented Jun 23, 2020

yywing commented Jun 23, 2020 • edited

yywing commented Jun 23, 2020

superandrew commented Jul 6, 2020

superandrew commented Jul 6, 2020

superandrew commented Jul 13, 2020

auvipy commented Jul 14, 2020

superandrew commented Jul 14, 2020

auvipy commented Jul 14, 2020

superandrew commented Jul 17, 2020

sfdye commented Aug 6, 2020

auvipy commented Aug 6, 2020

adl-asi commented Aug 25, 2020 • edited

adl-asi commented Sep 8, 2020

auvipy commented Sep 12, 2020

saethor commented Sep 18, 2020

belongwqz commented Sep 19, 2020

Lotram commented Sep 22, 2020

revoteon commented Sep 30, 2020

ghazi-git commented Oct 20, 2020

sfdye commented Oct 21, 2020

yywing commented Jun 23, 2020 •

edited

adl-asi commented Aug 25, 2020 •

edited