Job killed mid running not gracefully causes infinite loop

[rstalbow]

Describe the bug
A worker is killed mid running a job and the shutdown is not graceful. After this happens you get a 500 error accessing admin/scheduler/queue/ and one of admin/scheduler/{repeatablejob/cronjob/scheduledjob}/ depending on what type of job was killed when running. The exceptions are

RecursionError
maximum recursion depth exceeded while calling a Python object

and
Job repeatable_jobs:{jobname}:{jobid}: error while executing failure callback

This error is due to this change. The self.rqueue.started_job_registry.get_job_ids() calls cleanup which calls is_scheduled thus an infinite loop.

To Reproduce
Steps to reproduce the behavior:

1. First you need a long running job. For example:

@job()
def test():
    import time
    time.sleep(30)

2. Setup the job to run via the admin.
3. When the job is active (by checking the queue admin/scheduler/queue/) kill the worker process (for example by pkill -9 python)
4. The either:
   • Bring the worker back up and wait for the maintenance tasks to run
   • Run the clean_registries function from a python django shell

Expected behavior
The job gets - Moved to FailedJobRegistry, due to AbandonedJobError. This happens to the job if you revert back to version 1.0.2 but then you have this issue again.

Implementation idea

1. In the is_scheduled function when running self.rqueue.started_job_registry.get_job_ids() have an option to not run the cleanup inside of get_job_ids. You shouldn't need to run the cleanup from the is_scheduled method.
2. Revist the fix for [issue](Job becomes unscheduled if you access the admin list page when the job is active #32 and instead of adding in self.rqueue.started_job_registry.get_job_ids() remove the

self.job_id = None
super(BaseJob, self).save()

[cunla]

I reverted the change in #32.
Though it is not likely that rq classes call django-tasks-scheduler methods.

[rstalbow]

@cunla I don't follow what you are saying regarding tq classes. Now that you have reverted I wouldn't be able to use the latest version as the original bug in 32 is a major problem for use.

What do you think to the 2 implementation ideas above? I can make an MR on either tomorrow.

[cunla]

Please provide the full stack trace of the error you are getting

[rstalbow]

@cunla which error?

[cunla]

This error is due to #33 change. The self.rqueue.started_job_registry.get_job_ids() calls cleanup which calls is_scheduled thus an infinite loop.

This does not make sense, can you point me to where cleanup calls is_scheduled?

[cunla]

and you are getting http500, if you activate debug mode, you should get a stacktrace.

[rstalbow]

In Sentry it says Job repeatable_jobs:{job name}:{job id}: error while executing failure callback

On of the many Stacktrace's I see in Opensearch

Traceback (most recent call last):
  File "/home/nobody/.local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/core/handlers/base.py", line 220, in _get_response
    response = response.render()
               ^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/sentry_sdk/integrations/django/views.py", line 38, in sentry_patched_render
    return old_render(self)
           ^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/response.py", line 114, in render
    self.content = self.rendered_content
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/sentry_sdk/integrations/django/templates.py", line 75, in rendered_content
    return real_rendered_content.fget(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/response.py", line 92, in rendered_content
    return template.render(context, self._request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/backends/django.py", line 61, in render
    return self.template.render(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 175, in render
    return self._render(context)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 167, in _render
    return self.nodelist.render(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/loader_tags.py", line 157, in render
    return compiled_parent._render(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 167, in _render
    return self.nodelist.render(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/loader_tags.py", line 157, in render
    return compiled_parent._render(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 167, in _render
    return self.nodelist.render(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/loader_tags.py", line 157, in render
    return compiled_parent._render(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 167, in _render
    return self.nodelist.render(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/loader_tags.py", line 63, in render
    result = block.nodelist.render(context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/loader_tags.py", line 63, in render
    result = block.nodelist.render(context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/contrib/admin/templatetags/base.py", line 45, in render
    return super().render(context)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/template/library.py", line 258, in render
    _dict = self.func(*resolved_args, **resolved_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/contrib/admin/templatetags/admin_list.py", line 336, in result_list
    "results": list(results(cl)),
               ^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/contrib/admin/templatetags/admin_list.py", line 312, in results
    yield ResultList(None, items_for_result(cl, res, None))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/contrib/admin/templatetags/admin_list.py", line 303, in __init__
    super().__init__(*items)
  File "/home/nobody/.local/lib/python3.11/site-packages/django/contrib/admin/templatetags/admin_list.py", line 213, in items_for_result
    f, attr, value = lookup_field(field_name, result, cl.model_admin)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/django/contrib/admin/utils.py", line 285, in lookup_field
    value = attr()
            ^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():
       ^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job

RecursionError: maximum recursion depth exceeded while calling a Python object

[rstalbow]

This does not make sense, can you point me to where cleanup calls is_scheduled?

You can see in the above stacktrace:

 File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 91, in is_scheduled
    active_jobs = self.rqueue.started_job_registry.get_job_ids()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 152, in get_job_ids
    self.cleanup()
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/registry.py", line 238, in cleanup
    job.execute_failure_callback(
  File "/home/nobody/.local/lib/python3.11/site-packages/rq/job.py", line 1429, in execute_failure_callback
    self.failure_callback(self, self.connection, *exc_info)
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 34, in callback_save_job
    scheduled_job.unschedule()
  File "/home/nobody/.local/lib/python3.11/site-packages/scheduler/models/scheduled_job.py", line 210, in unschedule
    if self.is_scheduled():

[cunla]

fixed on 1.2.1

[rstalbow]

This change might have fixed one issue but its caused a bigger issue.
We have our scheduler interval configured like this:

SCHEDULER_CONFIG = {
    "SCHEDULER_INTERVAL": 1  # In seconds
}

The jobs now run with an interval of 1 second regardless of whether they are enabled or not and ignores the interval defined in the job.

Screenshot above shows the job is scheduled to run every hour.

Screenshot above shows the job is running every second

I believe the issues is in the callback_save_job where you have added force_schedule. Inside of force_schedule its runs self._schedule_time(). This is the datetime of the very first time it was set to run. It then does an enqueue_at at and as the schedule_time is now in the past it runs it. callback_save_job is run everytime the job is run.

@cunla reopen this issue or create a new one?

[cunla]

you can create a new one. if you can write a test to replicate this it would help fix it.