Jobs terminated by Worker report as "Running" forever

[glennmatthews]

Environment

• Python version:
• Nautobot version: 1.0.3

Steps to Reproduce

1. Run a Job that allocates more memory than is actually available (e.g. with the worker running in a memory-limited Docker container) so that it gets killed by the system.
2. Nautobot JobResult gets stuck as "Running" and never shows as completed in the user UI
3. An admin can access admin/background-tasks/ and see that the RQ task associated with the Job was killed, but this information is not captured in the JobResult or otherwise visible to non-admin users.

Similar behavior is likely to be seen if a job runs for longer than the configured maximum timeout and is killed by RQ because of that.

I recognize that this specific symptom may be changed as we move to replace RQ with Celery (#531) but these same sorts of scenarios likely need to be accounted for with the Celery worker as well.

Expected Behavior

Nautobot needs to be made aware when a worker task fails or aborts and update the JobResult accordingly. For RQ, the docs have some possible approaches; I'm sure there are similar options for Celery.

Observed Behavior

[jathanism]

As RQ is deprecated we won't need to solve for that.

We addressed the timeout scenario w/ the Celery workers by implementing soft/hard time limits. But what about if a Celery worker dies/disappears, where are we expecting the call flow to result in the job getting updated by something else? (Such as the connective tissue between the Celery task state and the JobResult object).

[glennmatthews]

Thinking a bit more about this, this may be something we want to address once #275 / #374 get implemented - have a periodic task that looks for JobResults that are marked as Running but aren't actually being used by any task, perhaps?

[lampwins]

See #765

[bryanculver]

See #1622 as solving this.

[jathanism]

As a follow-up even with the work in #1622, when a hard time_limit is reached, Celery does a SIGKILL (-9) on the worker process executing the long-running task, leaving the JobResult in a "hung" state, not receiving a status update or even storing a traceback or any other incremental updates. Internally, a mark_as_failure event is fired on the database backend, but the process is killed before the event can culminate and update the backend correctly.

The only real workaround here is making proper use of soft_time_limit on jobs that are known to take a long time an have them catch SoftTimeLimitExceeded and clean up appropriately as we have recommended in our documentation.

Longer term, there may be a way to solve this by use of subtasks when a hard time-limit is detected, but in the current default implementation this is results in the JobResult being out of phase with the task being terminated. As far as I know, this is the only lingering edge case here, and it is one that should be incredibly rare in practice given proper use of soft_time_limit by Job authors.

[bryanculver]

@jathanism can we close this because of #3085 and #3084?

[jathanism]

@jathanism can we close this because of #3085 and #3084?

Yep!