Hard time limit does not run postrun signal

[wilcoschoneveld]

Here is a minimally reproducible testcase (tested against celery 4.4.2):

from celery import Celery
from celery.signals import task_postrun
import numpy as np

app = Celery('tasks', backend='rpc://', broker='pyamqp://guest@rabbit//')
app.conf.task_time_limit = 2

@app.task
def long_running_task():
    # this should take a couple of seconds
    return np.sum(np.random.rand(20000, 20000))


@task_postrun.connect
def task_postrun_handler(**kwargs):
    print("postrun handler", kwargs)

When long_running_task is interrupted with the TimeLimitExceeded exception, it does not run the postrun handler. This seems to happen only if the task is not easily interruptible (e.g. with a heavy numpy calculation instead of a simple while loop).

Related issue: #4932

[wilcoschoneveld]

I've uploaded the minimal reproducible testcase here: https://github.com/wilcoschoneveld/celery-docker-quickstart/tree/timeout_failure (branch timeout_failure)

[auvipy]

were you able to track down the root cause of this?

[xinydev]

after hard time limit, the ForkPoolWorker is killed before it sends a postrun event.

postrun is sent from ForkPoolWorker while the kill signal is sent from main process. I think it may be a little hard to control the timing.

[wilcoschoneveld]

@auvipy no

@Hsiny it only happens when the task is not easily interruptible like a heavy numpy calculation, but not with a simple python while True loop

[teodoroanca]

Did you manage to tackle this somehow?