Celery inspect ping doesn't respond if workers are busy running a task

[hassant4]

Hi, I run my nodes using the following command:

CUDA_VISIBLE_DEVICES=1 poetry run celery -A ts2 worker --loglevel=INFO --pool gevent --concurrency=1 --prefetch-multiplier=1

Most of my tasks are long running hence prefetch multiplier=1 (and late acknowledge in django settings). When running poetry run celery -A ts2 inspect ping -d celery@$HOSTNAME I succesfully get a response ONLY if the worker is idle. If the worker is processing a task I get Error: No nodes replied within time constraint

I have read that only the pool=solo mode is blocking, hence why I changed to gevent, but this is still the case. I rely on this command to let kubernetes know that the worker is live/ready (livenessProbe and readinessProbe).

Here is an equivalent stackoverflow question, I don't have enough rep to comment and ask the poster if they solved

Can anyone help (the pre submit for issue suggested asking here first)

Using python3 (3.8.2)
name         : celery
version      : 5.1.2
description  : Distributed Task Queue.

[warvenSniper]

any gays can response?

[Hoohaha]

I faced same issue, replaced gevent with threads, the ping works

[GitRon]

@hassant4 This worked for me: #4079 (comment)

[enricodvn]

Does it? I tried but it didn't work for me. Using gevent and that file based healthcheck still causes the k8s to restart the container when the worker is busy.

[GitRon]

I didn't use gevent... but I wrote down the solution a little more verbose: https://medium.com/ambient-innovation/health-checks-for-celery-in-kubernetes-cf3274a3e106

[ibachar-es]

Didn't work for me either when using gevent. While the worker is working (blocking), the heartbeat file is not updated

[JupiterWalker]

The inspect ping will braodcast ping command and then consume the reply from workers, if it get nothing or timeout(default is 0.2 S), it will raise "Error: No nodes replied within time constraint". So the problem looks like your worker didn't send back the reply in time. So I suggest you to add --timeout for the inspect ping or change the another concurrency solution like gevent or prefork.

[juspence]

Hi, I run my nodes using the following command:

CUDA_VISIBLE_DEVICES=1 poetry run celery -A ts2 worker --loglevel=INFO --pool gevent --concurrency=1 --prefetch-multiplier=1

Most of my tasks are long running hence prefetch multiplier=1 (and late acknowledge in django settings). When running poetry run celery -A ts2 inspect ping -d celery@$HOSTNAME I succesfully get a response ONLY if the worker is idle. If the worker is processing a task I get Error: No nodes replied within time constraint

I have read that only the pool=solo mode is blocking, hence why I changed to gevent, but this is still the case. I rely on this command to let kubernetes know that the worker is live/ready (livenessProbe and readinessProbe).

I don't think you need to switch to gevent. The "prefork" pool is the default and won't block when the worker is busy. If you do want to use gevent, set --concurrency to a very high value, and definitely not to one:

See https://www.distributedpython.com/2018/10/26/celery-execution-pool/

[crazyscientist]

Without specifying --pool (i.e. using the "prefork" default) I run into the same issue: I have tasks that can run for longer than the timeout (both "inspect ping" and k8s "readiness") and my pods keep getting killed when under load.