Task sumiteed but never picked by worker #6071

shubhamitc · 2020-05-04T09:15:47Z

Checklist

[ x] I have verified that the issue exists against the master branch of Celery.
This has already been asked to the discussion group first.
[ x] I have read the relevant section in the
contribution guide
on reporting bugs.
[ x] I have checked the issues list
for similar or identical bug reports.
[ x] I have checked the pull requests list
for existing proposed fixes.
[ x] I have checked the commit log
to find out if the bug was already fixed in the master branch.
[ x] I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

[ x] I have included the output of celery -A proj report in the issue.
(if you are not able to do this, then at least specify the Celery
version affected).
[ x] I have verified that the issue exists against the master branch of Celery.
[x ] I have included the contents of pip freeze in the issue.
[ x] I have included all the versions of all the external dependencies required
to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version
and/or implementation.
I have tried reproducing the issue on more than one message broker and/or
result backend.
I have tried reproducing the issue on more than one version of the message
broker and/or result backend.
I have tried reproducing the issue on more than one operating system.
I have tried reproducing the issue on more than one workers pool.
I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled.
I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:
** celery:4.3.0 (rhubarb) kombu:4.6.5 py:3.7.4 billiard:3.6.1.0 sqs:N/A**

celery report Output:

software -> celery:4.3.0 (rhubarb) kombu:4.6.5 py:3.7.4
            billiard:3.6.1.0 sqs:N/A
platform -> system:Darwin arch:64bit
            kernel version:19.3.0 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:sqs results:disabled

BROKER_URL: 'sqs://localhost//'
ENV: 'production'
DEBUG: False
TESTING: False
PROPAGATE_EXCEPTIONS: None
PRESERVE_CONTEXT_ON_EXCEPTION: None
SECRET_KEY: '********'
PERMANENT_SESSION_LIFETIME: datetime.timedelta(days=31)
USE_X_SENDFILE: False
SERVER_NAME: None
APPLICATION_ROOT: '/'
SESSION_COOKIE_NAME: 'session'
SESSION_COOKIE_DOMAIN: None
SESSION_COOKIE_PATH: None
SESSION_COOKIE_HTTPONLY: True
SESSION_COOKIE_SECURE: False
SESSION_COOKIE_SAMESITE: None
SESSION_REFRESH_EACH_REQUEST: True
MAX_CONTENT_LENGTH: None
SEND_FILE_MAX_AGE_DEFAULT: datetime.timedelta(seconds=43200)
TRAP_BAD_REQUEST_ERRORS: None
TRAP_HTTP_EXCEPTIONS: False
EXPLAIN_TEMPLATE_LOADING: False
PREFERRED_URL_SCHEME: 'http'
JSON_AS_ASCII: True
JSON_SORT_KEYS: '********'
JSONIFY_PRETTYPRINT_REGULAR: False
JSONIFY_MIMETYPE: 'application/json'
TEMPLATES_AUTO_RELOAD: None
MAX_COOKIE_SIZE: 4093
BROKER_TRANSPORT_OPTIONS: {
    'broker_pool_limit': 0,
    'polling_interval': 0.3,
    'region': 'us-west-2',
    'visibility_timeout': 1320}
BUNDLE_ERRORS: False
CELERY_BROKER_URL: 'sqs://localhost//'
DB_NAME: 'dns_mapper'
DB_PASS: '********'
DB_URL: '0.0.0.0'
DB_USER: 'root'
SQLALCHEMY_DATABASE_URI: '********'
SQLALCHEMY_ECHO: False
SQLALCHEMY_ENGINE_OPTIONS: {
 'pool_pre_ping': True, 'pool_recycle': 120, 'pool_size': 50}
SQLALCHEMY_POOL_RECYCLE: 3600
SQLALCHEMY_TRACK_MODIFICATIONS: False
SQS_VISIBILITY_TIMEOUT: 1320
SQLALCHEMY_BINDS: None
SQLALCHEMY_NATIVE_UNICODE: None
SQLALCHEMY_RECORD_QUERIES: None
SQLALCHEMY_POOL_SIZE: None
SQLALCHEMY_POOL_TIMEOUT: None
SQLALCHEMY_MAX_OVERFLOW: None
SQLALCHEMY_COMMIT_ON_TEARDOWN: False

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown
Minimal Celery Version: N/A or Unknown
Minimal Kombu Version: N/A or Unknown
Minimal Broker Version: N/A or Unknown
Minimal Result Backend Version: N/A or Unknown
Minimal OS and/or Kernel Version: N/A or Unknown
Minimal Broker Client Version: N/A or Unknown
Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

alembic==1.2.1
amqp==2.5.1
aniso8601==8.0.0
appdynamics==4.5.8.0
appdynamics-bindeps-osx-x64==10.0
appdynamics-proxysupport-linux-x64==1.8.0.212.1
arrow==0.15.2
attrs==19.3.0
billiard==3.6.1.0
binaryornot==0.4.4
blinker==1.4
boto3==1.10.1
botocore==1.13.1
celery==4.3.0
certifi==2019.9.11
cffi==1.13.2
chardet==3.0.4
Click==7.0
configparser==4.0.2
cookiecutter==1.6.0
coverage==5.0.3
croniter==0.3.30
cryptography==2.8
decorator==4.4.1
dnspython==1.16.0
docutils==0.15.2
Empty==0.4.4
eventlet==0.25.1
Flask==1.1.1
Flask-Admin==1.5.3
Flask-DebugToolbar==0.10.1
Flask-Fixtures==0.3.8
Flask-JSGlue==0.3.1
Flask-Login==0.4.1
Flask-Mail==0.9.1
flask-marshmallow==0.10.1
Flask-Migrate==2.5.2
Flask-Principal==0.4.0
Flask-RESTful==0.3.7
Flask-RQ2==18.3
Flask-Script==2.0.6
Flask-Security-Fork==2.0.1
Flask-SocketIO==4.2.1
Flask-SQLAlchemy==2.4.1
Flask-Testing==0.7.1
Flask-WTF==0.14.2
future==0.18.0
gevent==1.4.0
greenlet==0.4.15
gunicorn==20.0.4
idna==2.8
importlib-metadata==0.23
itsdangerous==1.1.0
Jinja2==2.10.3
jinja2-time==0.2.0
jmespath==0.9.4
jsonpath-ng==1.4.3
kombu==4.6.5
Mako==1.1.0
MarkupSafe==1.1.1
marshmallow==3.2.1
marshmallow-enum==1.5.1
marshmallow-sqlalchemy==0.19.0
monotonic==1.5
more-itertools==7.2.0
mysql-connector==2.2.9
nose==1.3.7
packaging==20.1
passlib==1.7.1
pluggy==0.13.1
ply==3.11
poyo==0.5.0
py==1.8.1
pycparser==2.19
pycurl==7.43.0.3
PyMySQL==0.9.3
pyparsing==2.4.6
pytest==5.3.5
python-dateutil==2.8.0
python-editor==1.0.4
python-engineio==3.9.3
python-socketio==4.3.1
pytz==2019.3
PyYAML==5.1.2
redis==3.3.11
requests==2.22.0
retrying==1.3.3
rq==1.1.0
rq-scheduler==0.9.1
s3transfer==0.2.1
six==1.12.0
SQLAlchemy==1.3.10
urllib3==1.25.6
vine==1.3.0
wcwidth==0.1.8
Werkzeug==0.16.0
whichcraft==0.6.1
WTForms==2.2.1
zipp==0.6.0

Other Dependencies

N/A

Minimally Reproducible Test Case

@app.route('/mappings/v1/hello')
def hello_world():
    task_id = add_together.delay(1,2)
    LOGGER.info("Hello World! {}".format(task_id))
    return 'Hello World!'
2020-05-01 10:53:04,178 - [140481776820552] - [DummyThread-49797] - app - INFO - Hello World! 644d207d-d806-419b-a56f-dadb707cb9ab

Got the above message from logs, however there is an intemittent issue where these messages are not getting picked up by worker. In rabbitmq, there was a publish_confirm based sync call, that helped us to get confirmation, however on SQS we are not able to get anything.

Expected Behavior

Message should be picked up on high load as well.

Actual Behavior

We are seeing this issue when we had 400m/sec. Some messages are not getting picked even after messageid was received

The text was updated successfully, but these errors were encountered:

auvipy · 2020-05-04T11:08:58Z

try latest version please

shubhamitc · 2020-05-09T15:37:44Z

We have tried updating it, will monitor is for some time.

auvipy · 2020-05-16T05:03:46Z

update? any failing test?

shubhamitc · 2020-06-08T11:49:13Z

I have encountered this issue again. I needed more logging so added that.
I am waiting for this to reproduce.

shubhamitc · 2020-06-24T12:01:54Z

This time a different behaviour, celery worker got stuck. Not able to process messages.
Celery updated version: celery==4.4.2
Please suggest what to do in this case.

shubhamitc · 2020-07-16T10:46:49Z

This time a different behaviour, celery worker got stuck. Not able to process messages.
Celery updated version: celery==4.4.2
The same issue encountered again. Need to know who to solve it.

auvipy · 2020-07-16T12:10:14Z

try https://github.com/celery/celery/releases/tag/v4.4.6 or master

shubhamitc · 2020-07-16T16:25:39Z

We will try that and let you know.

…

On Thu, Jul 16, 2020, 5:40 PM Asif Saif Uddin ***@***.***> wrote: try https://github.com/celery/celery/releases/tag/v4.4.6 or master — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6071 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXEPRIZFWRS273TNXIEH33R33U3NANCNFSM4MYSYYKQ> .

shubhamitc · 2020-09-08T09:44:12Z

We upgraded the celery to v.4.4.6 and not seeing this issue anymore. This is being monitored as of now. I will update as it progresses.

shubhamitc · 2020-09-23T12:28:13Z

@auvipy , which file to add test cases for this. I would like to contribute if I can :)

shubhamitc · 2020-09-28T13:39:59Z

We have put this in production, no issues reported. Feel free to close this.

shubhamitc added the Issue Type: Bug Report label May 4, 2020

auvipy added Status: Needs Testcase ✘ Status: Needs Verification ✘ labels Jun 8, 2020

shubhamitc closed this as completed Sep 30, 2020

auvipy added Status: Not a Bug and removed Issue Type: Bug Report Status: Needs Testcase ✘ Status: Needs Verification ✘ labels Oct 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Task sumiteed but never picked by worker #6071

Task sumiteed but never picked by worker #6071

shubhamitc commented May 4, 2020

auvipy commented May 4, 2020

shubhamitc commented May 9, 2020

auvipy commented May 16, 2020

shubhamitc commented Jun 8, 2020

shubhamitc commented Jun 24, 2020

shubhamitc commented Jul 16, 2020 •

edited

auvipy commented Jul 16, 2020

shubhamitc commented Jul 16, 2020 via email

shubhamitc commented Sep 8, 2020

shubhamitc commented Sep 23, 2020

shubhamitc commented Sep 28, 2020 •

edited

Task sumiteed but never picked by worker #6071

Task sumiteed but never picked by worker #6071

Comments

shubhamitc commented May 4, 2020

Checklist

Mandatory Debugging Information

Optional Debugging Information

Related Issues and Possible Duplicates

Related Issues

Possible Duplicates

Environment & Settings

Steps to Reproduce

Required Dependencies

Python Packages

Other Dependencies

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior

auvipy commented May 4, 2020

shubhamitc commented May 9, 2020

auvipy commented May 16, 2020

shubhamitc commented Jun 8, 2020

shubhamitc commented Jun 24, 2020

shubhamitc commented Jul 16, 2020 • edited

auvipy commented Jul 16, 2020

shubhamitc commented Jul 16, 2020 via email

shubhamitc commented Sep 8, 2020

shubhamitc commented Sep 23, 2020

shubhamitc commented Sep 28, 2020 • edited

shubhamitc commented Jul 16, 2020 •

edited

shubhamitc commented Sep 28, 2020 •

edited