'NoneType' object has no attribute 'replace' while killing rabbit group task

[ericman93]

Checklist

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have included the contents of pip freeze in the issue.
• I have verified that the issue exists against the master branch of Celery.
• I have tried reproducing the issue on more than one message broker and/or result backend.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with retries, ETA/Countdown & rate limits disabled.

Environment & Settings

Celery version: 4.2.1 (tests also in master version)
Report:

Traceback (most recent call last): File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/app/builtins.py", line 81, in unlock_chord ret = j(timeout=3.0, propagate=True) File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/result.py", line 818, in join_native raise value celery.exceptions.TaskRevokedError: terminated 2019-02-17 19:38:30,378 - 54148 - celery.app.trace - ERROR - Task celery.chord_unlock[05c824b4-f200-47ba-9c2a-31522198dbfe] raised unexpected: AttributeError("'NoneType' object has no attribute 'replace'",) Traceback (most recent call last): File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/app/builtins.py", line 81, in unlock_chord ret = j(timeout=3.0, propagate=True) File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/result.py", line 818, in join_native raise value celery.exceptions.TaskRevokedError: terminated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/app/trace.py", line 382, in trace_task
R = retval = fun(_args, *_kwargs)
File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/app/trace.py", line 641, in protected_call
return self.run(_args, *_kwargs)
File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/app/builtins.py", line 89, in unlock_chord
app.backend.chord_error_from_stack(callback, ChordError(reason))
File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/backends/base.py", line 232, in chord_error_from_stack
return backend.fail_from_current_stack(callback.id, exc=exc)
File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/backends/base.py", line 239, in fail_from_current_stack
self.mark_as_failure(task_id, exc, exception_info.traceback)
File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/backends/base.py", line 160, in mark_as_failure
traceback=traceback, request=request)
File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/backends/amqp.py", line 113, in store_result
routing_key, correlation_id = self.destination_for(task_id, request)
File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/backends/amqp.py", line 108, in destination_for
return self.rkey(task_id), task_id
File "/Users/eric/.pyenv/versions/3.6.1/envs/romee/lib/python3.6/site-packages/celery/backends/amqp.py", line 103, in rkey
return task_id.replace('-', '')
AttributeError: 'NoneType' object has no attribute 'replace'

Required Dependencies

• Minimal Python Version: 3.6
• Minimal Broker Version: 2.4.1
• Minimal Result Backend Version: 3.7.11
• Minimal OS and/or Kernel Version: : debian (running python:3.6-slim docker)

Python Packages

alembic==0.9.5
amqp==2.4.1
aniso8601==4.1.0
asn1crypto==0.24.0
attrs==18.1.0
aws-xray-sdk==0.95
bcrypt==3.1.3
beautifulsoup4==4.4.1
billiard==3.5.0.5
blinker==1.4
boto==2.48.0
boto3==1.7.5
botocore==1.10.84
cachetools==3.1.0
celery==4.2.1
certifi==2018.11.29
cffi==1.11.0
chardet==3.0.4
Click==7.0
click-plugins==1.0.4
cligj==0.5.0
configparser==3.5.0
cookies==2.2.1
coverage==4.4.1
cryptography==2.4.2
cx-Oracle==6.1
cycler==0.10.0
cymem==2.0.2
Cython==0.26.1
cytoolz==0.9.0.1
decorator==4.1.2
descartes==1.1.0
dill==0.2.9
distro==1.0.4
docker==3.5.1
docker-pycreds==0.4.0
docutils==0.14
dominate==2.3.5
ecdsa==0.13
eli5==0.8
et-xmlfile==1.0.1
eventlet==0.22.1
Faker==0.9.0
fastparquet==0.2.1
Fiona==1.8.4
Flask==0.12.2
Flask-Bcrypt==0.7.1
Flask-Bootstrap==3.3.7.1
Flask-Cors==3.0.3
Flask-DebugToolbar==0.10.1
Flask-Login==0.4.1
Flask-Redis==0.3.0
Flask-RESTful==0.3.6
Flask-Script==2.0.6
Flask-SocketIO==3.1.2
Flask-SQLAlchemy==2.3.2
Flask-Testing==0.6.2
funcsigs==1.0.2
future==0.17.1
futures==3.1.1
geocoder==1.38.1
geopandas==0.3.0
google-api-core==1.7.0
google-auth==1.6.2
google-auth-oauthlib==0.2.0
google-cloud-bigquery==1.9.0
google-cloud-core==0.29.1
google-resumable-media==0.3.2
googleapis-common-protos==1.5.8
goslate==1.5.1
graphviz==0.8
greenlet==0.4.15
gunicorn==19.9.0
html5lib==0.9999999
httplib2==0.9.1
idna==2.8
itsdangerous==1.1.0
jdcal==1.4
Jinja2==2.9.6
jmespath==0.9.3
joblib==0.11
jsondiff==1.1.1
jsonpickle==1.1
jsonschema==2.6.0
kiwisolver==1.0.1
kombu==4.3.0
llvmlite==0.27.1
logzio-python-handler==2.0.10
lxml==4.1.1
Mako==1.0.7
Markdown==2.6.9
MarkupSafe==1.1.0
matplotlib==2.0.2
minio==3.0.4
mock==2.0.0
more-itertools==6.0.0
moto==1.3.5
msgpack==0.5.6
msgpack-numpy==0.4.3.2
munch==2.3.2
murmurhash==1.0.1
numba==0.42.1
numexpr==2.6.9
numpy==1.15.4
oauthlib==3.0.0
openpyxl==2.5.3
pandas==0.23.4
pandas-gbq==0.6.1
pandasql==0.7.3
passlib==1.7.1
pbr==5.1.2
plac==0.9.6
pluggy==0.6.0
preshed==2.0.1
protobuf==3.6.1
psycopg2==2.7.3.2
py==1.7.0
pyaml==18.11.0
pyasn1==0.4.5
pyasn1-modules==0.2.4
pycparser==2.19
pycryptodome==3.7.3
PyDictionary==1.5.2
PyHive==0.6.1
PyMySQL==0.9.2
pyodbc==4.0.22
pyparsing==2.3.1
pyproj==1.9.6
pytest==3.5.0
pytest-runner==4.2
python-dateutil==2.6.1
python-debian==0.1.31
python-editor==1.0.4
python-engineio==3.3.2
python-jose==2.0.2
python-socketio==3.1.2
pytz==2017.2
PyYAML==3.13
pyzmq==16.0.2
ratelim==0.1.6
redis==2.10.6
regex==2018.1.10
requests==2.18.4
requests-futures==0.9.9
requests-oauthlib==1.2.0
responses==0.10.5
rsa==4.0
Rtree==0.8.3
s3transfer==0.1.13
scikit-learn==0.20.1
scipy==0.19.1
Shapely==1.6.4.post2
singledispatch==3.4.0.3
six==1.11.0
sklearn==0.0
sortedcontainers==2.0.3
spacy==2.0.16
SQLAlchemy==1.2.12
sqlalchemy-redshift==0.7.3
tables==3.4.2
tabulate==0.8.3
teradatasql==16.20.0.40
teradatasqlalchemy==16.20.0.4
text-unidecode==1.2
thinc==6.12.1
thrift==0.11.0
toolz==0.9.0
tornado==4.5.3
tqdm==4.31.1
typing==3.6.6
ujson==1.35
urllib3==1.22
vine==1.2.0
visitor==0.1.3
websocket-client==0.54.0
Werkzeug==0.12.2
wrapt==1.10.11
xgboost==0.81
xlrd==1.1.0
xlwt==1.3.0
xmljson==0.1.9
xmltodict==0.12.0
zope.interface==4.1.3

Other Dependencies

N/A

Minimally Reproducible Test Case

When revoking a gorup task (with terminate=True and signal='SIGKILL') task is really terminated, but and exception is raised from amqp beckend (don't repreduce in redis)

celery_client.control.revoke([task.id for task in tasks], terminate=True, signal='SIGKILL')

Expected Behavior

No to throw exception

[auvipy]

could you try celery 4.3rc1?

[jrsmith3]

Do you understand the cause of this bug? I think I might be seeing it too.

[ericman93]

I can still reproduce it in 4.3rc1

[auvipy]

can you provide a testcase which we can run to reproduce this?

[ericman93]

it repreodce every time calling

celery_client.control.revoke(tasl_ids, terminate=True, signal='SIGKILL')

and the using amqp as broker

[danielkrich]

Hi, is there a solution? Happens to me as well.

[ofercaspi]

I'm also experiencing this exact issue... +1

[auvipy]

can you try celery==4.4.0rc4? and report again?

[omerfeinberg]

Hey there,
I checked 4.4.0rc4, sadly the issue remains.

[lazreg87]

Got this same issue on 4.4.0, slightly different message though: ‘NoneType’ object has no attribute ‘groups’“,.
We rolled back to 4.4.0rc4 and that worked

[mmmcorpsvit]

Got this same issue on 4.4.0, slightly different message though: ‘NoneType’ object has no attribute ‘groups’“,.
We rolled back to 4.4.0rc4 and that worked

confirm, have same problem, too dropdown version, th

[auvipy]

what is the update of this with the v5 version?