AttributeError: int in uuid.py when being called by celery/concurrency/asynpool.py #7989
Comments
|
Hey @fsteggink 👋, We also offer priority support for our sponsors. |
|
Note that if I 'patch' uuid.py in the following two places, then my task is being processed correctly. def __getstate__(self):
d = {'int': getattr(self, 'int', 0)}
if hasattr(self, 'is_safe') and self.is_safe != SafeUUID.unknown:
# is_safe is a SafeUUID instance. Return just its value, so that
# it can be un-pickled in older Python versions without SafeUUID.
d['is_safe'] = self.is_safe.value
return d def __str__(self):
hex = '%032x' % getattr(self, 'int', 0)
return '%s-%s-%s-%s-%s' % (
hex[:8], hex[8:12], hex[12:16], hex[16:20], hex[20:])The lines with Of course I'm not comfortable with this change, because I've no idea what happens under water. However, as a temporary workaround I can live with it. |
|
Unfortunately this workaround did break something. There was a task to which I passed an UUID. When converting the UUID to a string, this task worked. I had another using an UUID, and I'm converting this to a string as well. This enabled me to remove the workaround to uuid.py. So, it appears that an UUID cannot be pickled when put on the task queue, or retrieved from the task queue. Example: from celery import shared_task
@shared_task
def set_delivery_status(delivery_uuid, status):
Delivery.objects.filter(uuid=delivery_uuid).update(status=status)Putting the task on the queue: from celery import group
def django_post_function():
delivery_uuid = str(delivery.uuid) # This fixed it
res = chain(
set_delivery_status.si(delivery_uuid, 'in progress'),
init_delivery.si(delivery_uuid, subdirs),
group(*tasks),
finalize_delivery.si(delivery_uuid),
set_delivery_status.si(delivery_uuid, 'finished'),
).on_error(on_delivery_error.si(delivery_uuid))Django model: class Delivery(models.Model):
STATUS_TYPES = [
('created', 'Aangemaakt'),
('queued', 'In wachtrij'),
('in progress', 'In verwerking'),
('finished', 'Gereed'),
('cleared', 'Opgeruimd'),
('error', 'Fout'),
]
uuid = models.UUIDField(default=uuid.uuid4, editable=False)
status = models.CharField(max_length=16, choices=STATUS_TYPES, default='created')(Edit: added relevant part of Django model) |
|
You are using a customized JSON serializer so the problem is likely there. Could you please try to reproduce the issue with the latest code for Celery and Kombu as serializing and de-serializing UUID objects is now supported using the JSON serializer? |
Checklist
mainbranch of Celery.contribution guide
on reporting bugs.
for similar or identical bug reports.
for existing proposed fixes.
to find out if the bug was already fixed in the main branch.
in this issue (If there are none, check this box anyway).
Mandatory Debugging Information
celery -A proj reportin the issue.(if you are not able to do this, then at least specify the Celery
version affected).
mainbranch of Celery.pip freezein the issue.to reproduce this bug.
Optional Debugging Information
and/or implementation.
result backend.
broker and/or result backend.
ETA/Countdown & rate limits disabled.
and/or upgrading Celery and its dependencies.
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Environment & Settings
Celery version:
celery reportOutput:Steps to Reproduce
Required Dependencies
Python Packages
pip freezeOutput:Other Dependencies
OS: Linux Debian Buster - Bullseye (in Docker on Windows) Python 3.7 - 3.9 Django: 3.2.12 Celery 5.0.5 - 5.2.7 Kombu 5.0.2 - 5.2.4 librabbitmq 2.0.0 or amqp 5.1.1
Minimally Reproducible Test Case
Expected Behavior
The task submitted to the broker is being processed happily by Celery.
Actual Behavior
The following error occurs:
This happens after a task has been placed in RabbitMQ (broker) by Django, and is about to be handled by my worker (celery_1 container). The task is using the Celery canvas.
In Python 3.7 the actual error is a bit different, but it also occurs in the
__getstate__method in uuid.py. At that time I was able to patch this error by using my own version of uuid.py. I've added the checkself.is_safe is not None. Relevant part:After this everything seemed to proceed as normal.
Python 3.9 code for reference where this error occurs (see stack trace):
The file uuid.py in my container is identical to the one here: https://github.com/python/cpython/blob/3.9/Lib/uuid.py
However, now it is time to use Python 3.9 (provided by Debian Bullseye), and I'm not able to work around this issue by fiddling with uuid.py. When I added a fix to the missing 'int' attribute (for example by setting it to 0), a similar error occurs somewhere else. I haven't tried patching uuid.py all over the place. Since this issue is occurring with multiple Celery versions and two different Python versions, I finally took the time to submit the bug report.
I have no clue what this causes, since the stack trace is very deep, but I assume this is related to serialization / deserialization of the message ID (which is a UUID). It seems to me that the
__init__method is never called when the UUID object is created. I could verify this by adding print statements to uuid.py in both the__init__method and the__getstate__method. The print statement in the__getstate__method reported a different ID (print(id(self))) than any of the ID's printed in__init__.I've tried to mitigate this issue by using Celery versions 5.0.5 - 5.2.7 (originally I was using 5.2.3). I've also switched from librabbitmq to the amqp Python lib. I've also tried to use Kombu 5.0.2 (together with Celery 5.0.5).
It is possible that this is actually caused by an issue in Kombu or Billiard.
The text was updated successfully, but these errors were encountered: