Anomalous AdminMiddleware serialization behavior

[sanjioh]

Hi,

first of all, thanks for your work on dramatiq and django-dramatiq.

I'd like to report a strange behaviour of the AdminMiddleware, in which it raises an exception upon task serialization.

Full reproducer at: https://github.com/sanjioh/django-dramatiq-repro

The core of the problem seems to be in the following files:

views.py

from uuid import uuid4

from django.http import response

from .tasks import a_task


def repro(request):
    a_task.send(
        {
            'value': str(uuid4()),
        },
    )
    return response.HttpResponse()

tasks.py

from uuid import UUID

import dramatiq


class Klass:

    def __init__(self, value):
        self.value = value

    @classmethod
    def fromdict(cls, attrs):
        attrs['value'] = UUID(attrs['value'])
        return cls(**attrs)


@dramatiq.actor
def a_task(attrs):
    Klass.fromdict(attrs)

Stacktrace:

[2020-04-24 10:17:13,490] [PID 15544] [Thread-4] [dramatiq.broker.RedisBroker] [CRITICAL] Unexpected failure in after_process_message.
Traceback (most recent call last):
  File "/Users/fabio/.virtualenvs/django-dramatiq-repro/lib/python3.8/site-packages/dramatiq/broker.py", line 98, in emit_after
    getattr(middleware, "after_" + signal)(self, *args, **kwargs)
  File "/Users/fabio/.virtualenvs/django-dramatiq-repro/lib/python3.8/site-packages/django_dramatiq/middleware.py", line 53, in after_process_message
    Task.tasks.create_or_update_from_message(
  File "/Users/fabio/.virtualenvs/django-dramatiq-repro/lib/python3.8/site-packages/django_dramatiq/models.py", line 19, in create_or_update_from_message
    "message_data": message.encode(),
  File "/Users/fabio/.virtualenvs/django-dramatiq-repro/lib/python3.8/site-packages/dramatiq/message.py", line 101, in encode
    return global_encoder.encode(self._asdict())
  File "/Users/fabio/.virtualenvs/django-dramatiq-repro/lib/python3.8/site-packages/dramatiq/encoder.py", line 49, in encode
    return json.dumps(data, separators=(",", ":")).encode("utf-8")
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type UUID is not JSON serializable

Please let me know if I can provide further details.

Thanks,
Fabio

[Bogdanp]

You'll have to set up a custom encoder to be able to serialize UUIDs. See:

https://dramatiq.io/reference.html#dramatiq.Encoder

You can set a custom encoder in your settings using the DRAMATIQ_ENCODER setting.

[sanjioh]

@Bogdanp Thanks for your answer.

Just to be sure: is a custom encoder needed even if what’s not json serializable are local variables in the task?

Please note that in the code I provided there’s no UUID in the task argument, it’s a plain dict with both key and value as strings, that shouldn’t have any serialization issue.

The UUID is created in the body of the task, which should be executed in the worker.

Thanks

[Bogdanp]

@sanjioh sorry! I misread your code. I'll take another look this weekend because that is odd.

[sanjioh]

No problem, thanks for your help! 👍🏻

[AndreCimander]

@sanjioh are you creating/overwriting custom message_ids in one of your middlewares? :)

I took a quick glance into the code and the only situation where this issue might crop up, as far as I can see, is when dramatiq.message.Message receives a custom message_id (dramatiq/message.py:78 might be sensible to typecast custom message_ids @Bogdanp but this also might conflict with customizations...).

[sanjioh]

Hi @AndreCimander ,

no, I'm not fiddling with message_ids at all.
The code at the repo https://github.com/sanjioh/django-dramatiq-repro should be enough as a reproducer.
Please let me know if I can be of further help!

[Bogdanp]

Thanks for providing the reproduction repo, @sanjioh . I opened a PR with a fix: sanjioh/django-dramatiq-repro#2