Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with dill & celery #2526

Closed
max-sixty opened this issue Mar 3, 2015 · 1 comment

Comments

Projects
None yet
1 participant
@max-sixty
Copy link

commented Mar 3, 2015

We'd like to be able to pass objects that aren't pickle-able, and so we're using dill.

In a few places in celery & kombu, pickle is used, regardless of what is registered in the kombu.serialization registry. (In particular, celery.concurrency.asynpool and celery.worker.state).

This stops us using dill for chains & chords. We can monkey patch, but are having some issues doing that, and I'm wondering whether this was an oversight or someone has thought about this already and settled on forcing pickle.

Cheers,
Max

@max-sixty

This comment has been minimized.

Copy link
Author

commented Mar 16, 2015

We got this working with the following. You need to monkey patch the pickle in kombu.serialization because that's used when workers are serializing objects between each other (in chains, for example), regardless of what serializers you have registered.

def add_dill():
    registry = kombu.serialization.registry
    kombu.serialization.pickle = dill

    registry.unregister('pickle')

    def pickle_loads(s, load=dill.load):
    # used to support buffer objects
        return load(BytesIO(s))

    def pickle_dumps(obj, dumper=dill.dumps):
        return dumper(obj, protocol=kombu.serialization.pickle_protocol)

    registry.register('pickle', pickle_dumps, pickle_loads,
                      content_type='application/x-python-serialize',
                      content_encoding='binary')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.