ENH use cloudpickle wrapper for initializer #147
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ogrisel
approved these changes
Aug 24, 2018
docs/API.rst
Outdated
| Task & results serialization | ||
| ---------------------------- | ||
|
|
||
| To share function definition across multiple python processes, it is necessary to rely on a serialization protocol. The standard protocol in python is :mod:`pickle` but this protocol has several limitation. For instance, it cannot serialize functions which are defined interactively or in the :code:`__main__` module. |
There was a problem hiding this comment.
Actually, cloudpickle respects the pickle protocol: you can use pickle.load to unpickle something generated by cloudpickle.dump
So instead I would write:
The standard protocol in python is :mod:pickle but its default implementation in the standard library has several limitations.
docs/API.rst
Outdated
|
|
||
| To share function definition across multiple python processes, it is necessary to rely on a serialization protocol. The standard protocol in python is :mod:`pickle` but this protocol has several limitation. For instance, it cannot serialize functions which are defined interactively or in the :code:`__main__` module. | ||
|
|
||
| To avoid this limitation, :mod:`loky` relies on |cloudpickle| when it is present. |cloudpickle| is a fork of the pickle protocol which allows the serialization of a greater number of objects and it can be installed using :code:`pip install cloudpickle`. As this library is slower than the standard protocol, by default, :mod:`loky` uses it only to serialize objects which are detected to be in the :code:`__main__` module. |
There was a problem hiding this comment.
As this library is slower than the pickle module in the standard library, ...
docs/API.rst
Outdated
| There is two way to temper with the serialization in :mod:`loky`: | ||
|
|
||
| - Using the arguments :code:`job_reducers` and :code:`result_reducers`, it is possible to register custom reducers for the serialization process. | ||
| - Setting the variable :code:`LOKY_PICKLER` to an available and valid serialization module. This module must present a valid :code:`Pickler` object. For instance, setting :code:`LOKY_PICKER=cloudpickle` will force :mod:`loky` to serialize everything with |cloudpickle| instead of just serializing the object detected to be in the :code:`__main__` module. |
There was a problem hiding this comment.
Setting the environment variable :code:LOKY_PICKLER
tomMoral
force-pushed
the
FIX_pickling_initializer
branch
2 times, most recently
from
3a9f6b3
to
a115f17
Compare
|
Merged! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Consistently serialize function from
__main__withcloudpicklewhen it is available.Fix #146