-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow concat_arrays
to be pickable
#8549
Conversation
This is an snippet reproducing the issue import pickle
class A:
def __init__(self, func):
self.func = func
def decorator():
def wrap(func):
return A(func)
return wrap
@decorator()
def test():
pass
class ThingWithBoundedFunc:
def __init__(self, func=test):
self.func = test
if __name__ == '__main__':
obj = ThingWithBoundedFunc()
pickle.dumps(obj) # Fails
pickle.dumps(test) # Fails either pickling
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current approach LGTM.
I tried to solve this using functools.wrap but the case its trickier than it seems at first.
Maybe because #7489 introduced the check if isinstance(converter, Converter)
.
chainer/chainer/dataset/convert.py
Line 128 in a2804df
if isinstance(converter, Converter): |
chainer/dataset/convert.py
Outdated
@@ -106,6 +106,11 @@ def converter(): | |||
If the batch were requested to be converted to ChainerX with such | |||
converters, :class:`RuntimeError` will be raised. | |||
|
|||
.. notes:: Converters using this decorator can't be pickled, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you fix this error?
Warning, treated as error:
/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/chainer/dataset/convert.py:docstring of chainer.dataset.converter:41:Unknown directive type "notes".
.. notes:: Converters using this decorator can't be pickled,
c1c3804
to
cf70171
Compare
Jenkins, test this please. |
cf70171
to
72fe9a9
Compare
Jenkins CI test (for commit cf70171, target branch master) failed with status FAILURE. |
Finally travis passed! Sorry about it |
72fe9a9
to
807cb5f
Compare
PTAL, Thanks!! |
Jenkins, test this please. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. LGTM.
Jenkins CI test (for commit 807cb5f, target branch master) succeeded! |
I think failures are not related. thanks |
Co-Authored-By: Toshiki Kataoka <tos.lunar@gmail.com>
TravisCI: |
Jenkins, test this please. |
Jenkins CI test (for commit c6a9a9c, target branch master) failed with status FAILURE. |
Same 403, |
@toslunar This pull-request is marked as |
Jenkins, test this please. |
Jenkins CI test (for commit c6a9a9c, target branch master) succeeded! |
This tries to fix a bug that happens when using
MultiprocessParallelUpdater
withforkserver
orspawn
as the start mode.There is an error because the multiprocessing module tries to pickle the converter function and this one has lost its complete import route from the
func_name
or__name__
attribute due being decorated withconverter
.Essentially I avoid the name classing in the module by not using the decorator.
I tried to solve this using
functools.wrap
but the case its trickier than it seems at first.