TypeError: cannot pickle 'property' object

[adonig]

I experience a problem with Python class properties and joblib. Somehow it is not possible to pickle them. See this minimal reproducible example:

import multiprocessing

from joblib import Parallel, delayed


class Task:
    def run(self):
        print(self.hello_world)

    @property
    def hello_world(self):
        return "hello world"


if __name__ == "__main__":
    Parallel(n_jobs=multiprocessing.cpu_count())([delayed(lambda: Task().run())()])

This code fails raising TypeError: cannot pickle 'property' object. It works if you replace the property with a method get_hello_world and call self.get_hello_world() in Task.run.

Environment:

• Python 3.8.1
• joblib 0.14.1
• macOS Catalina 10.15.2

[KristianHolsheimer]

I'm running into the same issue, using the ray package.

[KristianHolsheimer]

It looks like this is an issue with the loky backend:

from joblib import Parallel, delayed


class WithProperty:
    
    @property
    def foo(self):
        return 'foo'
      
    def __call__(self):
        return self.foo


class WithoutProperty:
    
    def bar(self):
        return 'bar'
      
    def __call__(self):
        return self.bar()


# different backends
loky_backend = Parallel(n_jobs=2, backend='loky')
multiprocess = Parallel(n_jobs=2, backend='multiprocessing')

# create instances
with_property = WithProperty()
without_property = WithoutProperty()

# test
multiprocess([delayed(without_property)()])  # ok
multiprocess([delayed(with_property)()])     # ok
loky_backend([delayed(without_property)()])  # ok
loky_backend([delayed(with_property)()])     # broken

[KristianHolsheimer]

FWIW I also checked whether cloudpickle (which I thought was the pickler that loky uses) correctly pickles the object.

import cloudpickle

class WithProperty:
    
    @property
    def foo(self):
        return 'foo'
      
    def __call__(self):
        return self.foo

with_property = WithProperty()
cloudpickle.dumps(with_property)  # ok

[pierreglaser]

Hi, thanks for the report.

FWIW I also checked whether cloudpickle (which I thought was the pickler that loky uses) correctly pickles the object.

FYI, joblib vendors its own cloudpickle (inside joblib.externals.cloudpickle), which gets updated once in a while. So cloudpickle and joblib.externals.cloudpickle may or may not behave the same way depending on which version of joblib and cloudpickle is installed on your environment.

That being said, the last joblib release vendors cloudpickle==1.2.2 which had a bug affecting the pickling of property objects on Python 3.8 (see cloudpipe/cloudpickle#329)

The future joblib release will contain cloudpickle == 1.4.1, and thus should not have this issue.

[KristianHolsheimer]

Thank you @pierreglaser

I ran into this issue in the ray package, which also packages its own copy of cloudpickle. The problem of the unpicklable property is resolved (as per your PR from earlier this year.. thanks for that)

I'm now having the same issue with abstractproperty. I'll see if I can put a quick PR ready for cloudpickle. If not, I'll submit an issue there.

Thanks again,
Kris