fails to load a @dataclass instance when serialized=False, fails to dump when serialized=True

[jvickroy]

klepto 0.1.6 (running in Python 3.7.3) raises a ReferenceError exception in _dill.py @ line 1148 when attempting to dump a dir_archive with serialized=True (the default).

The following sample script illustrates the behavior (see SERIALIZED).

from   dataclasses import dataclass
@dataclass (frozen=True)
class Foo:
    name : str

foo = Foo (name='klepto')
key = foo.name
d   = dict (key=foo)
import klepto
SERIALIZED = False ## WORKS!
SERIALIZED = True  ## raises ReferenceError: {} does not reference a class __dict__ @ ..Python37\lib\site-packages\dill\_dill.py", line 1148, in save_dictproxy
a = klepto.archives.dir_archive (dict=d, serialized=SERIALIZED)
a.dump()

[mmckerns]

Thanks for reporting. In terms of the case serialized=True, that's an issue for dill, not klepto -- and it should be fixed by dill. I'd close this issue, and suggest you report to dill. However, you might note that serialized=False actually doesn't work -- at least for me. See below:

>>> from   dataclasses import dataclass
>>> @dataclass (frozen=True)
... class Foo:
...     name : str
... 
>>> foo = Foo (name='klepto')
>>> key = foo.name
>>> d   = dict (key=foo)
>>> import klepto
>>> SERIALIZED = False
>>> a = klepto.archives.dir_archive (dict=d, serialized=SERIALIZED)
>>> a.dump()
>>> 
>>> a.clear()
>>> a.load()
Traceback (most recent call last):
  File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 632, in _lookup
    exec(string, globals()) #FIXME: unsafe, potential name conflict
  File "<string>", line 1, in <module>
  File "/Users/mmckerns/src/python-tmp/memo/K_key/__init__.py", line 3, in <module>
    memo = Foo(name='klepto')
TypeError: Foo() takes no arguments

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 152, in load
    self.update(self.archive.__asdict__())
  File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 406, in __asdict__
    return dict((key,self.__getitem__(key)) for key in keys)
  File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 406, in <genexpr>
    return dict((key,self.__getitem__(key)) for key in keys)
  File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 436, in __getitem__
    return self._lookup(key)
  File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 636, in _lookup
    raise KeyError(key)
KeyError: 'key'
>>> 

Since most objects serialize well, serialized=True is the default, and the dataclass object is used less frequently than some other objects... it seems this case is a klepto issue. I'll change the title to reflect the issue.

[jvickroy]

Thanks for the follow-up. You are absolutely correct! I'm seeing the same behavior. At the moment I'm using @DataClass (frozen=False, unsafe_hash=True) for my class decorators in conjunction with fast=True as a dir_archive parameter and that seems to work. frozen=False seems to be required to allow klepto load() (i.e., unpickle) to succeed. unsafe_hash=True is included because my dict values are really python set containers, and with frozen=False, hash() is unspecified by default.

…

On 5/6/2019 6:58:43 AM, "Mike McKerns" ***@***.***> wrote: Thanks for reporting. In terms of the case serialized=True, that's an issue for dill, not klepto -- and it should be fixed by dill. I'd close this issue, and suggest you report to dill. However, you might note that serialized=False actually doesn't work -- at least for me. See below: >>> from dataclasses import dataclass >>> @DataClass (frozen=True) ... class Foo: ... name : str ... >>> foo = Foo (name='klepto') >>> key = foo.name >>> d = dict (key=foo) >>> import klepto >>> SERIALIZED = False >>> a = klepto.archives.dir_archive (dict=d, serialized=SERIALIZED) >>> a.dump() >>> >>> a.clear() >>> a.load() Traceback (most recent call last): File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 632, in _lookup exec(string, globals()) #FIXME: unsafe, potential name conflict File "<string>", line 1, in <module> File "/Users/mmckerns/src/python-tmp/memo/K_key/__init__.py", line 3, in <module> memo = Foo(name='klepto') TypeError: Foo() takes no arguments During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 152, in load self.update(self.archive.__asdict__()) File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 406, in __asdict__ return dict((key,self.__getitem__(key)) for key in keys) File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 406, in <genexpr> return dict((key,self.__getitem__(key)) for key in keys) File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 436, in __getitem__ return self._lookup(key) File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 636, in _lookup raise KeyError(key) KeyError: 'key' >>> Since most objects serialize well, serialized=True is the default, and the dataset object is used less frequently than some other objects... it seems this needs investigation on why it fails. I'll change the title to reflect the issue. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#71 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACIOVTRVHKFETAIM5JDBVCTPUATQHANCNFSM4HK4R3SA>.

[mmckerns]

@jvickroy: the bug appears to be fixed in dill in master.

[jvickroy]

Thanks for alerting me to this. /jim