Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incompatibility between pickled node attributes py2<>py3 #560

Closed
tomkooij opened this issue Jul 1, 2016 · 1 comment · Fixed by #561
Closed

Incompatibility between pickled node attributes py2<>py3 #560

tomkooij opened this issue Jul 1, 2016 · 1 comment · Fixed by #561

Comments

@tomkooij
Copy link
Contributor

tomkooij commented Jul 1, 2016

Issue #368 contains a lot of useful background info. Most of the problems reported there were fixed (upstream), but a related issue still remains:

When porting code from py2 to 3 that stores classes in node attributes, I ran into unpickling errors, caused by incompatibility between python 2 and 3 pickles. (Although they are suppossed to be compatible).

Most of these problems were fixed in numpy-1.9 (see also #368) but some classes still fail to be unpickled correctly on python 3. It is easiest demonstrated with datetime.datetime objects (see code below).

The problem is non-ascii characters in the pickled string. Pickle throws an UnicodeDecodeError on these malformed pickled strings in python 3, so it should be possible to fix. I have a fix that I need to test.

"""
write datetime object to node attribute on py2
read datetime from node attribute on py3
"""
import sys
import tables
from datetime import datetime

FILENAME = 'py2pickled.h5'

if sys.version_info[0] == 2:
    with tables.open_file(FILENAME, 'w') as data:
        data.set_node_attr('/', 'datetime', datetime.now())

if sys.version_info[0] > 2:
    with tables.open_file(FILENAME, 'r') as data:
        print('datetime = ', data.get_node_attr('/', 'datetime'))
tomkooij added a commit to tomkooij/PyTables that referenced this issue Jul 1, 2016
Issue_560.h5 contains a datetime object that was pickled on py2.
The pickled string contains non-ascii characters.
tomkooij added a commit to tomkooij/PyTables that referenced this issue Jul 1, 2016
@tomkooij
Copy link
Contributor Author

tomkooij commented Jul 1, 2016

Fixed in #561

@tomkooij tomkooij closed this as completed Jul 1, 2016
tomkooij added a commit to tomkooij/PyTables that referenced this issue Jul 13, 2016
py2_pickled_dict is a dictionary that contains a numpy array that has
illegal (non-ascii) characters when pickled on python 2.
It will throw a UnicodeDecodeError when unpickled on python 3, but
needs encoding='latin1' instead of encoding='bytes'. To check that
the dict is unpickled correctly we check the attribute 's' which is
u'just a string'.
tomkooij added a commit to tomkooij/PyTables that referenced this issue Jul 13, 2016
py2_pickled_dict is a dictionary that contains a numpy array that has
illegal (non-ascii) characters when pickled on python 2.
It will throw a UnicodeDecodeError when unpickled on python 3, but
needs encoding='latin1' instead of encoding='bytes'. To check that
the dict is unpickled correctly we check the attribute 's' which is
u'just a string'.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant