New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
UnicodeDecodeError when loading pickle from ZODB that was migrated from Python 2 to 3 #285
Comments
That's just a display difference; the byte value is the same. In the opcode Speaking of "kinda": Python 2 had no way to produce a pickle of an This specific pickle is of an It was only starting with ZODB 5.4.0 that ZODB 5.3 switched to using pickle protocol 2, which starts with a distinct header byte. This pickle doesn't have that, so we know that it was written with ZODB 5.2 or earlier. That means this object hasn't been modified since early 2017. Which doesn't actually tell us much 😄 Other than that, I guess the question is: should zodbupdate have been expected to locate this particular |
I ran into this same problem and was also unable to run fsrefs to get all problem oids: |
Fixed by #295 but needs a release now. |
I have 18 seemingly broken
OISet
pickles in by ZODB after migrating it from 2.7 to 3.7.In py2 all seems to be ok and I can load them from the DB. The same pickle throws a
UnicodeDecodeError
when I try to load it in Python 3.I don't know what the actual object is or where in the tree it is used (it's from a medium-sized Plone site). The transaction-description points to
plone.app.iterate
(working-copies for content) and to content that no longer exists. So I think I could simply remove them.I migrated the ZODB with zodbupdate:
./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding=utf8
. I've done this multiple times so far for various projects and never had this issue.Below is information I gathered while debugging in py2 and py3.
Python 2:
Here the same in Python 3:
The only noteable difference I see is the SHORT_BINSTRING where
\xb7
is turned into a middle dot·
but I've no idea if that means something.I worked around the whole problem by emptying the problematic OISets manually in Python 2 and then redoing the migration to Python 3. Packing and fsrefs then worked again:
It would be better to fix the underlying issue but so far I don't know what it is.
The text was updated successfully, but these errors were encountered: