Python 2.6 can't read sets pickled with Python 3.0

[hagen]

BPO	3675
Nosy	@terryjreedy, @pitrou, @devdanzin, @avassalotti
Superseder	bpo-6137: Make pickle generated by Python 3.x compatible with 2.x and vice-versa.
Files	pickle.diff: Adds a 3.0:2.6 modules mapping and uses that in find_class

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-06-04.01:21:36.799>
created_at = <Date 2008-08-25.12:15:41.707>
labels = ['type-bug', 'library']
title = "Python 2.6 can't read sets pickled with Python 3.0"
updated_at = <Date 2009-06-04.01:21:36.798>
user = 'https://bugs.python.org/hagen'

bugs.python.org fields:

activity = <Date 2009-06-04.01:21:36.798>
actor = 'alexandre.vassalotti'
assignee = 'none'
closed = True
closed_date = <Date 2009-06-04.01:21:36.799>
closer = 'alexandre.vassalotti'
components = ['Library (Lib)']
creation = <Date 2008-08-25.12:15:41.707>
creator = 'hagen'
dependencies = []
files = ['11293']
hgrepos = []
issue_num = 3675
keywords = ['patch']
message_count = 10.0
messages = ['71916', '72026', '72096', '72097', '72100', '72164', '72165', '74328', '85299', '88862']
nosy_count = 6.0
nosy_names = ['collinwinter', 'terry.reedy', 'pitrou', 'ajaksu2', 'alexandre.vassalotti', 'hagen']
pr_nums = []
priority = 'critical'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = '6137'
type = 'behavior'
url = 'https://bugs.python.org/issue3675'
versions = ['Python 2.6', 'Python 3.0', 'Python 3.1', 'Python 2.7']

[hagen]

After pickling a set of ints with Python 3.0 and pickle protocol 2:

[hagenf@chage ~]$ python3.0
Python 3.0b3 (r30b3:65927, Aug 21 2008, 11:48:29)
[GCC 4.1.0 20060304 (Red Hat 4.1.0-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> f = open("test", "wb")
>>> pickle.dump({1,2,3}, f, 2)
>>> f.close()

I get the following error when trying to read this with Python 2.6:

[hagenf@chage ~]$ python
Python 2.6b3 (r26b3:65922, Aug 21 2008, 11:42:25)
[GCC 4.1.0 20060304 (Red Hat 4.1.0-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> f = open("test", "rb")
>>> pickle.load(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 1370, in load
    return Unpickler(file).load()
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 1090, in
load_global
    klass = self.find_class(module, name)
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 1124, in
find_class
    __import__(module)
ImportError: No module named builtins

[hagen]

Well, this is obviously caused by renaming "__builtin__" to "builtins"
and the fact that set (as well as frozenset) doesn't have its own opcode
and therefore gets looked up in "builtins". The problem therefore
extends to all builtin objects without opcode special casing (e.g.
object, slice, property, ...) I'm afraid that means we have to pickle
"builtins" as "__builtin__" for backwards compatibility in protocols <= 2.

But aside from that, wouldn't it be more consistent to have opcodes for
set/frozenset in protocol 3?

[devdanzin]

Hagen,
does this simple patch (against 2.6) solve it for you?

Index: Lib/pickle.py
===================================================================
--- Lib/pickle.py (revision 66050)
+++ Lib/pickle.py (working copy)
@@ -1121,6 +1121,8 @@

     def find_class(self, module, name):
         # Subclasses may override this
+        if module == "builtins":
+            module = "__builtin__"
         __import__(module)
         mod = sys.modules[module]
         klass = getattr(mod, name)

I think a dict mapping the moved modules would work better, perhaps
having it in PyPI would be enough?

[hagen]

Well, Python <= 2.5 still wouldn't be able to unpickle those built in
objects.

[devdanzin]

FWIW, there's a mapping of 2.6:3.0 modules in lib2to3:

from lib2to3.fixes.fix_imports import MAPPING

The attached patch uses that for a quick and dirty way of loading 3.0
pickles in 2.6.

[terryjreedy]

Was it really intended that 3.0 pickles unpickle on 2.6?
What about other changes like moving something from one module to
another (reduce from built-in to functools), changing all classes to new
style (several examples), or adding methods to a built-in class (floats)?

[pitrou]

Was it really intended that 3.0 pickles unpickle on 2.6?

He used protocol 2, so he explicitly asked for something inpickleable
with 2.6. If it's not the intended behaviour, then protocols < 3 should
be deprecated.

[avassalotti]

I agree with Antoine, protocols <= 2 should remain compatible with
Python 2.x or be deprecated. Keeping compatibility will require a hack,
in addition to the proposed patch, in Pickler.save_global to map Python
3's module names to the ones of Python 2.

[avassalotti]

Would it would be sufficient to add a function in the pickletools
capable of converting old Python 2 pickles to the new format in Python
3? Alternatively, we could add a compatibility Unpickler class that
would override the find_class method to do the name translation on-the-fly.

[avassalotti]

Superseded by issue bpo-6137.