problem with pickling newstyle class instances

[tovrstra]

BPO	892902
Nosy	@birkenfeld, @devdanzin, @avassalotti, @serhiy-storchaka
Files	pickletest.py: pickletest.py pickle_recursive-2.7.patch pickle_recursive-2.7_2.patch pickle_recursive-2.7_3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/avassalotti'
closed_at = <Date 2015-11-07.09:22:53.231>
created_at = <Date 2004-02-08.15:55:57.000>
labels = ['type-bug', 'library']
title = 'problem with pickling newstyle class instances'
updated_at = <Date 2015-11-07.18:05:43.674>
user = 'https://bugs.python.org/tovrstra'

bugs.python.org fields:

activity = <Date 2015-11-07.18:05:43.674>
actor = 'python-dev'
assignee = 'alexandre.vassalotti'
closed = True
closed_date = <Date 2015-11-07.09:22:53.231>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2004-02-08.15:55:57.000>
creator = 'tovrstra'
dependencies = []
files = ['1199', '40859', '40947', '40961']
hgrepos = []
issue_num = 892902
keywords = ['patch']
message_count = 13.0
messages = ['19940', '19941', '58057', '82010', '114318', '188288', '253435', '254003', '254091', '254191', '254264', '254265', '254291']
nosy_count = 6.0
nosy_names = ['georg.brandl', 'tovrstra', 'ajaksu2', 'alexandre.vassalotti', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue892902'
versions = ['Python 2.7']

[tovrstra]

My python version:

Python 2.3.3 (#1, Jan 25 2004, 21:45:01)
[GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r3,
propolice)] on linux2

Please try the example. That will explain a lot. The
problem is that not all new style class instances are
picklable. In my example a class is derived from a list
and a base class. The list-descendant contains an
instance of the base class, which has a reference to
the list containing it. with cPickle things work fine,
but not for the normal pickle routines

class subitem:
	def __init__(self, parent):
		self.parent = parent
		if parent != None:
			parent.append(self)

class group(subitem, list):
	def __init__(self, parent):
		subitem.__init__(self, parent)
		

g = group(None)
s = subitem(g)

import cPickle
print cPickle.dumps(g)

import pickle
print pickle.dumps(g)

[birkenfeld]

Logged In: YES
user_id=1188172

Verified in 2.5cvs.

[avassalotti]

Please assign this bug to me.

Note that neither cPickle or pickle is able to load the stream generated
by cPickle correctly:

   >>> g = group(None)
   >>> subitem(g)
   >>> g[0].parent is g
   True
   >>> gp = cPickle.loads(cPickle.dumps(g))
   >>> gp[0].parent is gp
   False

I don't think that will be easy to fix, but I will try to see what I can do.

[devdanzin]

Confirmed on trunk.

[BreamoreBoy]

This is still a problem with 2.6 and 2.7. I don't know how to test this with py3k as cPickle doesn't exist, can someone advise, thanks.

[avassalotti]

I fixed this while working on PEP-3154 [http://hg.python.org/features/pep-3154-alexandre/rev/eed9142d664f]. The relevant piece is

@@ -420,7 +424,13 @@ class _Pickler:
             write(REDUCE)
 
         if obj is not None:
-            self.memoize(obj)
+            # If the object is already in the memo, this means it is
+            # recursive. In this case, throw away everything we put on the
+            # stack, and fetch the object back from the memo.
+            if id(obj) in self.memo:
+                write(POP + self.get(self.memo[id(obj)][0]))
+            else:
+                self.memoize(obj)
 
         # More new special cases (that work with older protocols as
         # well): when __reduce__ returns a tuple with 4 or 5 items,

It would be pretty easy to backport this to 2.7 and 3.3. It is also good to mention that that only protocol 0 and 1 are affected.

[serhiy-storchaka]

Here is a patch that backports recursive objects handling to 2.7.

[serhiy-storchaka]

Could you please make a review Alexandre?

[serhiy-storchaka]

Updated patch addresses Alexandre's comments.

[serhiy-storchaka]

Here is revised patch that uses different approach to tests. cPickleFastPicklerTests overridden old recursive tests to check that they raises an exception. The patch extends this to new recursive tests.

[python-dev]

New changeset 9ad1fd251ddf by Serhiy Storchaka in branch '2.7':
Issue bpo-892902: Fixed pickling recursive objects.
https://hg.python.org/cpython/rev/9ad1fd251ddf

New changeset 2071d16ed5e6 by Serhiy Storchaka in branch '3.4':
Issue bpo-892902: Added new tests for pickling recursive collections.
https://hg.python.org/cpython/rev/2071d16ed5e6

New changeset f33ce913220b by Serhiy Storchaka in branch '3.5':
Issue bpo-892902: Added new tests for pickling recursive collections.
https://hg.python.org/cpython/rev/f33ce913220b

New changeset 2c81a883d8ca by Serhiy Storchaka in branch 'default':
Issue bpo-892902: Added new tests for pickling recursive collections.
https://hg.python.org/cpython/rev/2c81a883d8ca

[serhiy-storchaka]

Thank you for your review Alexandre.

In 3.x old tests test_recursive_set and test_recursive_frozenset now are implemented in test_recursive_set_and_inst and test_recursive_frozenset_and_inst. Instead new test_recursive_set now tests protocol 4 ability of pickling recursive sets itself.

[python-dev]

New changeset 77184a429dae by Serhiy Storchaka in branch '2.7':
Issue bpo-892902: Disable newly added tests in test_xpickle.
https://hg.python.org/cpython/rev/77184a429dae