__setstate__ is called for false values

[eltoder]

BPO	12290
Nosy	@birkenfeld, @abalkin, @pitrou, @avassalotti, @bitdancer, @eltoder, @serhiy-storchaka, @iritkatriel
Files	setstate.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2011-06-09.02:20:57.341>
labels = ['3.11', 'type-bug', '3.9', '3.10', 'docs']
title = '__setstate__ is called for false values'
updated_at = <Date 2021-12-05.23:10:56.438>
user = 'https://github.com/eltoder'

bugs.python.org fields:

activity = <Date 2021-12-05.23:10:56.438>
actor = 'iritkatriel'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2011-06-09.02:20:57.341>
creator = 'eltoder'
dependencies = []
files = ['22375']
hgrepos = []
issue_num = 12290
keywords = ['patch']
message_count = 6.0
messages = ['137935', '137936', '138410', '138573', '265087', '407755']
nosy_count = 9.0
nosy_names = ['georg.brandl', 'belopolsky', 'pitrou', 'alexandre.vassalotti', 'r.david.murray', 'docs@python', 'eltoder', 'serhiy.storchaka', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue12290'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-56499: Update the pickle library's note section for the __setstate__ function #101062
• [3.12] gh-56499: Update the pickle library's note section for the __setstate__ function (GH-101062) #115597
• [3.11] gh-56499: Update the pickle library's note section for the __setstate__ function (GH-101062) #115598

[eltoder]

Pickle documentation [1] says:

""" Note: If __getstate__() returns a false value, the __setstate__() method will not be called upon unpickling. """

However, this isn't quite true. This depends on the version of pickle protocol. A small example:

>>> class Pockle(object):
	def __getstate__(self):
		return 0
	def __setstate__(self, state):
		sys.stdout.write('__setstate__ is called!\n')

>>> for p in range(4):
	sys.stdout.write('protocol %d: ' % p)
	pickle.loads(pickle.dumps(Pockle(), p))

protocol 0: <main.Pockle object at 0x0000000002EAE3C8>
protocol 1: <main.Pockle object at 0x0000000002EAE358>
protocol 2: __setstate__ is called!
<main.Pockle object at 0x0000000002EAE3C8>
protocol 3: __setstate__ is called!
<main.Pockle object at 0x0000000002EAE358>

So for protocols >= 2 setstate is called. This is caused by object.__reduce_ex__ returning different tuples for different protocol versions:

>>> for p in range(4):
	sys.stdout.write('protocol %d: %s\n' % (p, Pockle().__reduce_ex__(p)))

protocol 0: (<function _reconstructor at 0x0000000001F03048>, (<class '__main__.Pockle'>, <class 'object'>, None))
protocol 1: (<function _reconstructor at 0x0000000001F03048>, (<class '__main__.Pockle'>, <class 'object'>, None))
protocol 2: (<function __newobj__ at 0x0000000001F03148>, (<class '__main__.Pockle'>,), 0, None, None)
protocol 3: (<function __newobj__ at 0x0000000001F03148>, (<class '__main__.Pockle'>,), 0, None, None)

Implementation of reduce_ex for protos 0-1 in copy_reg.py contains the documented check: http://hg.python.org/cpython/file/f1509fc75435/Lib/copy_reg.py#l85

Implementation for proto 2+ in typeobject.c is happy with any value: http://hg.python.org/cpython/file/f1509fc75435/Objects/typeobject.c#l3205

Pickle itself only ignores None, not any false value: http://hg.python.org/cpython/file/f1509fc75435/Lib/pickle.py#l418

I think this is a documentation issue at this point.

[1] http://docs.python.org/py3k/library/pickle.html#pickle.object.\_\_setstate__

[bitdancer]

See also bpo-6827, just for some background on the current docs.

[eltoder]

So how about this correction?

[birkenfeld]

Well, this looks correct then.

[serhiy-storchaka]

Actually this wording is not quite correct. __setstate__() is called for any pickled state. It is not called only if the state is not pickled. The state is not pickled if reducing method (reduce_ex or __reduce__) doesn't return state or returns None as a state. Default reducing method for protocol 0 and 1 doesn't return a state if __getstate__() returns false value.

There are many other details of pickle protocol that are not correctly documented in the documentation of the pickle module. PEP-307 documents pickle protocol more correctly.

[iritkatriel]

See also bpo-26695.

[furkanonder]

@serhiy-storchaka I opened a PR to update the documentation based on PEP-307.