cPickle - stored data differ for same dictionary

[PhilippMlders]

BPO	12596
Nosy	@pitrou, @avassalotti, @bitdancer, @serhiy-storchaka
Files	cPickletest.py: Sample script to show the bug cPickletest2.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-05-03.01:17:15.311>
created_at = <Date 2011-07-20.16:20:51.818>
labels = ['extension-modules', 'type-bug']
title = 'cPickle - stored data differ for same dictionary'
updated_at = <Date 2013-05-03.01:17:15.309>
user = 'https://bugs.python.org/PhilippMlders'

bugs.python.org fields:

activity = <Date 2013-05-03.01:17:15.309>
actor = 'alexandre.vassalotti'
assignee = 'none'
closed = True
closed_date = <Date 2013-05-03.01:17:15.311>
closer = 'alexandre.vassalotti'
components = ['Extension Modules']
creation = <Date 2011-07-20.16:20:51.818>
creator = 'Philipp.M\xc3\xb6lders'
dependencies = []
files = ['22706', '29108']
hgrepos = []
issue_num = 12596
keywords = []
message_count = 11.0
messages = ['140750', '140751', '140752', '181586', '181594', '181595', '181597', '181598', '181603', '182289', '188287']
nosy_count = 6.0
nosy_names = ['pitrou', 'alexandre.vassalotti', 'r.david.murray', 'Philipp.M\xc3\xb6lders', 'Ramchandra Apte', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'works for me'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue12596'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

[PhilippMlders]

I think there is a problem within cPickle. I wanted to store a dictionary with only one entry with cPickle.dump() this works fine and can be loaded with cPickle.load(). But if you store the loaded data with cPickle.dump() again, the stored data differ from the first stored data. But the load works fine only the written data on disk differ. I've written a sample script, that shows the problem within code.
This problem occurs only in the 2.7 version of Python and only with dictionaries with one entry.

[bitdancer]

If the load produces the same result, why does it matter that what is on disk differs?

[PhilippMlders]

The file on disk matters for a replication service, so if a file is touched but not changed it will not be replicated, but in this special case the data change even when the structures have not changed. So if this happens very often it could cause a lot of replication which is not needed because nothing changed.

[pitrou]

As soon as hash randomization is turned on (and it's the default starting with Python 3.3), the pickled representation of dicts will also vary from run to run:

$ python -R -c "import pickle; print pickle.dumps({'a':1, 'b':2})" |md5sum
c0ae6b7f62b9c0839be883dd1efee84e  -
$ python -R -c "import pickle; print pickle.dumps({'a':1, 'b':2})" |md5sum
b03bf608516f3e0244a96d740139b050  -

[serhiy-storchaka]

It is surprising that the pickled representation of 1-element dict varies from run to run.

[RamchandraApte]

Try ./python -R -c "import pickle; print(pickle.dumps({'a':1, 'v':1}))" |md5sum. The output will differ on subsequent run, while trying ./python -R -c "import pickle; print(pickle.dumps({'a':1}))" |md5sum, the output is always the same. I suspect because the order of dicts are different on every run (try repr).

[RamchandraApte]

Darn, last sentence has some mistakes.
I suspect this issue is happening because the order of a dictionary is different on every run (try repr).

[RamchandraApte]

Further proof:
here are the results of two invocations of ./python -R -c "import pickle; print(pickle.dumps({'a':1, 'v':1}))"

b'\x80\x03}q\x00(X\x01\x00\x00\x00vq\x01K\x01X\x01\x00\x00\x00aq\x02K\x01u.'
b'\x80\x03}q\x00(X\x01\x00\x00\x00aq\x01K\x01X\x01\x00\x00\x00vq\x02K\x01u.'
Notice that in the second pickled data, the pickled data for 'v' has exchanged places with the one for 'a'! ('v' has become 'a' and at the second-last character 'a' has become 'v')

[serhiy-storchaka]

It is most probable that the difference is caused by the string interning.

[serhiy-storchaka]

Here is a minimal reproducer. Results:

pickle.dumps('spam', 2)
    0: \x80 PROTO      2
    2: U    SHORT_BINSTRING 'spam'
    8: q    BINPUT     0
   10: .    STOP
highest protocol among opcodes = 2

pickle.dumps('spam1'[:-1], 2)
    0: \x80 PROTO      2
    2: U    SHORT_BINSTRING 'spam'
    8: q    BINPUT     0
   10: .    STOP
highest protocol among opcodes = 2

cPickle.dumps('spam', 2)
    0: \x80 PROTO      2
    2: U    SHORT_BINSTRING 'spam'
    8: q    BINPUT     1
   10: .    STOP
highest protocol among opcodes = 2

cPickle.dumps('spam1'[:-1], 2)
    0: \x80 PROTO      2
    2: U    SHORT_BINSTRING 'spam'
    8: .    STOP
highest protocol among opcodes = 2

The difference between 3rd and 4th examples is "BINPUT 1". In the last case the string has refcount=1 and BINPUT doesn't emitted due to optimization. Note that Python implementation emits BINPUT with different number.

[avassalotti]

There is no guarantee the binary representation of pickled data will be same between different runs. We try to make it mostly consistent when we can, but there are cases, like this one, where we cannot ensure consistency without hurting performance significantly.