Loading the same session after dump creates a different file

[sb2nov]

I have some trouble understanding why the file c.pkl would be different from b.pkl as we're loading the session we just dumped. Thank you for the help.

import dill
import os
import tempfile

tmpdir = tempfile.mkdtemp()

class A(object):
  def __init__(self, a=5):
    self.a = a

  def printa(self):
    print self.a

print 'Directory Path', tmpdir
dill.dump_session(os.path.join(tmpdir, 'b.pkl'))
dill.load_session(os.path.join(tmpdir, 'b.pkl'))
dill.dump_session(os.path.join(tmpdir, 'c.pkl'))
dill.load_session(os.path.join(tmpdir, 'c.pkl'))
dill.dump_session(os.path.join(tmpdir, 'd.pkl'))
dill.load_session(os.path.join(tmpdir, 'd.pkl'))
dill.dump_session(os.path.join(tmpdir, 'e.pkl'))
dill.load_session(os.path.join(tmpdir, 'e.pkl'))
dill.dump_session(os.path.join(tmpdir, 'f.pkl'))
dill.load_session(os.path.join(tmpdir, 'f.pkl'))

from subprocess import check_output
print 'Hash of all the files: '
print check_output("sha1sum " + os.path.join(tmpdir, '*'), shell=True)

Sample output

Hash of all the files: 
d03af7a9fda8f141cb75ef787c2bc274609edd43  /tmp/sourabhbajaj/tmp7rsR3h/b.pkl
c57c16d3ddaaaf2954aa550f699607baa74e6edd  /tmp/sourabhbajaj/tmp7rsR3h/c.pkl
c57c16d3ddaaaf2954aa550f699607baa74e6edd  /tmp/sourabhbajaj/tmp7rsR3h/d.pkl
c57c16d3ddaaaf2954aa550f699607baa74e6edd  /tmp/sourabhbajaj/tmp7rsR3h/e.pkl
c57c16d3ddaaaf2954aa550f699607baa74e6edd  /tmp/sourabhbajaj/tmp7rsR3h/f.pkl

cc: @mmckerns

[matsjoyce]

You can use pickletools.dis(open("c.pkl")) to get a closer look. There's so much stuff that happens in dump/load_session that it's not surprising something changed. It seems in this case that in the first dump, pickle reuses some of the strings it defines, while in the following cases it redefines them. If you print the id of A.__name__, it changes after each load so I guess the first time "A" is shared with something else, say the module __dict__, but after dump+load, it is no longer shared and so pickle serializes it twice. Pickle has tons of oddities like this (dict randomisation also has effects like this, but that is between runs) so I wouldn't rely on hashes to check for differences. If you want a more stable storage, use a hacked version of json which sorts dicts before dumping (what I did in the past). Bonus is that it is readable.

[sb2nov]

@matsjoyce thanks for the explanation. I tried using pickletools but wasn't able to figure much out from it. I do see the id of A change across each load which I think is consistent with your hypothesis.

In terms of the ordered json serialization. Did you mean using that over pickle ?

[matsjoyce]

If you really need consistency, then yes, use json (or msgpack, which is faster and more compact). The only problem you would then have to deal with is dict order randomisation, which can be solved reasonably easily. Of course this won't work if you need to pickle any python object, as json and friends are limited in the number of types they can handle.

[mmckerns]

@sb2nov: I don't see the need to do anything here. Is there any reason to not close this issue?

[sb2nov]

@mmckerns closing it now