Increase pickle compatibility

[sbt]

BPO	13566
Nosy	@terryjreedy, @pitrou, @vstinner, @avassalotti, @serhiy-storchaka, @vajrasky, @MojoVampire
Files	pickle_old_strings.patch pickle-old-strings-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2011-12-09.13:25:03.645>
labels = ['3.7', 'type-feature', 'library']
title = 'Increase pickle compatibility'
updated_at = <Date 2017-03-06.16:02:42.072>
user = 'https://bugs.python.org/sbt'

bugs.python.org fields:

activity = <Date 2017-03-06.16:02:42.072>
actor = 'josh.r'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2011-12-09.13:25:03.645>
creator = 'sbt'
dependencies = []
files = ['39348', '46703']
hgrepos = []
issue_num = 13566
keywords = ['patch']
message_count = 16.0
messages = ['149092', '149101', '149104', '205445', '206504', '206532', '206536', '242949', '245048', '245049', '245050', '245051', '245056', '245057', '288158', '289119']
nosy_count = 9.0
nosy_names = ['terry.reedy', 'pitrou', 'vstinner', 'alexandre.vassalotti', 'sbt', 'Ramchandra Apte', 'serhiy.storchaka', 'vajrasky', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13566'
versions = ['Python 3.7']

[sbt]

If you pickle an array object on python 3 the typecode is encoded as a unicode string rather than as a byte string. This makes python 2 reject the pickle.

#########################################

Python 3.3.0a0 (default, Dec  8 2011, 17:56:13) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle, array
>>> pickle.dumps(array.array('i', [1,2,3]), 2)
b'\x80\x02carray\narray\nq\x00X\x01\x00\x00\x00iq\x01]q\x02(K\x01K\x02K\x03e\x86q\x03Rq\x04.'

#########################################

Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> pickle.loads(b'\x80\x02carray\narray\nq\x00X\x01\x00\x00\x00iq\x01]q\x02(K\x01K\x02K\x03e\x86q\x03Rq\x04.')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\Python27\lib\pickle.py", line 1382, in loads
    return Unpickler(file).load()
  File "c:\Python27\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "c:\Python27\lib\pickle.py", line 1133, in load_reduce
    value = func(*args)
TypeError: must be char, not unicode

[RamchandraApte]

The problem is that pickle is calling array.array(u'i',[1,2,3]) and array.array in Python 2 doesn't allow unicode strings as a typecode (typecode is the first argument)

The docs in Python 2 and Py3k doesn't specify the type of the typecode argument of array.array.
In Python 2 it seems that typecode has to be a bytes string.
In Python 3 it seems that typecode has to be a unicode string.

I suggest that array.array be changed in Python 2 to allow unicode strings as a typecode or that pickle detects array.array being called and fixes the call.

[sbt]

I suggest that array.array be changed in Python 2 to allow unicode strings
as a typecode or that pickle detects array.array being called and fixes
the call.

Interestingly, py3 does understand arrays pickled by py2. This appears to be because py2 pickles str using BINSTRING or SHORT_BINSTRING which will unpickle as str on py2 and py3. py3 pickles str using BINUNICODE which will unpickle as unicode on py2 and str on py3.

I think it would be better to fix this in py3 if possible, but that does not look easy: modifying array.__reduce_ex__ alone would not be enough.

The only thing I can think of is for py3 to grow a "_binstr" type which only supports ascii strings and is special-cased by pickle to be pickled using BINSTRING. Then array.__reduce_ex__ could be something like:

  def __reduce_ex__(self, protocol):
    if protocol <= 2:
      return array.array, (_binstr(self.typecode), list(self))
    else:
      ...

[avassalotti]

Adding a special type is not a bad idea. We have to keep the code for loading BINSTRING opcodes anyway, so we might as well use it. It could be helpful for unit-testing our Python 2 compatibility support for pickle.

We should still fix array in 2.7 to accept unicode object for the typecode though.

[vajrasky]

Alexandre Vassalotti said: "We should still fix array in 2.7 to accept unicode object for the typecode though."

I created issue bpo-20014 (with the patch) for this feature.

[serhiy-storchaka]

See bpo-20015 for more general approach.

[vstinner]

If you pickle an array object on python 3 the typecode is encoded as a unicode string rather than as a byte string. This makes python 2 reject the pickle.

Pickles files of Python 3 are supposed to be compatible with Python 2?

It looks very tricky to produce pickle files compatible with both versions.

[serhiy-storchaka]

Proposed patch pickles all ascii strings with protocols < 3 and fix_import=True with compatible opcodes (STRING, BINSTRING and SHORT_BINSTRING). Pickled strings are unpickled as str in Python 2 and Python 3 (unless encoding="bytes").

As a side effect, short ascii strings (length < 256) are pickled more compact with protocols < 3.

[serhiy-storchaka]

Alexandre, Antoine, what are your thoughts?