unpickling of a datetime object in 3.5 fails when pickled with 2.7

[vadimf]

BPO	36499
Nosy	@abalkin, @serhiy-storchaka, @MojoVampire, @pganssle, @tirkarthi
Superseder	bpo-22005: datetime.setstate fails decoding python2 pickle
Files	pickle_unpickle.tar.gz: tgz with the source files needed to reproduce the problem

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-04-02.10:48:52.425>
created_at = <Date 2019-04-01.23:02:42.992>
labels = ['library', 'type-crash']
title = 'unpickling of a datetime object in 3.5 fails when pickled with 2.7'
updated_at = <Date 2019-04-02.10:48:52.423>
user = 'https://bugs.python.org/vadimf'

bugs.python.org fields:

activity = <Date 2019-04-02.10:48:52.423>
actor = 'josh.r'
assignee = 'none'
closed = True
closed_date = <Date 2019-04-02.10:48:52.425>
closer = 'josh.r'
components = ['Library (Lib)']
creation = <Date 2019-04-01.23:02:42.992>
creator = 'vadimf'
dependencies = []
files = ['48240']
hgrepos = []
issue_num = 36499
keywords = []
message_count = 3.0
messages = ['339308', '339325', '339328']
nosy_count = 6.0
nosy_names = ['belopolsky', 'serhiy.storchaka', 'josh.r', 'p-ganssle', 'xtreak', 'vadimf']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '22005'
type = 'crash'
url = 'https://bugs.python.org/issue36499'
versions = ['Python 2.7', 'Python 3.5']

[vadimf]

Unpickling fails when pickling is performed with 2.7 (pickledatetime.py) and unpickling is done with 3.5 (Tested on Ubuntu 16.04)
Please see detailed error description and workaround in the comments to the attached files.

[tirkarthi]

This seems to have been fixed with 8452ca1. I checked out your example locally and renamed pickle.py to another name to avoid module collision. Ran the below commands to pickle using Python 2 and to unpickle from master before and after the relevant commit. The fix is present in 3.7 and 3.6.

➜ bpo-36499 python2 pickledatetime.py

# with 8452ca1
➜ bpo-36499 ../cpython/python.exe unpickledatetime.py
Pickle: {'timenode': datetime.datetime(2019, 4, 2, 15, 13, 2, 675110)}

# with 8452ca15f41061c8a6297d7956df22ab476d4df4~1
➜  bpo36499 ../cpython/python.exe unpickledatetime.py
Traceback (most recent call last):
  File "unpickledatetime.py", line 22, in <module>
    main()
  File "unpickledatetime.py", line 18, in main
    pkl = unpickler.load()
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/pickle.py", line 1081, in load
    dispatch[key[0]](self)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/pickle.py", line 1429, in load_reduce
    stack[-1] = func(*args)
TypeError: an integer is required (got type str)

# pickledatetime.py

import io
import pickle
from datetime import datetime


def main():
    """
    Uses python 2.7 to create a pickle file with a datetime object inside a dictionary.
    """
    data = {}
    data['timenode'] = datetime.now()
    with io.open('written_by_py27.pickle', 'wb') as handle:
        pickle.dump(data, handle, protocol=pickle.HIGHEST_PROTOCOL)

if __name__ == "__main__":
    main()

# unpickledatetime.py

import io
import pickle


def main():
    """
    Attempts to unpickle a dictionary with a datatype object inside
    """
    with io.open('written_by_py27.pickle', 'rb') as handle:
        unpickler = pickle._Unpickler(handle)
        unpickler.encoding = 'latin1'
        pkl = unpickler.load()
        print("Pickle: ", pkl)

if __name__ == "__main__":
    main()

[MojoVampire]

As noted, this is fixed in 3.6+. 3.5 is in security fix only mode ( https://devguide.python.org/#status-of-python-branches ), which is why the fix for bpo-22005 wasn't backported.