datetime object fails to restore from reduction

[jaraco]

BPO	28752
Nosy	@tim-one, @jaraco, @abalkin, @vstinner, @avassalotti, @ned-deily, @serhiy-storchaka
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	datetime-reduce.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = <Date 2016-11-21.22:35:22.500>
created_at = <Date 2016-11-20.15:55:23.401>
labels = ['extension-modules', 'type-bug', '3.7']
title = 'datetime object fails to restore from reduction'
updated_at = <Date 2017-03-31.16:36:16.587>
user = 'https://github.com/jaraco'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:16.587>
actor = 'dstufft'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2016-11-21.22:35:22.500>
closer = 'belopolsky'
components = ['Extension Modules']
creation = <Date 2016-11-20.15:55:23.401>
creator = 'jaraco'
dependencies = []
files = ['45564']
hgrepos = []
issue_num = 28752
keywords = ['patch']
message_count = 13.0
messages = ['281278', '281279', '281280', '281284', '281287', '281288', '281292', '281388', '281395', '281396', '281397', '281399', '281521']
nosy_count = 8.0
nosy_names = ['tim.peters', 'jaraco', 'belopolsky', 'vstinner', 'alexandre.vassalotti', 'ned.deily', 'python-dev', 'serhiy.storchaka']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28752'
versions = ['Python 3.6', 'Python 3.7']

[jaraco]

On Python 3.5, the datetime would reduce and restore cleanly.

$ python3.5 -c "import datetime; func, params = datetime.datetime.now().__reduce__(); func(*params)"

With Python 3.6.0b3, it now fails with a TypeError.

$ python3.6 -c "import datetime; func, params = datetime.datetime.now().__reduce__(); func(*params)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: an integer is required (got type bytes)

[jaraco]

Pickle still works, so pickle must be relying on a different protocol for serialization.

$ python
Python 3.6.0b3 (v3.6.0b3:8345e066c0ed, Oct 31 2016, 18:05:23) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> import datetime
>>> pickle.loads(pickle.dumps(datetime.datetime.now()))
datetime.datetime(2016, 11, 20, 10, 56, 43, 264436)

[serhiy-storchaka]

Now pickling of the datetime.datetime objects is implemented with __reduce_ex__. __reduce__ is not defined in datetime.datetime and is inherited from datetime.date.

>>> datetime.datetime.__reduce_ex__
<method '__reduce_ex__' of 'datetime.datetime' objects>
>>> datetime.datetime.__reduce__
<method '__reduce__' of 'datetime.date' objects>

__reduce_ex__ has higher priority and is called instead of __reduce__. It is weird that __reduce_ex__ and __reduce__ are not consistent. __reduce__ should be defined as

    def __reduce__(self):
        return self.__reduce_ex__(2)

or just

    __reduce__ = object.__reduce__

Other way is to define __reduce_ex__ instead of __reduce__ in datetime.date.

[serhiy-storchaka]

Proposed patch restores the __reduce__() methods and makes Python and C implementations more consistent.

[abalkin]

On Nov 20, 2016, at 11:10 AM, Serhiy Storchaka <report@bugs.python.org> wrote:

Other way is to define __reduce_ex__ instead of __reduce__ in datetime.date.

I would prefer this solution.

[serhiy-storchaka]

Other way is to define __reduce_ex__ instead of __reduce__ in datetime.date.

Sorry, I was wrong. This would wouldn't work with C implementation. And explicitly setting __reduce__ = object.__reduce__ doesn't work. The only way is to define both __reduce_ex__ and __reduce__ for time and datewtime.

[abalkin]

On Nov 20, 2016, at 12:34 PM, Serhiy Storchaka <report@bugs.python.org> wrote:

The only way is to define both __reduce_ex__ and __reduce__ for time and datewtime.

OK. I'll review your patch and get it committed shortly.

[abalkin]

The patch LGTM. I'll commit it tonight unless Serhiy beats me to it.

[ned-deily]

If you can push this in the next hour or two, it can still make b4.

[abalkin]

If you can push this in the next hour or two ...

I'll do it now.

[python-dev]

New changeset 0a2a0061e425 by Serhiy Storchaka in branch '3.6':
Issue bpo-28752: Restored the __reduce__() methods of datetime objects.
https://hg.python.org/cpython/rev/0a2a0061e425

New changeset 23140bd66d86 by Serhiy Storchaka in branch 'default':
Issue bpo-28752: Restored the __reduce__() methods of datetime objects.
https://hg.python.org/cpython/rev/23140bd66d86

[abalkin]

It looks like Serhiy has already committed it. Thanks!

[jaraco]

Thanks all! So pleased to see this fixed.