enum with inherited type won't pickle

[TomBrown]

BPO	44342
Nosy	@ethanfurman, @miss-islington
PRs	bpo-44342: [Enum] change pickling from by-value to by-name #26658 [3.10] bpo-44342: [Enum] changed pickling from by-value to by-name (GH-26658) #26660 [3.9] bpo-44342: [Enum] improve pickle support #26666 bpo-44342: [Enum] fix data type search #26667 bpo-44342: [Enum] improve test, add andrei kulakov to ACKS #26726 [3.10] bpo-44342: [Enum] fix data type search (GH-26667) #26746 [3.10] bpo-44342: [Enum] improve test, add andrei kulakov to ACKS (GH-26726) #26747 bpo-44342: [Enum] sync current docs to 3.10 #26750

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ethanfurman'
closed_at = <Date 2021-06-19.21:53:03.719>
created_at = <Date 2021-06-08.00:56:53.414>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.11', 'library']
title = "enum with inherited type won't pickle"
updated_at = <Date 2021-06-19.21:53:03.718>
user = 'https://bugs.python.org/TomBrown'

bugs.python.org fields:

activity = <Date 2021-06-19.21:53:03.718>
actor = 'ethan.furman'
assignee = 'ethan.furman'
closed = True
closed_date = <Date 2021-06-19.21:53:03.719>
closer = 'ethan.furman'
components = ['Library (Lib)']
creation = <Date 2021-06-08.00:56:53.414>
creator = 'Tom.Brown'
dependencies = []
files = []
hgrepos = []
issue_num = 44342
keywords = ['patch']
message_count = 10.0
messages = ['395300', '395578', '395585', '395589', '395609', '395613', '395615', '395898', '395902', '395903']
nosy_count = 3.0
nosy_names = ['ethan.furman', 'Tom.Brown', 'miss-islington']
pr_nums = ['26658', '26660', '26666', '26667', '26726', '26746', '26747', '26750']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44342'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

[TomBrown]

The following script runs without error in 3.8.5 and raises an error in 3.8.6, 3.9.5 and 3.10.0b1.

Source:

import enum, pickle

class MyInt(int):
    pass
    # work-around: __reduce_ex__ = int.__reduce_ex__

class MyEnum(MyInt, enum.Enum):
    A = 1

pickle.dumps(MyEnum.A)

Error (same in 3.8.6, 3.9.5 and 3.10.0b1):

Traceback (most recent call last):
  File "/home/thecap/projects/covid-data-model/./enum-pickle.py", line 12, in <module>
    pickle.dumps(MyEnum.A)
  File "/home/thecap/.pyenv/versions/3.10.0b1/lib/python3.10/enum.py", line 83, in _break_on_call_reduce
    raise TypeError('%r cannot be pickled' % self)
TypeError: MyEnum.A cannot be pickled

Like https://bugs.python.org/issue41889 this seems to be related to the fix for https://bugs.python.org/issue39587 which changes member_type from int to MyInt. A work-around is in the comment above.

[ethanfurman]

Looking into this I think the root of the problem is the way reduce is handled -- currently, Enum's __reduce_ex__ works by returning the class, and the value to use to lookup the member. Because that lookup can fail with complex enums, EnumType will sabotage reduce if it can't find support in the new enum class.

However, if __reduce_ex__ working by returning

`getattr, (self.__class, self._name_)`

then we should be fine, as that should never fail.

[ethanfurman]

New changeset 62f1d2b by Ethan Furman in branch 'main':
bpo-44342: [Enum] changed pickling from by-value to by-name (GH-26658)
62f1d2b

[ethanfurman]

New changeset b613132 by Miss Islington (bot) in branch '3.10':
bpo-44342: [Enum] changed pickling from by-value to by-name (GH-26658) (GH-26660)
b613132

[ethanfurman]

Changing __reduce_ex__ is too big a change for 3.9, so making the search for pickle support more robust there.

[ethanfurman]

New changeset 3a7cccf by Ethan Furman in branch 'main':
bpo-44342: [Enum] fix data type search (GH-26667)
3a7cccf

[ethanfurman]

Also fixing the search for the data type so that in

A -> B -> C

the data type is C and not B.

[ethanfurman]

New changeset 0f99324 by Miss Islington (bot) in branch '3.10':
bpo-44342: [Enum] fix data type search (GH-26667)
0f99324

[ethanfurman]

New changeset 41c2a4a by Ethan Furman in branch '3.10':
[3.10] bpo-44342: [Enum] improve test, add andrei kulakov to ACKS (GH-26726)
41c2a4a

[ethanfurman]

New changeset 741b8ae by Ethan Furman in branch 'main':
bpo-44342: [Enum] sync current docs to 3.10 (GH-26750)
741b8ae